Test Anti-Pattern: Proving The Code is Written Like the Code Is Written

21 Oct 2005 • Software Development

Testing is suppose to verify that code does what we want. However, I have seen many “unit” tests that spend a lot of time verifying that code works how they wrote it, rather than verifying that it achieves the desired result. The worst cases of this I have seen involve the use of mock objects. This is an real example (I changed the names to protect the guilty) of that anti-pattern (using Java and EasyMock):


public void testDoSomthingUseful()
{
  MockControl mockObjToTestControl = MockClassControl.createControl(ClassToTest.class);
  ClassToTest mockObjToTest = (ClassToTest)mockObjToTestControl.getMock();
  
  mockObjToTest.doSubtask1();
  mockObjToTestControl.setReturnValue("abc");
  
  mockObjToTest.doSubtask2();
  mockObjToTestControl.setReturnValue(1);

  ClassToTest objToTest = new ClassToTest()
  {
    String doSubtask1()
    {
      return mockObjToTest.doSubtask1();
    }

    String doSubtask2()
    {
      return mockObjToTest.doSubtask2();
    }
  }

   mockObjToTest.replay();

   objToTest.doSomethingUseful();

   mockObjToTest.verify();
}

As you can see this code proves that the doSomethingUseful() calls do doSubtask1() and doSubtask2(). The problem is that this is a total useless test. The only thing it proves is that the code is written the way the code is written. There are not even any assertions that the desired result as achieved. Even if there were some assertions made about the the results of doing something useful you still have a test which is completely and absolutely tied to the current implementation of the doSomethingUseful() method.

Tests should check for the desired behavior1 and avoid checking implementation details. Our tests should not care if next week the doSomethingUseful() method is refactored such that it does not call doSubtask1() as long as it still achieves the same result. The above example is an extreme case and very few people write tests as pathologically broken as that. However, a disturbing number of people write tests with little thought to whether their are verifying desired behavior or implementation details.

A key way to encourage this behavior oriented approach is to only use an objects, globally, public interface in the tests. One objection I have heard to this approach is that you will miss bugs because you correctness checks will, inevitably, be made at a higher level. This argument is just wrong. Any bug that is undetectable from the public interface does not matter. Of course, detecting some bugs via the public interface might be a bit more difficult than immediately reaching into the guts of your objects but it will be better because you will be testing the intended behavior of those objects, not their implementations. As a side benefit you will be implicitly documenting the public interface with is a Good Thing.

As with any rule there are a few situation in which I do write tests against non-public parts of an objects interface. For example, if you have a non-public method that you suspect is likely to break (e.g. it embodies a non-trivial algorithm or involves non-trivial interaction with other components) and it is difficult to diagnose it’s failures from the public interface. However, I consider this confluence of situations to be extremely suspect. It will most often result from sub-optimal design, better solved by refactoring than than by writing tests against the non-public interface.

1. Dave Astels is working on something he calls BDD (these slides are good too). I think he has got it just about right and I am looking forward to trying out RSpec real soon now.

Rain

19 Oct 2005 • Miscellaneous • Personal

It is a beautiful gray and rainy day here in Colorado. It almost never rains here and I do miss the rain. Elliot likes rain too. I had a hard time getting him to day care today because he wanted to keep standing in the rain. The whole time he was telling me about how it was hitting his hair.

New Job

18 Oct 2005 • Miscellaneous

I have managed to get a paying RoR job! Yay! (Thanks, Donnie.)

While I was looking for a job this time I was surprised by the demand for dynamic language skills. There are still a lot more jobs doing static languages but there are a few companies in the Denver area that use languages you can love (like Ruby and Python). This is much nicer than the last time I was looking when the only choices were Java and C++ (with a few .net jobs here and there). It feels like dynamic languages are gaining some serious ground lately and I am extremely happy to be part of that trend.