Don't get me wrong -- I like code coverage. You just have to understand what you're looking at.

Code coverage tells you what lines of program code that your tests hit. It doesn't tell you that you have perfect software when your tests run 100% green and you have 100% coverage. All that means is that your tests cover what's been implemented.

Often bugs include what hasn't been implemented. Have you covered error possibility 14 in your code? Is feature ABC in the codebase yet? These are easy questions to answer if you know how to read the tests ... but that is difficult. Compounding this problem is that your test's intentions may be correct but the test implementation could be wrong. Don't think that only your program code has bugs in it -- your tests do too. However, you could always just test your tests and test your test's tests .... etc. (please don't do this, I'm kidding)

Good tests and program code are like two playing cards resting against each other: if either are wrong the test still fails. Sometimes though you get a situation where a bug is symmetrical on both sides and the test passes anyway. These tests can give you a false sense of security and are usually only found when someone else tries to use the API in a real life situation or during a code/test audit.

Tests that have poor asserts can also give you a false sense of security. Say you have a function that returns an array: Widget[] getWidgets(int parameter). If you are testing on a known set of Widgets in your unit test, you'll expect a specific array to be returned by this function. Someone may write the test

assertEquals(3, getWidgets(42).length);

and it will pass. But what were the three widgets? Were they the right three widgets? Were they in the correct order? Do we care about order? Relaxed testing like this can result in really hard to find bugs when someone else comes along and trusts the getWidgets() method too much.

I'm finding that unit tests are really hard to manage. Not only is it difficult to discover the intent of the test, but also what part of the program code it actually exercises. But we put up with these disadvantages because having tests that we have to figure out is still better than having no tests at all.

However, it would be nice if I could go one method at a time in an IDE like Eclipse and see in one perspective:

• The tests I am using for that method and a brief synopsis of what each of them cover (in plain English or shorthand)
• If I click on a test I see the code coverage for that method (and I can select multiple or all tests in the list) overlayed on the code.
• The IDE tells me if a test overlaps another test or if the test does a subset of another test so that I can have the simplest set of tests possible to increase agility (see XP)
• I can execute the tests on a per-method basis. Better yet, Eclipse does the compiling and code coverage analysis on the fly while I write the tests (ie. program code on top view and test code on bottom view).

The reason I say per-method is because this is what I find I'm typically dealing with when I write and implement an API: I usually go one method at a time because it's easiest to keep it all in my brain. If I jumped back and forth a lot I'd probably just confuse myself. Sure, I delegate to other functions inside the one I'm writing but at that time I think of them at a high level of abstraction. I don't need to concern myself with their details.

So that would be nice. Anyone want to write me an Eclipse plugin?

Posted at June 25, 2003 at 06:48 PM EST
Last updated June 25, 2003 at 06:48 PM EST