Writing good tests is still hard these days and it's a lot harder than it needs to be. I see a lot of places where things can be better organized. The main problem is that most people look at tests in source code files.

Writing tests is pretty easy but tedious. The hard part is keeping track of what you've tested and what you have left to test. Then you have to keep track of this as your code is refactored over months and years. Code coverage analysis tools help a little bit here but they are far from perfect. Even if you test first you can still miss things.

I described the testing coverage problem a year ago from an IDE angle. In that post you can see how Eclipse's Outline view makes it easier to see all of the method names in the source code file (test case for a class, which usually contains several methods). If you name your methods descriptively you can get a decent idea of how much your testing covers. So that's a basic example of a view that's not source code.

It would be much better if I could get a condensed table like the one in that post. Then I could look at it and the code and check the code paths off. This kind of table would be handy in an Eclipse view when I'm editing a test case. It would even be handy when doing test-first.

Generating a table like this from regular unit tests would take some parsing, data mining and guessing. Even if you went into the test and looked at the methods it used, how would you known which one was the "main testing target"? Maybe there isn't one for that method, it's just a sequence of calls that must work. People don't put enough data in their unit tests to describe their intentions, they just make test methods of code sequences they think the API will have to do and bang on methods to make sure they don't break. Organization is only as good as the team's discipline.

That's where the tests-generated-from-XML approach I mentioned yesterday differs a bit. Since it's generated from specific data in the first place, you can get your testers to enter this data or at least be able to tell when it's missing. The generated tests might not use half this data but you can use it in other places and do much more powerful data mining and reporting on your testing effort.

When I say "get your testers to do X" I mean that they will see the benefits of entering it because it's used in a report that helps them be more efficient day-to-day. The data they are entering doesn't get lost or never used like a Word file that gets archived on some file server. This stuff goes right into the CVS alongside the code. It's a first class citizen.

An alternative to this is to store all of this data in the source files themselves, like Javadoc does for code. That might clutter things up, but more importantly there's no standard for that and I don't feel like sitting around waiting for one. I need more test data now.