~ryanlowe/blog: Agility vs Testing

Agility vs Testing

I have been thinking about the right approach to take with agile software development and testing. If the project demanded high quality, there are two ways that could be done:

1. Test everything as you add it to the system, which is less agile than
2. Adding things to the system and then testing them later

Even with the second approach you could do test driven development (TDD), just not cover every single possible error case. The less testing you do, the less code maintenance there will be and features will get in faster.

This contrasts with the first approach where you try to test everything really well up front to avoid problems later, especially integration problems and subtle bugs caused by errors in corner cases.

Getting the features in quickly is important for feedback, but how good is that feedback if the product is broken? It seems as though a combination of the two should be done: do enough up front testing to avoid the majority of problems but still not too much that it will slow you down.

Then at the end of an iteration for a week you can go through the development you did and add tests to tighten the code you know will be sticking around. Having the discipline to go back and do this would be the hard part, which is why the first approach is much easier to advocate.

Pushing this secondary testing and tightening to a team of quality people is also an option. The upside is that this work can be done in parallel with new development. The downside, and I've mentioned this before, is that the communication overhead increases and developers don't learn from their mistakes if they don't have to test their own code.

Posted at May 14, 2004 at 04:52 PM EST
Last updated May 14, 2004 at 04:52 PM EST

Comments

I try not to write code unless a test is failing (which is the whole idea behind tdd produces better coveraged code).

As issues are uncovered, more tests are added that expose those cases that you didn't think of.

It is not that the in the first round of testing of poor quality or inadequate - rather, it seems moreso that your understanding of how the stuff should work is not complete.

Now, I do not completely follow how testing everything isn't agile - provide that it is automated (aka regresionnable)

» Posted by: aforward at May 14, 2004 11:10 PM

Because it takes more time to write all of those tests, rather than just moving on to new features.

At some point, in order to keep agility at a reasonable level, you have to abandon testing and add more features. You only have time for so much testing before you have to add new value again. Of course you could argue that increased testing is adding value ...

» Posted by: Ryan at May 15, 2004 01:39 AM

Have you considered automation in your testing approach?

I would think that if you're using TDD and then automating your use cases and use case exceptions then you're fairly well covered.

I'd be quite leary of approach #1 since as we all know, just because something works when you write it doesn't mean that it'll stay working for months on end..

(Unit Tests and automation on the other hand will ensure that it does..)

» Posted by: peter at May 17, 2004 06:55 PM

Peter: yep, exactly. The more tests there are, the more there are to maintain when things change. This makes the system less agile on the whole.

Even if you were to tighten the testing at the end of the interation (after giving some time for user feedback), you are still making the system less agile between iterations.

At least within an iteration you aren't committing to write all of those tests up front -- it doesn't make sense to do that until the users really like the new features.

It's quite easy to remove features (and tests) entirely but it's another to refactor them and maintain them as the code morphs. So in that case, the less number of tests (to maintain) the better.

So there's no silver bullet here -- there no use using an agile process if you are slowed down by too much testing. I don't think that's the point. The point is to use good judgement on how much testing is needed.

However, if you want a product with reputable quality eventually you will have to write all of these tests, automatic and manual tests, to cover every situation. If you just wait until people report bugs, people will get sick of reporting bugs and stop using the product. COTS products don't have the luxury of many many test users like open source projects!

And even open source projects, unless they are quite popular, don't have these resources either. Build-it-Fix-it just doesn't work well without good bug reporting.

» Posted by: Ryan at May 17, 2004 09:59 PM