~ryanlowe/blog: Agility vs Quality

Agility vs Quality

The XP guys seem to want it both ways. They tell you that with XP you're agile but can have quality too. I disagree, great quality takes time and resources. The coupling that testing creates between code and tests degrades the product's agility to a degree. However it also improves agility by giving developers the confidence to refactor without fear that they may be introducing new bugs.

Agility is sometimes said to be the YAGNI philosophy. YAGNI leads to not writing a lot of unit tests -- just ones for important or tricky situations. Unless you're a lucky or amazing programmer your product's quality will not be great.

The subtle tricky boundary cases that are covered by comprehensive boundary value analysis are often the ones that break when you refactor. During refactoring you want to know immediately if any of these subtle boundary cases have regressed. Anyone who's done a lot of refactoring knows that these tests can often provide a valuable safety net and give the developer confidence to refactor the code without introducing as many defects. Jim and I have talked about this a few times.

Good quality takes time and effort. Even if the majority of your tests are automated you still have to put a lot of resources into making them. When the code changes -- as part of your plan to be agile -- you lose some of the investment you've made in testing. This is a reasonable sacrifice to make in the name of forward progress.

Striving for agility may discourage developers from writing tests on areas they think have to be more agile. Lazy programmers may realise the downside of uncomprehensive testing after refactoring poorly-tested areas of code. The time they saved not making tests previously comes back to haunt them as they try to track down defects introduced by the refactoring. These areas of code become regression defect magnets until someone goes back to complete the unit testing.

Update 10:26 PM: James Robertson responds to this post. I actually have used a lot of the XP practises this year during AudioMan development.

As I mentioned above, we found that testing does improve developer confidence during refactoring but that does not discount the fact that with more tests you have increased test-coupling. When you refactor the code you have to bring the tests with it, which can be a lot of work depending on how drastic the change is. IDEs are not smart enough to take care of all of this work yet. This could discourage programmers from refactoring because they see it as a nuisance. That affects the agility of a project.

YAGNI does refer to features but I believe that the XP 'purists' would also say it extends to testing as well. I got this impression after reading Beck's Extreme Programming Explained. I will find the exact passage when I get the book back.

Do XP purists like Beck check for simple things like NullPointerExceptions (Java) in their APIs? I don't think so. They would see these checks as unnecessary. Of course they've been programming for years and they don't make these stupid mistakes. Expert programmers can leave these testing holes to reduce test-code coupling but most of us cannot afford to.

Update 12:22 AM James responds again. I made the mistake of not mentioning that the AudioMan project did not follow the YAGNI approach to testing, I apologize. I do not advocate it, I only meant to say that from what I read I thought that Beck did (obviously I have not read enough).

In fact there were a few refactorings that took more work because of the high degree of test-code coupling we had. With our unit tests we tried to test every possible input to our API methods, doing a sort of informal boundary value analysis. We have 450+ unit tests on a relatively small project (80KB of code plus 130KB of UI code and 208KB of unit tests).

Of course this kind of testing gives you a great safety net for refactoring but does it affect agility? Maybe in the short term because refactoring is more painful (more things to manually change) but in the long term heavily unit testing an agile product can keep quality in check. In my limited experience this has been the case.

Posted at December 03, 2003 at 04:40 AM EST
Last updated December 03, 2003 at 04:40 AM EST

Comments

"XP Explained" doesn't have much about testing - it doesn't even describe test-driven development. You're better off reading "Extreme Programming Installed," which has examples of test-driven development, and the various books on TDD - "Test Driven Development: By Example" by Kent Beck, and "Unit Testing in Java: How Tests Drive the Code" by Johannes Link, et al, etc.

I've found that extensive unit tests (a.k.a. programmer tests) free me up to make extensive refactorings safely, which would be very dangerous to do without the "change detection" that the unit test suites provide.

With proper use of the rule "once and only once" and various design patterns to reduce coupling and increase cohesion, I don't find tests to be a drag on making progress, as long as the tests compile and run quickly.

» Posted by: keith ray at December 3, 2003 11:09 PM

In my experience the XP 'purist' (if I can use such a term) would not have a YAGNI attitude to testing. YAGNI applies to application code only.

Indeed, since the XP purist is coding test first they shouldn't be writing any code without a failing test first.

I've never seen the problem of not-enough unit tests causing regression problems in any team that's been following all of the XP coding practices. Have you actually seen this happening or is this just a hypothesis?

» Posted by: Adrian Howard at December 4, 2003 10:04 AM

Well yeah, we had areas that were not tested 'perfectly' in our code. When the bug was discovered that's when we wrote the test for it and made it pass. Sometimes these bugs are detected during refactoring because you're using a boundary case in a slightly different way and exposing the bug.

So that is an instance where not enough testing caused regression problems. I agree that developers should try to test all of the possibilities up front, but sometimes that just doesn't happen because of ignorance or laziness. This is the real world I'm talking about now, not an ideal XP situation. Personally, we strive to test every boundary case right off the bat -- it saves so much time later that it is worth it.

I got the impression that guys like Beck didn't write tests for every single case because they saw it as a waste of time ... I extrapolated that to You Aren't Going To Need to Test It (YAGNTI). When I talk about XP purists, I'm referring to them. Perhaps I should call them XP idealists. I remember reading about it but I'm not sure if it was in XP Explained or on a web site like xprogramming.com. I could be remembering wrong.

» Posted by: Ryan Lowe at December 4, 2003 12:14 PM

Are there any figures about the penetration of XP programming to real-world companies?

I still haven't worked for a company where developers do more than unit test; usually it is QA that is responsible for writing the test cases, post production.

What company has time/money to pay for extensive design/doc/code cycles...

» Posted by: Travis at December 4, 2003 03:08 PM

I haven't seen any figures but I am curious as well. XP seems to be gaining grass-roots acceptance, brought in by curious developers. The problem with 'ideal XP' is that it takes a radical change and everyone on the team has to drink the kool-aid at once or it won't work.

It also completely changes the relationship with the customer. The customer has to be more committed to the project and sometimes even offer to staff someone on your site to monitor progress and write acceptance tests. The customer then has to believe this person is worth the effort and that it will improve the end product.

You could probably adapt some of the principles of XP slowly, going from just unit-testing to test-driven-development (TDD) to refactoring to pair programming. I'm not sure if this would be more or less painful than just jumping in head first.

It's probably a lot easier to use XP on a new project as well, if you started using XP in the middle of an old project you'd probably have a lot of unit tests to write. That might discourage developers (and managers and the customer) because it would seem like they are making little progress for a while.

But all that is just speculation on my part given what I know about XP. It would be cool to see some extensive real life experiences with it. Andrew went through some pains getting people to us XP practises like TDD, but he says now people really enjoy it. I don't want to put words in his mouth though, maybe he'll blog about it more. :)

» Posted by: Ryan at December 4, 2003 04:54 PM

I don't think it's it going to happen where I work. Though I'ld love to have it. From the work Andrew has written, it's so great to be able to know so many functions still do what they're supposed to do. For us, there is no addional time alloted in the project plan for that extra time needed to write tests.

Of course, you could argue that XP makes you spend less time at the end of the project cycle, but try convincing the managers that have been doing it their way for years.

I'm sure my problem has been discussed in millions of threads already.

» Posted by: roy at December 4, 2003 05:51 PM