I've gotten quite a bit out of the book Testing Extreme Programming. The XP Series books are somewhat general in nature and they have a certain level of non-prescriptive freedom -- more like suggestions in the way that some aspects of quality are handled.

For example, the issue of customer testing is brought up in a less than explicit way (and I found that in the original XP book -- Extreme Programming Explained -- even more so). I think I found a page or two explaining exactly how to approach customer testing, and they were buried amongst other things. Given the importance of customer testing in the process this was pretty surprising.

So let's review a bit: XP advises automating testing as much as possible so that testing can be done very often. It also advises that the customer write acceptance tests to verify that stories are complete. Now, sometimes this isn't practical. What if your customer doesn't know how to program? Well, he can still write out the situations that the story should solve and with the help of an XP tester figure out all of the error possibilities and variants. Then someone else can write the customer tests on the customer's behalf.

So now you have a pile of variants to test. Great, more GUI testing right? Not really.

Most GUI-based software has a clear boundary between the GUI and the underlying business logic. As much code should be put in the business logic as possible so it can be unit tested. In XP, the top-most level just below the GUI code has integration unit tests. Since they are the top-most level that you can unit test, they are also automatic customer acceptance tests.

Traditional quality assurance (QA) is usually a bunch of people banging on the working program trying to break it. They could be running through use cases (and variants), they could be running build verification tests (usage scripts meant to test basic use cases) or they could just be banging on the software trying to break it. Usually these testers have no clue how the software is implemented, they just try to test it to death, find bugs and ultimately improve the quality of the product. This is called black-box testing because the tester can't see inside the software he's testing.

XP doesn't assume that the programmer writing the customer acceptance tests doesn't have access to the code. Why shouldn't he have access to all of the code? So now the test writer can do white-box (also called clear-box) customer acceptance testing. How does this help?

Let's say you have a method called by the GUI that has one correct use and 6 error cases. In all of the error cases, an error message is returned by the method and the GUI displays that error message in a model alert dialog. With black box testing you would have to figure out how to trigger the proper error situation in the GUI to generate each of the six errors and verify that the proper error dialog appears for each one. Then you have to make sure that the non-error way of using the method works as well, so that's seven manual GUI tests.

But if you look at the code, you'll realize that the only thing the GUI is doing is calling the method and checking to see if an error occured. If one did occur, it takes the error message and displaying it in a dialog. So that's two cases to check in the GUI (the correct way and one error), not seven. Then you use automated customer acceptance tests to test all six error cases and the one good case for that lower level method. You verify that the error is returned for each use case variant in an automatic unit test instead of manually in the GUI. So now you have two manual tests and seven automatic tests -- and the automatic ones will be run multiple times a day to catch regressions.

This kind of savings can really add up. Not only do you get the added benefit of testing all of the variants automatically instead of manually but you also save a lot of time doing manual test runs as well. You still have to do some manual testing but because it's done in a white box fashion you only need to manually test what the GUI code is doing, not what all of the lower layers are doing -- that's already handled by the automatic unit tests, why do it again?

By the way, even though you are only testing one error case manually you might want to rotate through the six different error situations for some variety. That way you won't get a strange false positive on one error when one of the other five is really broken. Then the worst you might be broken in that situation is a week or so, depending on how often you manual test. With the white-box example I gave that kind of oversight might seem pretty unlikely but real GUIs are often far more complicated.

As a member of the XP team, you can't spend all of your time setting up and running manual tests anyway, it's just not worth it. Given the code turnover of the XP process, your tests could get nuked at any moment -- that's why you want them to be automatic, they are much easier to create and less painful to delete, making the code and tests more easily refactorable. Some developers resist refactoring code they have perfected; you never want to do that in XP.

Your customer/users will pick up the majority of the remaining GUI regressions just in casual use just the iteration for feedback anyway. As long as you fix these regressions quickly, you can get away with missing a few minor regressions with your less-than perfect (but more agile) manual testing once in a while. But don't leave too much of this regression catching to your customer -- they could get annoyed and it distracts them from what they are trying to do: give you great feedback for the next iteration.