I was talking to Jim today about this problem with JUnit and came up with a solution. Give unit your test methods classifications.

For example, the team wants to ensure that the project is "always green". Tests that work should stay working. These tests would be called A tests methods. However if a bug is found in the code good practise dictates you want to write a test for it right away, exposing the defect. This new defect test is a B test method.

So you don't want to break the "always green" A methods, but you still allow for the other B tests to exist so they reveal bugs.

We also want to store the results of the unit test for each and every checkin we do, or at least at major intervals (maybe only daily builds) so that we can compare them and notice changes in the B test methods (ie. new passes). For these we shouldn't have to worry about A methods because developers aren't allowed to check in code if it breaks any existing A test method - don't break the build!