«« Code Covering ParserConfigurationException Smalltalk Eclipse IDE Presentation »»
blog header image
Java Unit Testing Exceptions with JUnit

When writing unit tests for a method you want to make sure you cover all of the boundary cases and possible inputs. Some inputs will cause the method to throw an exception and you'll want to verify that output.

Exceptions are caught using a try/catch block. You're trying out a bit of code in the try block and if an exception is thrown by a line in the try block, the catch blocks are there to catch certain types of exceptions you are interested in.

There are two types of exceptions: checked and unchecked exceptions. The differences are outlined extremely well by Josh Bloch in his book Effective Java. This book really is a must for any Java programmer and I can't recommend it enough.

Checked exceptions are for expected errors and should be dealt with programmatically. For example, the checked exception FileNotFoundException happens when you try to open a file that cannot be found. You want your code to work around this situation because users can easily enter bogus files or paths. If a function throws a checked exception, it must indicate that in its method header:

public int read(byte[] b) throws IOException

Unchecked exceptions on the other hand are for programming errors and are also called runtime exceptions. All runtime exceptions are subclasses of the RuntimeException class and do not have to be declared in the method headers of methods that throw those exceptions. They also do not have to be (and should not be) explicitly caught by the code that calls the method. Runtime exceptions should be allowed to trickle all the way up to the virtual machine uncaught and terminate the program because they indicate situations that the program is not meant to recover from. So here's a good general rule: unless you are debugging, never catch Exception because you'll catch all of the runtime exceptions too.

As a side note, Eclipse is great at telling you when you need to catch checked exceptions as you edit code. It even knows that subclasses of RuntimeException do not need to be explicitly caught. It's a fairly simple thing but it saves tons of time because you don't realise it oops! at compile time.

Now, one of Bloch's good pieces of advice is "only throw exceptions in exceptional cases". Part of the rationale is that it creates weird flow in your programs, which can be hard to debug (Joel Spolsky wrote a good article on the downsides of exceptions). But the main reason is even simpler: throwing and catching exceptions is a computationally expensive operation and so it should be done rarely.

In tests however, you should catch all exceptions to make sure they are working properly. For example, my code from yesterday called a method with null to make sure a NullPointerException was thrown:

public void testSetupBuilder_Null()
{
   XMLRepository repository = XMLRepository.getInstance();

   try
   {
      repository.setupBuilder(null);
   }
   catch (NullPointerException npe)
   {
      return;
   }

   fail("Expected NullPointerException");
}

Let's examine the flow. The call we are interested in is repository.setupBuilder(null);, which is surrounded by a try/catch block. If a NullPointerException is thrown by the method call, the flow will go into the catch block and hit the return; call, ending the test method (by default tests pass if no assertions are made). If NullPointerException is not thrown the catch block will be skipped and JUnit's fail() method will be called, which fails that test immediately.

Couldn't the fail() call be in the try block below the call to repository.setupBuilder(null);?

public void testSetupBuilder_Null()
{
   XMLRepository repository = XMLRepository.getInstance();

   try
   {
      repository.setupBuilder(null);
      fail("Expected NullPointerException");
   }
   catch (NullPointerException npe)
   {
      return;
   }

   //other runtime exceptions are caught by JUnit
}

If an exception isn't thrown the fail() call will be hit. True, but let's say hypothetically for a moment that repository.setupBuilder(); also throws the unchecked exception IllegalArgumentException. If that exception is thrown instead of NullPointerException the test will pass because we're only catching NullPointerException and the fail() is in the try block and is missed. We don't want the test to pass in that situation because the wrong exception was thrown. Having the fail() call at the end of the test method is more comprehensive.

What I wrote before isn't right so I've crossed it out. The Eclipse JUnit plugin will actually catch the thrown IllegalArgumentException and fail the test, not pass it. Depending on the robustness of your unit testing framework this may work as well. Having the fail() call at the end is more explicit but allowing the exception to be thrown outside of the test will let you examine the stack trace, like I do in the next example. Thanks Andrew for correcting me in the comments.

I feel like making this post longer, so let's do another example. Let's say I have a method call that throws a checked exception but I want to verify an unchecked exception. Here's the (admittedly fictitious) method header:

public int read(File f) throws IOException

and the test method for an input of null:

public void testRead_Null() throws IOException
{
   try
   {
      int byte = read(null);
   }
   catch (NullPointerException npe)
   {
      return;
   }

   fail("Expected NullPointerException");
}

Notice that because we're not explicitly catching the checked IOException we have to make our test method throw it. That's OK though because in the event that an IOException occurs instead of NullPointerException the JUnit testing framework will display a proper error message. The JUnit plugin in Eclipse even shows a stack trace for you. So it's actually in your advantage to write the test methods like this when using Eclipse.

But there is another way. You can handle the IOException yourself which takes a bit more work:

public void testRead_Null()
{
   try
   {
      int byte = read(null);
   }
   catch (NullPointerException npe)
   {
      return;
   }
   catch (IOException ioe)
   {
      e.printStackTrace();
   }

   fail("Expected NullPointerException");
}

This will print the stack trace to standard out, and then hit the fail() call at the end. I prefer the other way because it handles the checked exception better with JUnit. It might depend more on your unit testing framework and how robust it is so I've included this alternate way.

I hope this post helps people out. If you have any comments or corrections let me know.

Links:
JUnit FAQ: testing exceptions

Posted at February 13, 2004 at 01:45 PM EST
Last updated February 13, 2004 at 01:45 PM EST
Comments

I indeed prefer this style better - and will start using it, but I want to point out a few subtle things.

Is Exception handling slower? Personally, I don't care - I think the exception handling approach to coding is just plain silly (I prefer response driven... I wrote about this I think, pretty much your GUI elements 'attempt' to do things, and response appropriately - getting off point). But, when I brought up speed (to try and convince the boss away from exception) he said, 'nada, makes little to no difference'.

Brb, I am going to check this out in NUnit.

» Posted by: aforward at February 13, 2004 02:42 PM

Of course Java and .NET use different archtectures so the performance hit of exceptions might not be the same. It would be interesting to write a little Java test application to verify the penalty.

» Posted by: Ryan at February 13, 2004 02:51 PM

Exceptions are slower in .Net..

Returning false a million times took 0.15 miliseconds, verus almost 21 seconds (that's right 21000 miliseconds).

normal: 0.15
exception: 20.703

» Posted by: aforward at February 13, 2004 03:34 PM

I cannot speak for java right now, but Ryan's justification is flawed for .Net (please note, I still prefer his method of doing things).

According to Ryan, the following *should* pass... I am throwing an InvalidOperationException, but I am expecting a NullReferenceException.

[Test]
public void ShouldStillPassAccordingToRyanLowe()
{

try
{
throwInvalidOperationException();
Assert.Fail("no good");
}
catch (NullReferenceException goodToGo)
{
goodToGo.GetType();
}
}


Okay, so there's a small flaw with the argument... that does not change the fact that the fail should go at the bottom of the method... much easier to see that's going on... and it is far to easy to forget about even included the fail() - and for those that do not make sure your test fails first... you'd be out of luck

» Posted by: aforward at February 13, 2004 03:37 PM

Let's go crosseyed together....

My example test fails, but from ryan's understading of java - it should falsely pass.

Ryan is (or has) fixed up the sheeat - so my comments will make no sense now!

» Posted by: aforward at February 13, 2004 03:44 PM

Yeah, it shouldn't falsely pass -- you are right. I've changed my post to reflect that. Thanks buddy.

» Posted by: Ryan at February 13, 2004 04:25 PM

Here are some updated numbers from Andrew for .NET. I'm not sure what "return response" is though. We'll have to ask him on Monday.

return bool: 0.187
return response: 1.187
throw exception: 28.453
return exception: 1.984

» Posted by: Ryan at February 14, 2004 12:33 AM

I'm not sure how you did these benchmarks but they could be skewed by compiler optimizations. Microbenchmarks usually aren't a good indicator of performance. I'll write a post about it this weekend but here's a link (specific to Java) to tid you over until then:

http://java.sun.com/products/hotspot/Q+A.html

» Posted by: Ryan at February 14, 2004 01:58 AM

Hi.

Return exception does just that, it returns an exception as opposed to throwing it...

public Exception returnException()
{
return new Exception();
}

That way, you cannot argue about the time it takes to create the exception versus the time it takes to throw and catch it.

» Posted by: aforward at February 14, 2004 07:37 AM

Sorry to wake up an old discussion, but there's an important point that you seem to have missed: If you let the exception escape, at least in IntelliJ's JUnit plugin, it gives an error rather than a failure, and thus the tearDown() method doesn't get called, which can be problematic in some cases.

» Posted by: Lars Clausen at November 4, 2004 05:59 AM

I just checked out in my Eclipse version (3.0, 200310101454) with some simple print out statments in the setup and teardown methods. Even when the test throws an exception like NullPointerException the teardown still is run.

I think you would WANT the test to throw an error. I think that it's good to be able to identify the difference between the code not passing the test (failing what you were testing for) and if the test failed because of one of your assumptions for that test failed (like an IOException for example)

» Posted by: Jim at November 4, 2004 05:48 PM
Google
 
Search scope: Web ryanlowe.ca