Planning Out Unit Tests

Test driven development is the practice of writing unit tests before writing any code. It's very helpful in knowing when to stop writing code (when all the tests pass) so you don't create unnecessary code bloat. TDD is a common practice in Agile or Extreme Programming methodologies.

I'm a fan of test driven development (TDD) about 99% of the time. The 1% I don't like it is when writing code that I'm not sure how to write. When you're prototyping, it's hard to write unit tests ahead of time, because you're not sure how the code is supposed to work or even what it's supposed to do yet. But this post is about the 99% of the time it is worthwhile (perhaps I'll talk about that other 1% in more detail at a later time).

So for the 99% of the time that it is beneficial to use TDD, how do you know how many unit tests to write and what to test? A web app I’m currently working on is used to process applications for licenses through a large state agency. When it came time to write a unit test for the application approval method, ApproveApplication(), we sat down and came up with all of the variables that could affect the approval process. A few mathematical calculations later and we discovered that there were roughly 3.2 billion combinations of the variables. Clearly, we were not going to write 3.2 billion unit tests for a single method. This is the first rule to remember when writing unit tests – you do not need to test every possible combination of variables.

We went into a conference room and wrote out all of the most likely use cases and came up with 523 tests. If this number results in an amount that fits your scope and schedule, then you can stop reading here and go write those tests! But for us, 523 tests for a single approval method was not acceptable, because our business domain consists of 11 different applications, so we were really talking about 5,753 unit test. We decided to see if any of these tests could be combined to reduce the total number even further. At this point, I bet one of my coworkers, Christine Lambden,  that we couldn't get the number down to 80 tests or less. She disagreed. But more on this later…

When trying to combine unit tests, avoid combining two tests into one if you just end up with a single test that is twice as long. This won’t gain you anything, except a meaningless reduction in the number of tests. The second rule is do not combine tests unless it gains you something meaningful.

First we looked at combining tests that were testing the same business cases, or where business cases overlapped. This got us down to roughly 140 tests, give or take a few. It also makes use of the third rule: combine tests with overlapping or duplicated use cases.

The next step was to combine tests that exercised the same section of code. Since the same piece of code doesn’t need to be tested more than once, combine tests that exercises that same piece of code.

The final step was to improve the performance of the unit tests. In our particular case, the setup for each test is very time-consuming, so by combining tests where we could would save us a considerable amount of time every time we ran unit tests. So if it makes a significant impact, combine tests if it will help performance.

In summary, my rules of planning out unit tests in order of importance are:

1. Do not test every combination of variables.
2. Combine tests when you can, but not unless it gains you something meaningful.
3. Combine or eliminate tests with overlapping or duplicates business use cases.
4. Combine or eliminate tests that exercise the same piece of code.
5. Combine tests if it will significantly improve the performance of unit test execution.

Oh, and about that bet…our final tally was 43 tests, so I definitely owe my coworker lunch. That should be Rule #6: Never bet against coworkers with more real-world experience than yourself. That's okay, though, because whenever I speak with her, I usually end up learning something new, so I don't think I really lost at all. :)