Unit Testing Guidelines
1. Keep unit tests small and fast
Ideally the entire test suite should be executed before every code check in. Keeping the tests fast reduce the development turnaround time.
2. Unit tests should be fully automated and non-interactive
The test suite is normally executed on a regular basis and must be fully automated to be useful. If the results require manual inspection the tests are not proper unit tests.
3. Make unit tests simple to run
Configure the development environment so that single tests and test suites can be run by a single command or a one button click.
4. Measure the tests
Apply a coverage analysis to the test runs so that it is possible the read the exact execution coverage and investigate which paths of the code is executed.
5. Fix failing tests immediately
Each developer is responsible for making sure a test runs successfully upon check in, and that all tests runs successfully upon code check in. If a test fails as part of a regular test execution the entire team should drop what they are currently doing and make sure the problem gets fixed.
6. Keep testing at unit level
Unit testing is about testing classes. There should be one test class per ordinary class and the class behavior should be tested in isolation. Avoid the temptation to test an entire work-flow using a unit testing framework, as such tests are slow and hard to maintain. Work-flow testing may have its place, but it is not unit testing and it must be set up and executed independent.
7. Keep tests independent
To ensure testing robustness and simplify maintenance, test should never rely on other test nor should they depend on the ordering in which tests are executed.
8. Keep tests close to the class being tested
If the class to test is Foo the test class should be called FooTest and kept in the same package (directory) as Foo. The build environment must be configured so that the test classes doesn't make its way into production code.
9. Name tests properly
Make sure each test method test one distinct feature of the class being tested and name the test methods accordingly. The typical naming convention is test[what] such as testSaveAs(), testAddListener(), testDeleteProperty() etc.
10. Test public API
Unit testing can be defined as testing classes through their public API. Some testing tools makes it possible to test private content of a class, but this should be avoided. If there is private content that seems to need explicit testing, consider refactoring it into public methods in utility classes instead. But do this to improve the general design, not to aid testing.
11. Test the trivial cases too
It is sometimes recommended that all non-trivial cases should be tested and that trivial methods like simple setters and getters can be omitted. However, there are several reasons why trivial cases should be tested too: trivial is hard to define. It may mean different things to different people.
From a black-box perspective there is no way to know which part of the code is trivial.
The trivial cases can contain errors too, often as a result of copy-paste operations:
private double weight_;
private double x_, y_;
public void setWeight(int weight)
{
weight = weight_; // error
}
public double getX()
{
return x_;
}
public double getY()
{
return x_; // error
}
The recommendation is therefore to test everything. The trivial cases are simple to test after all.
12. Focus on execution coverage first
Differentiate between execution coverage and actual test coverage. The initial goal of a test should be to ensure high execution coverage. This will ensure that the code is actually executed on some input parameters. When this is in place, the test coverage should be improved. Note that actual test coverage cannot be easily measured (and is always close to 0% anyway). Consider the following public method:
void setLength(double length);
By calling setLength(1.0) you might get 100% execution coverage. 100% actual test coverage can only be reached by calling the method for every possible double value and ensure that the class behaves correctly for all of them. Surly an impossible task.
13. Cover boundary cases
Make sure the parameter boundary cases are covered. For numbers, test negatives, 0, positive, smallest, largest, NaN, infinity, etc. For strings test empty string, single character string, non-ASCII string, multi-MB strings etc. For dates, test January 1, February 29, December 31 etc. The class being tested will suggest the boundary cases for each specific case. The point is to make sure as many as possible of these are tested properly as these cases are the prime candidates for errors.
14. Provide a random generator
When the boundary cases are covered, a simple way to improve test coverage further is to generate random parameters so that the tests can be executed with different input every time. To achieve this, provide a simple utility class that generates random values of the base types like doubles, integers, strings, dates etc. The generator should produce values from the entire domain of each type.
If the tests are fast, consider running them inside loops to cover as many possible input combinations as possible. The following example verifies that converting twice between little endian and big endian representations gives back the original value. As the test is fast, it is executed on one million different values each time.
void testByteSwapper()
{
for (int i = 0; i < 1000000; i++) {
double v0 = Random.getDouble();
double v1 = ByteSwapper.swap(v0);
double v2 = ByteSwapper.swap(v1);
assertEquals(v0, v2);
}
}
The test cannot prove that the byte swapping logic is correct, but if it fails, it can prove that is wrong.
15. Test each feature once
When being in testing mode it is sometimes tempting to assert on "everything" in every test. This should be avoided as it makes maintenance harder. Test exactly the feature indicated by the name of the test method. As for ordinary code, it is a goal to keep the amount of test code as low as possible.
16. Use explicit asserts
Always prefer assertEquals(a, b) to assertTrue(a == b) (and likewise) as the former will give more useful information of what exactly is wrong if the test fails. This is in particular important in combination with random value parameters as described above when the input values are not known in advance.
17. Provide negative tests
Negative tests intentionally misuse the code and verify robustness and appropriate error handling. Consider this method that throws an exception if called with a negative parameter:
void setLength(double length) throws IllegalArgumentException;
Testing correct behavior for this particular case can be done by:
try {
setLength(-1.0);
fail();
}
catch (IllegalArgumentException exception) {
}
18. Design code with testing in mind
Writing and maintaining unit tests are costly, and minimizing public API and reducing cyclomatic complexity in the code are ways to reduce this cost and make high-coverage test code faster to write and easier to maintain. Some suggestions:
Make class members immutable by establish state at constructor time. This reduce the need of setter methods.
Restrict use of excessive inheritance and virtual public methods.
Reduce the public API by utilizing friend classes (C++) or package scope (Java).
Avoid unnecessary branching.
Keep as little code as possible inside branches.
Restrict the use of convenience methods. When wearing the testers hat, you are supposed to know nothing about the implementation of a particular method, so every method must be tested equally well. Consider the following trivial example:
public void scale(double x0, double y0, double scaleFactor)
{
// scaling logic
}
public void scale(double x0, double y0)
{
scale(x0, y0, 1.0);
}
Leaving out the latter simplifies testing on the expense of slightly extra workload for the client code.
19. Don't connect to predefined external resources
Unit tests should be written without explicit knowledge of the environment context in which they are executed so that they can be run anywhere at anytime. In order to provide required resources for a test these resources should instead be made available by the test itself. Consider for instance a class for reading files of a certain type. Instead of picking a sample file from a predefined location, put the file content inside the test, write it to a temporary file in the test setup process and delete the file when the test is done.
20. Know the cost of testing
Not writing unit tests is costly, but writing unit tests is costly too. There is a trade-off between the two, and the typical industry standard for execution coverage is about 80%. The typical areas where it is hard to get full execution coverage is on error and exception handling dealing with external resources. Simulating a database breakdown in the middle of a transaction is possible, but might prove too costly compared to just trusting the exception handling code at this point.
21. Prioritize testing
Unit testing is a typical bottom-up process, and if there is not enough resources to test all parts of a system priority should be put on the lower levels first.
22. Prepare test code for failures
Consider the simple example:
Handle handle = manager.getHandle();
assertNotNull(handle);
String handleName = handle.getName();
assertEquals(handleName, "handle-01");
If the first assertion is false, the code crashes in the subsequent statement and none of the remaining tests will be executed. Always prepare for test failure so that the failure of single test doesn't bring down the entire test suite execution. In general rewrite as follows:
Handle handle = manager.getHandle();
assertNotNull(handle);
if (handle == null) return;
String handleName = handle.getName();
assertEquals(handleName, "handle-01");
23. Write tests to reproduce bugs
When a bug is reported, write a test to reproduce the bug (i.e. a failing test) and use this test as a success criteria when fixing the code.
