CppUnitLite2 v1.1

http://www.gamesfromwithin.com/articles/0512/000103.html

CppUnitLite2_1_1.tar.gz

About a year ago, I wrote an article comparing unit test frameworks
that went to become one of the most popular in this site. The article
concluded with the recommendation of one of the less well known
frameworks: CxxTest.

Shortly after writing that article, it came time to roll out
test-driven development at work. Interestingly, once I weighted in
all the requirements, I ended up not following my own advice.
Instead, I decided to go with a customized version of CppUnitLite,
which was listed in the article as the second choice. The idea of
an extremely simple unit testing framework was too attractive to
pass up in an environment where we have to support a large variety
of environments and target platforms.

At this point, we have been using CppUnitLite2 for a year at
High Moon Studios to do TDD on Windows, Xbox 360, and
some PS3 (we even have a stripped-down version that runs on
a PS3 SPU). It has been used to unit test libraries of an
engine, pipeline tools, GUI applications, and production game code.

CppUnitLite2 was originally released along with my article a
year ago. It was a heavily modified version of the original
CppUnitLite by Michael Feathers. This version introduces the
changes we made at work as different issues came up and we needed
more functionality from the framework. In particular, Jim Tilander
and Charles Nicholson were responsible for a lot of those changes.
By now, I doubt there's a single line left from the original code. Additionally,
CppUnitLite2 itself has a set of tests using itself in an interestingly
recursive way.

What's new

Fixtures created and destroyed around each test.

I can't believe that the original
version of CppUnitLite2 didn't have this feature. It was the
biggest glaring problem with the framework (I even pointed it out
in the article). Each fixture used to be created at static
initialization time, which, apart from being a really bad idea,
caused a lot of objects to be created and live in memory at the
same time. Considering that some of those objects would
initialize/shutdown some important game systems, we clearly needed
to manage their lifetime a bit better.

The test macro was changed to
register a temporary test with the TestRegistry, which in turn will
create the fixture at execution time. Now fixtures and tests are
created just before each test, and destroyed right after, just as
you would expect from any sane framework.

For those masochists out there, here's the ugly macro for a test
with fixture in all its glory. Notice how the TestRegistrar
registers the dummy test class that we created as part of the
macro.

#define TEST_F(fixture, test_name)                                                      
    struct fixture##test_name : public fixture {                                        
        fixture##test_name(const char * name_) : m_name(name_) {}                       
        void test_name(TestResult& result_);                                            
        const char * m_name;                                                            
    };                                                                                  
    class fixture##test_name##Test : public Test                                        
    {                                                                                   
        public:                                                                         
            fixture##test_name##Test ()                                                 
                : Test (#test_name "Test", __FILE__, __LINE__) {}                       
        protected:                                                                      
            virtual void RunTest (TestResult& result_);                                 
    } fixture##test_name##Instance;                                                     
    TestRegistrar fixture##test_name##_registrar (&fixture##test_name##Instance);       
    void fixture##test_name##Test::RunTest (TestResult& result_) {                      
        fixture##test_name mt(m_name);                                                  
        mt.test_name(result_);                                                          
    }                                                                                   
    void fixture ## test_name::test_name(TestResult& result_)

Improved check statements

Another one of the major weaknesses of CppUnitLite as it stood
was the check statements. You needed to have a custom check macro
for each data type you cared to check. The generic CHECK_EQUAL
macro will now work on any data type as long as it can be compared
with operator == and output to a stream.

This version of CppUnitLite2 also includes some special macros
to check arrays with one and two dimensions. Again, this works on
any data type that supports operator == and operator[]. It came it
very handy for checking equality of vectors and matrices in a game
engine. For example, the following code checks that two matrices
are equal:

CHECK_ARRAY2D_CLOSE (m1, m2, 4, 3, kEpsilon);

Ironically, in some heavily restricted environments (like the
PS3 cell processors), we can't use streams, so in that case we're
forced to go back to the old specialized check macros.

Invariant statements in checks

Another really annoying feature of the previous version of
CppUnitLite2 was that the code in a check statement was called
multiple times, which could cause strange side effects.

Consider the following code:

CHECK_EQUAL (FunctionWithSideEffects1(), FunctionWithSideEffects1());

The way the check statement was expanded out, both those
functions could be evaluated multiple times. If the functions had
side effects, they would be executed several times, causing
unexpected results (for example, the check could fail but the
printed result could be different). The fix wasn't trivial because
we couldn't save off the value of each statement because they could
be of any type and we have no way of finding the type from within
the macro.

The check macro now expands out to a templated function, which
is able to evaluate both statements only once and work with the
saved value to avoid any surprising side effects.

Optional fixture initialization

As we continued hammering on the unit test framework for all our
development, one clear pattern appeared that wasn't handled by the
framework: Being able to create fixtures with parameters defined in
the test. For example, the fixture might do ten different steps,
but we want to set a particular value in one of those steps, and by
the time control reaches the test code itself it's too late.

We started getting around that by creating different fixtures,
and then by inhereting from other fixtures and changing some
parameters. Both of those solutions were less than ideal (lots of
code duplication and added complexity). Things got out of hand when
we made the fixtures a templated class on a value, and created them
by specifying the value as part of the texture name. Now it was a
pain to debug and things were even less clear.

So we finally implemented it natively in CppUnitLite2 as
fixtures with initialization parameters. We introduced a new macro,
TEST_FP (test with fixture and parameters), which takes any
parameters you want to pass to the fixture constuctor.

Now you can declare tests like this:

TEST_FP (VectorFixture, VectorFixture(10), MyTest)

{

}

We should probably simplify that to avoid repeating the fixture
name, but that works fine for now.

Include cleanup

The original version of CppUnitLite was pretty casual about what
headers it included. Unfortunately, that means that all test code
were automatically exposed to a variety of standard headers,
whether you wanted it or not. This version is a lot more strict,
and the only header that will get pulled in from using it will be
iosfwd, which is even a fairly lightweight one.

Memory allocations

We have very strict memory allocation wrappers around our tests,
and if any memory leaks, it is reported as a failed test. Since
checking for memory allocations is very platform specific, it is
left out of the framework (although it could be added pretty easily
in the platform-specific sections). In any case, some of the
static-initialization time allocations were confusing the memory
tracker, so we removed them completely. All it means is that we use
an array to register tests instead of a std::vector and we cap the
maximum number of tests to a fixed amount. If you need more than
the default 5000 tests, go right ahead and add a few zeros.

New license

Previous versions of CppUnitLite were not released with an
explicit license. To make things clear, I explicitly added the
license text for the MIT license with an added restriction.
The MIT license allows you to use the code in any project,
commercial or otherwise, without any restrictions.
The added restriction to the license prohibits its use in any
military applications or projects with any military funding.

Ratings

How does this release of the CppUnitLite2 compare with respect
to my initial set of features I used to rate the different unit
tests frameworks?

• Minimal amount of work needed to add new tests. Nothing
  has changed there. Still passes with flying colors.


• Easy to modify and port. That has always been the strong
  suite of this framework.


• Supports setup/teardown steps (fixtures). Much improved
  now by creating them correctly around each test.


• Handles exceptions and crashes well. No changes there.
  Still one of the best I've seen.


• Good assert functionality. The check macros have been
  much improved to match the best frameworks out there.


• Supports different outputs. Yup. Still there.


• Supports suites. It still doesn't. It just shows that I
  really haven't needed this feature yet.

Future work

So, what's next for CppUnitLite2? It's really quite usable right
now, but I could see a few changes happening in the next year, and
maybe a few others will come up and surprise me.

Test registration is based on static initialization of global
variables, which is not very reliable across compliers. For
example, if you try to put the tests in a static library, MSVC will
strip out the registration code leaving you with no tests. To get
around that, we could introduce a custom build step in a scripting
language that would generate a simple file with explicit
registration of tests. This could also be used to collect tests
from many different libraries into a single executable.

A couple of times it would have been convenient to have
parameterized tests (I think that's the correct term for it).
Basically, to have tests that could run with different data.
Something like this:

TEST_PARAM(MyCustomTest, int a, int b)
{
    CHECK (something);
    CHECK(something else);
    CHECK(yadda yadda);
}

TEST(MyTest)
{
    CALL_TEST(MyCustomTest, 10, 5);
}
TEST(MyTest)
{
    CALL_TEST(MyCustomTest, 12, 7);
}

The key point is that a test fails in the parameterized test, it
should report the error correctly indicating where it came
from.

This is actually pretty easy to implement with a couple of
macros. We simply haven't needed it more than a couple of times, so
we haven't bothered yet. But I imagine next time we run up against
this we'll bite the bullet and implement it.

Wrap-up

Charles had so much fun working with this framework, he decided
to write his own from scratch. He made it available on his web
site, so go check it out.

Thanks to High Moon Studios for being open and letting us
release the changes we made and share them back with everybody. If
you end up using the framework and you make any changes, drop me an
email and I'll add them for the next release.