oreilly.comSafari Books Online.Conferences.


Testing C with Libtap

by Stig Brautaset

Libtap is a library for testing C code. It implements the Test Anything Protocol, which is emerging from Perl's established test framework.

Design for Today, Code for Tomorrow

One of the ideas behind Extreme Programming (XP) is to "design for today, code for tomorrow." Rather than making your design cover all eventualities, you should write code that is simple to change should it become necessary.

Having a good regression test suite is a key part of this strategy. It lets you make modifications that change large parts of the internals with the confidence that you have not broken your API. A good test suite can also be a way to document how you intend people to use your software.

Having worked where people thought that writing tests was a waste of time, I can't tell you how much time I wasted trying to fix bugs that had emerged as a result of bugs being fixed or new features added. If we'd had a proper regression test suite, we could have found those immediately, and I would have lots of extra time to write new features. Taking the time to produce good tests (and actually running them) actually ends up saving a lot of time, not wasting it.

Introducing the Test Anything Protocol

Perl distributions normally ship with a test suite written using Test::Simple, Test::More, or the older (and now best avoided) Test module. These modules contain functions to produce plain-text output according to the Test Anything Protocol (TAP) based on the success or failure of the tests. The output from a TAP test program might look something like this:

ok 1 - the WHAM is overheating
ok 2 - the overheating is detected 
not ok 3 - the WHAM is cooled 
not ok 4 - Eddie is saved by the skin of his teeth

Related Reading

C in a Nutshell
By Peter Prinz, Tony Crawford

The 1..4 line indicates that the file expects to run four tests. This can help you detect a situation where your test script dies before it has run all the intended tests. The remaining lines consist of a test success flag, ok or not ok, and a test number, followed by the test's "name" or short description. Obviously, the second and third lines indicate a successful test, while the last two indicate test failures.

Perl modules usually invoke the tests either by running the prove program or by invoking make test or ./Build test (depending on whether you're using ExtUtils::MakeMaker or Module::Build). All three approaches use the Test::Harness module to analyze the output from TAP tests. If all else fails, you can also run the tests directly and inspect the output manually.

If Test::Harness is given a list of tests programs to run, it will run each one individually and summarize the result. Tests can run in quiet and verbose modes. In the quiet mode, the harness prints only the name of the test script (or scripts) and a result summary. Verbose mode prints the test "name" for each individual test.

Besides Perl, helper libraries for producing TAP output are available for many languages including C, Javascript, and PHP (see the Links & Resources section).

Suppose that you want to write tests for the module Foo, which provides the mul(), mul_str(), and answer() functions. The first two perform multiplication of numbers and strings, while the third provides the answer to life, the universe, and everything. Here is an extremely simple Perl test script for this module:

use Test::More tests => 3; 
use Foo;
ok(mul(2,3) == 6, '2 x 3 == 6'); 
is(mul_str('two', 'three'), 'six', 'expected: six'); 
ok(answer() == 42, 'got the answer to everything');

The tests => 3 part tells Test::More how many tests it intends to run (referred to as planning). Doing this allows the framework to detect whether you exit the test script without actually running all the tests. It is possible to write test scripts without planning, but many people consider this a bad habit.

On to the C Testing

Hey! Isn't this article supposed to be about testing C? It is. Libtap is a C implementation of the Test Anything Protocol. It is to C what Test::More is to Perl, though using it doesn't tie you into using Perl. However, for convenience you probably want to use the prove program to interpret the output of your tests.

Libtap implements a convenient way for your C and C++ programs to speak the TAP protocol. This allows you to easily declare how many tests you intend to run, skip tests (some apply only on specific operating systems, for example), and mark tests for unimplemented features as TODO. It also provides the convenient exit_status() function for indicating whether any of the tests failed through the program's return code.

How would you would write the test for the Foo module in C, using libtap? The #include <foo.h> line is analogous to the use Foo; of the Perl version. However, as this is C, you also need to link with the libfoo library (assuming this implements the functions declared in foo.h).

For this test, I will show the full source of the test program, including any #include lines; I will show only shorter fragments below. Notice again the difference in the number passed to the plan_tests() function and the number of actual tests that actually run:

#include <tap.h>
#include <string.h>
#include <foo.h>
int main(void) {
  ok1(mul(2, 3) == 6); 
  ok(!strcmp(mul_str("two", "three"), "six"), "expected: 6"); 
  ok(answer() == 42, "got the answer to everything"); 
  return exit_status();

The exit_status() function returns 0 if the correct number of tests ran and if they all succeeded; it returns nonzero otherwise. In the Perl version the test framework makes magic happen behind the scenes so that you don't have to twiddle the exit status by hand.

One notable difference between the Perl version and the C version is the ok1() macro, a wrapper around the ok() call. Instead of having to call ok() with a test condition as the first parameter and diagnostic as the second (and any subsequent) parameter, this macro stringifies its argument and uses that for the diagnostic message. This can be very convenient for simple tests.

Both the Perl and C tests above, when run, print something along the lines of:

not ok 1 - mul(2, 3) == 6 
#     Failed test (basic.c:main() at line 12) 
ok 2 - expected: 6 
ok 3 - got the answer to everything

The line starting with # is a diagnostic message; libtap prints these occasionally to help you find which test is failing. In this case, it identifies the line in the test file that contained the failing test.

Pages: 1, 2, 3

Next Pagearrow

Sponsored by: