ONLamp.com
oreilly.comSafari Books Online.Conferences.

advertisement


Testing C with Libtap
Pages: 1, 2, 3

Proper Planning Prevents Poor Performance

What happens when you get the number of tests to run wrong? Try it (why yes, this is a contrived test):



plan_test(3);
ok(1, "true");
ok(!0, "!false == true");
return exit_status();

When you compile and run this program, you should get the following output. (I've shown the command line I typed to run the test as well, and I am checking to see if the program is exiting cleanly.)

% ./plan-too-many && echo success || echo failure
1..3
ok 1 - 1 is true
ok 2 - !0 is true
# Looks like you planned 3 tests but only ran 2.
failure

Libtap issued a diagnostic to say that you planned too many tests. In this case the diagnostic was correct, but you could also get this message if some spectacular failure makes the program exit cleanly before all the tests have run. In addition, it printed failure--meaning that the test program returned nonzero, indicating a failure. This is the handiwork of the exit_status() function, and makes it possible to simply run the test and ignore the output to get a ok or not ok (although this would not catch the spectacular failing case here).

Responsibility of Tests

Unit tests have several responsibilities. Above all, they should detect when something has gone wrong. In order to help you quickly and easily find the problem, they should also attempt to give good clues as to what has gone wrong and where the problem might be.

I once needed to test an AI for an Othello (aka Reversi) game. To make sure that the AI picked the expected move at each point, I compared each state in a run with the states of a known good run. Simply doing square-by-square comparisons over the entire grid at each state would have detected errors, but it would have made it hard to see in which situation the errors occurred. Actually creating (not to mention updating) the tests would have been a nightmare also.

Instead I chose to code the test in two parts. The first was a very simple C program to search for and apply moves and print out the current state at each step. This part resembled:

state = init_othello_state(); 
do { 
   print_othello_state(state); 
   putchar('\n'); 
} while (state = ai_move(state));

I then wrote a simple Perl program to read the output of the first and compare each state against a list of canned states from a known good run. That part was also very simple:

local $/ = ''; 
my @states = qx( ./reversi ) 
  or die 'running reversi failed'; 
  
is shift @states, $_ while <DATA>; 
  
__DATA__ 
...... 
...... 
..ox.. 
..xo.. 
...... 
...... - x to move 

best branch: 6 (ply 3 search; 48 states visited) 
...... 
...... 
..ox.. 
..xx.. 
...x.. 
...... - o to move

best branch: 1 (ply 3 search; 45 states visited)
......
......
..ox..
..ox..
..ox..
...... - x to move

[etc]

Notice that this last test did not actually use libtap at all. Instead, it had a C component and a Perl component. Using Inline::C, you could improve on that and put both the C and Perl components in the same file. Note that using two separate parts has the benefit that if Perl is not available the C component can still run, although you must manually compare its results with the canned run in the Perl component.

Availability

Both Test::More and Test::Harness are part of the core in recent Perl distributions. Libtap is unfortunately not (yet) a commonly installed library. However, it has a liberal license and consists of only two source files, so if you want to make sure that your users can run your test suite, you can bundle libtap with your software.

Conclusion

This article explores various ways of using Perl to help testing your C code. First, it briefly examines the Test Anything Protocol, the heart of Perl's well-established test framework. It then shows how libtap can help you produce TAP output from C programs. It rounds off by using Perl to test programs by comparing their output with known good reference output.

Hopefully, this should give you some ideas for improving your own testing.

Links & Resources

Stig Brautaset works at Fotango, where he gets to do interesting things in Perl and eat a lot of fruit.


Return to ONLamp.com.



Sponsored by: