Fri 16 Feb 2007


I've been writing and running a lot of tests in the past few days, in two very different environments: Java + JUnit 4 + Eclipse, and Allegro Common Lisp.

In Eclipse I've followed convention (is there any other way in Java?!): I have tests defined using JUnit's annotations, and a Run target set up to run all of the tests in the project. For those who haven't seen JUnit at work, hitting "Run" shows me the green bar if my tests pass, and a red bar if they fail. It's as addictive as you might imagine: positive feedback every time your tests work.

It's still far from perfect: writing tests for uncovered areas, particularly testing textual output, is as much of a chore as ever, and there's no corresponding feedback about coverage. You can get a big green bar with just one test, and the positive feedback is the same: you feel a false sense of quality. Furthermore, with areas like text output it's very tempting to use the log window to judge by eye whether the run succeeded: this rather defeats the point of automated testing. I don't know how to make testing textual output easier, but it's a gap in the testing world.

My other testing has been of some of my Lisp code. I should be using LiFT, but for what I wanted to do it was easy enough to write a few macros and convenience functions. The results of running my tests are pretty-printed, with error messages on failure. A few minutes of work later, and I can (run-test-suite) and see the output in my REPL. Because this is Lisp, I simply opt to generate s-expressions as output and compare those; I can test the serializer later, without having to test it at the same time as the logic.

Which is better? Eclipse gives me a green bar in a shiny IDE, but my ability to customize the test framework itself is negligible. If I want to add some coverage information, I have to go install Quilt and hope that it shows up how I want it. Lisp, on the other hand, I use in a terminal, but it lets me roll my own domain-specific testing framework in a few minutes, and I can extend it or replace it when necessary.

Both situations are compelling: I can see where test-driven development advocates are coming from.

Posted at 2007-02-16 10:46:06 by RichardLink to Testing