On benchmarking

So if this is essentially what the benchmark boils down to, are we interested? Will it come as a surprise to anyone that assembly and near-assembly languages can make quick work of it? Or that some others have trouble generating really optimal code? In the latter case, should we care?

…

Oh, and one other thing, since this is nominally a blog (like I ever post!) about lisp. It would have taken the champion c code about an hour and a half to generate the improved image, and 3 hours at the resolution I originally wanted to use. I wasn't going to wait around for that so I wrote some lisp with a better approach. Even counting the time to write the code, I was still ahead as it took only 20 min to run and an hour or so to write.

Take benchmarks with a pinch of salt.

I'm engaged in a thread on the Clojure mailing list right now: one fellow has observed large memory consumption for a trivial program with his particular OS + JVM combination. I took issue with his sky-is-falling proclamation that the JVM is awful and worthless, particularly because I don't even see the same results (and neither does he on Windows).

The only real evaluation of a technology is how well it solves your problem in your situation. This is the root of why scripting languages are (rightly) popular, and also why there is still a place for C, Forth, and everything else down the abstraction stack. Sometimes the tradeoff is worthwhile. (I've even been in the position where writing a program in a low-level language, compiling, debugging, and running it took less time than the equivalent Ruby. C has a place for run-once scripts, too.)

Posted at 2009-06-08 21:24:00 by Richard Newman • Link to On benchmarking