Brian W Kernighan and Christopher J Van Wyk have written a set of small benchmarks to compare the perforance of several scripting languages, including Scheme. They benchmarked two implementations of Scheme: the pure interpreter of MIT Scheme, and the byte code interpreter of VSCM. The MIT Scheme interpreter was quite a bit slower than most of the other implementations that they tested; it was also unable to run some of the benchmarks for lack of heap space. VSCM ran somewhat faster.
One of the main conclusions drawn by Kernighan and Van Wyk is that compilation is faster than interpretation. They did not test any compilers for Scheme, however. For this note I duplicated their experiments for Scheme, on a single machine, using three different compilers and four different interpreters.
The main conclusion to be drawn from these results is that the major implementations of Scheme span an extremely wide range of performance. On several of these benchmarks, however, the speed of an implementation is determined primarily by its standard procedures, which may be written in some language other than Scheme.
Several of Kernighan and Van Wyk's other conclusions must be reconsidered in light of these results.
These benchmarks have already been described by Kernighan and Van Wyk, so I will limit myself to a few additional notes on the Scheme and C code.
The benchmarked systems:
-heap 2000
to allow its heap to expand to 2 megawords.
This interpreter is written in C.
vscm,
which was tested by Kernighan and Van Wyk.
The ,bench option gave the compiler permission
to generate inline code for standard procedures.
This byte code interpeter is written in C.
block and standard-bindings
compiler options allow inline code for known procedures.
(not safe)
and fixnum declarations added.
(optimize-level 2), which allows inline code
for known procedures.
(optimize-level 3), which allows unsafe code
to be generated.
Except for MIT Scheme, the numbers shown below represent the time in seconds for one run on a Sparcstation 10 running SunOS, with no other users. These times are unusually repeatable, perhaps because the benchmarks are so simple. The Scheme48 timings represent elapsed time; the others represent user+system cpu time, mainly because that is what Kernighan and Van Wyk measured.
I have not yet found a machine on which both MIT Scheme and Chez Scheme are installed properly. I therefore ran MIT Scheme on an Ultrasparc 1, which is a considerably faster machine, and scaled its timings by assuming that, for each benchmark, the performance of MIT Scheme relative to Larceny would be the same on both the Ultrasparc and the Sparcstation 10. Since the MIT Scheme interpreter is the slowest system tested on all but two benchmarks, this shouldn't matter too much.
The bar graphs show relative performance. Longer is better.
sumloop 1000000
Java 2.30 **
interp1 109.
interp2 37.7
interp3 14.6
interp4 9.11
Gambit-C 15.4
Gambit3 1.00 ****
Larceny 0.80 ******
no-wb 0.48 *********
Chez2 0.91 *****
Chez3 0.83 *****
gcc 0.28 ****************
gcc -O4 0.08 *******************************************************
ack(3,8)
Java 20.2 ****
interp1 236.
interp2 148. *
interp3 36.8 **
interp4 39.1 **
Gambit-C 41.7 **
Gambit3 3.33 ************************
Larceny 1.48 *******************************************************
Chez2 2.29 ************************************
Chez3 2.22 *************************************
gcc 13.5 ******
gcc -O4 11.6 *******
array1 200000
Java 1.60 *******
interp1 46.8
interp2 15.8 *
interp3 7.46 *
interp4 4.34 **
Gambit-C 6.78 **
Gambit3 0.30 ***********************************
Larceny 0.68 ***************
Larceny3 0.43 ************************
Chez2 0.63 *****************
Chez3 0.47 **********************
gcc 0.20 ****************************************************
gcc -O4 0.19 *******************************************************
string 500000
interp1 3.00 *******************************************************
interp2 16.7 **********
interp3 25.7 ******
interp4 430.
Gambit-C 18.2 *********
Gambit3 16.10 **********
Larceny 25.7 ******
Chez2 17.1 **********
gcc 4.00 *****************************************
gcc -O4 4.00 *****************************************
cat
Java 374. *
interp1 1090
interp2 244. *
interp3 211. *
interp4 44.6 *****
Gambit-C 106. **
Gambit3 104. **
Larceny 101. **
Chez2 13.3 ***************
Chez3 6.88 ******************************
gcc 5.70 ************************************
gcc -O4 3.70 *******************************************************
wc
Java 229.
interp1 1050
interp2 451.
interp3 307.
interp4 92.8 *
Gambit-C 82.6 *
Gambit3 53.9 **
Larceny 56.6 **
Chez2 10.7 **********
Chez3 5.83 ******************
gcc 4.10 *************************
gcc -O4 1.90 *******************************************************
tail
Java 502.
interp1 496.
interp2 342. *
interp3 233. *
interp4 275. *
Gambit-C 107. **
Gambit3 117. **
Larceny 94.7 **
Chez2 18.8 ***********
Chez3 14.9 **************
gcc 3.90 *******************************************************
gcc -O4 3.90 *******************************************************
sum1
Java 129. ********
interp1 560. **
interp2 60.7 ******************
interp3 66.5 ****************
interp4 933. *
Gambit-C 51.9 *********************
Larceny 58.1 *******************
Chez2 27.3 ****************************************
gcc 19.9 *******************************************************
gcc -O4 19.8 *******************************************************