The runbench.tcl application provides the overall management for the execution of benchmarks. If no specific benchmark files are provided on the command line itself the application will use all directory containing the application itself. If files are provided all files matching the glob pattern *tk* are assumed to be benchmarks for the Tk toolkit.
The following options are recognized by the application:
-help
Causes the application to print a list of the recognized option plus short explanations of their meaning
-deltaratio
Delta range for wiki highlight (default: 0.05)
-iterationscount
The value of this option is a positive integer number and declares how often the body of benchmarks should be executed to get accurate timing results. This is a global maximum value which can be overidden by the individual benchmarks.
-minversionversion
Declares the minimum version of tcl interpreters the application is allowed to use in the benchmarking.
-maxversionversion
Declares the maximum version of tcl interpreters the application is allowed to use in the benchmarking.
-matchpattern
Restricts the benchmarking to benchmarks whose description matches the glob pattern specified as the value of this option. If both -match and -rmatch are specified benchmarks have to fulfill both conditions to be executed.
-rmatchregexp
Restricts the benchmarking to benchmarks whose description matches the regexp pattern specified as the value of this option. If both -match and -rmatch are specified benchmarks have to fulfill both conditions to be executed.
-normalizeversion
Takes a version number as value and causes the application to normalize the timing values to the value of the given version.
-notcl
Forces the application to skip all tcl related benchmarks. Takes no additional value.
-notk
Forces the application to skip all tk related benchmarks. Takes no additional value.
-outputstyle
Declares the style to use when generating the output for the benchmarks. Accepts text, csv, list and wiki. text is the default.
-threadscount
Number of threads to use for running benchmarks (where possible).
-pathslist
Accepts a list of paths to search for tcl interpreters. If no paths are specified the paths in the environment variable PATH are used.
-throwerrors
Causes errors in benchmark files to be propagated out.
-verbose
Activates the output of interim status info. Takes no additional value
Misc
Kroc 2004-06-23: I compared two recents 8.4.6 Tclkit versions, a standard and an upx one, and an older 8.4.0. The bench were launched on three differents computers, running nothing else.
Laptop with Crusoe 600 MHz processor, 192 Mb RAM running linux 2.6.6 :
TCL tests : TK tests : Total
Diff. :
Tclkit 8.4.0 -> 08 min 48 sec 00 min 41 sec 10 min 12 sec + 24 %
Tclkit 8.4.6 -> 07 min 33 sec 00 min 40 sec 08 min 13 sec + 0 %
Tclkit 8.4.6 upx -> 09 min 03 sec 02 min 36 sec 11 min 39 sec + 42 %
PC with Celeron 600 MHz processor, 128 Mb RAM running Windows 98 SE :
TCL tests : TK tests : Total
Diff. :
Tclkit 8.4.0 -> 08 min 01 sec 01 min 35 sec 09 min 36 sec + 6 %
Tclkit 8.4.6 -> 07 min 32 sec 01 min 31 sec 09 min 03 sec + 0 %
Tclkit 8.4.6 upx -> 10 min 06 sec 05 min 32 sec 15 min 38 sec + 72 %
PC with Celeron 2400 MHz processor, 512 Mb RAM running Windows XP Family :
TCL tests : TK tests : Total
Diff. :
Tclkit 8.4.0 -> 06 min 16 sec 01 min 03 sec 07 min 19 sec + 33 %
Tclkit 8.4.6 -> 04 min 31 sec 01 min 00 sec 05 min 31 sec + 0 %
Tclkit 8.4.6 upx -> 05 min 05 sec 02 min 04 sec 07 min 09 sec + 30 %
First thing: a recent Tclkit version is always fastest than an older one, whatever platform and computer running it. So we could say TCT do their job ;^) jcw explained me upx results are bad because it is slower on startup. Anyway, the smaller the computer is, the slower upx was compared to standard Tclkit.
ferrieux 2010-10-20: Measuring perf effects in TEBC et al, I found it necessary to do some extra work to _denoise_ tclbench output:
Run it 10 times, and look at min, med, avg, max for each value. Verify that (med-min)/min stays small, so that there's strong accumulation on the min.
Vary spacing between functions in libtcl.so. A 16-byte "void foo(void){}" at the beginning of regcomp.c has a large impact on the recursive variant of STR.match.
Use the new autoscale feature, so that no test lasts below 0.1s. There may be undershoot due to clock granularity on the first measurement, so I added an extra step of re-evaluation of iteration count (with margin) in libbench.tcl:
if {$res * $iter < 100000} {
set iter [expr {int(200000.0 / $res)}]
set code [catch {uplevel #0 [list time $opts(-body) $iter]} res]
set res [lindex $res 0]
}
The effect of these 3 tricks is a much greater stability. Remaining noise no longer occurs from one run to another, but rather exposes sensitivity to function offsets, instruction cache associativity, etc. Here again, other "spacers" sprinkled over the library have drastic effects. However, optimizing such things across processors and compilers is a daunting task. The bottom line is that tclbench can be used as a reliable speed comparison tool, but that an observed slowdown or speedup may be a very indirect consequence of code changes.
According to MS, Tcl chatroom, 2015-10-26, wha is not good about Tclbench is the way it invokes test scripts, and also that for the micro-benchmarks the signal is drowned in the overhead noise.