Tclbench is a set of tools to compare performances between different [Tcl] and/or [Tk] versions.

**Acquiring**
Tclbench is a [cvs] module of the [tcllib] [sourceforge] project.

======
cvs -z3 -d:pserver:anonymous@tcllib.cvs.sourceforge.net:/cvsroot/tcllib co -P tclbench
======

**Synopsis**

    :   '''runbench.tcl''' ?''-option value ...''? ?''file ...''?

**Description**

The '''runbench.tcl''' application  provides the overall management for the execution of benchmarks. If no specific benchmark ''file''s are provided on the command line itself the  application  will  use  all directory containing the application itself. If files are provided all files matching the glob pattern ''*tk*'' are assumed to be benchmarks for the Tk toolkit.

The following options are recognized by the application:

   '''-help''':   Causes the application to print a list of the recognized option plus short  explanations of their meaning
   '''-delta''' ''ratio'':   Delta range for wiki highlight (default: 0.05) 
   '''-iterations''' ''count'':   The value of this option is a positive integer number and declares how often the body of benchmarks  should be executed to get accurate timing results. This is a global maximum value which can be overidden by the individual benchmarks.
   '''-minversion''' ''version'':   Declares the minimum version of tcl interpreters the application is allowed to use in the benchmarking.
   '''-maxversion''' ''version'':   Declares the maximum version of tcl interpreters the application is allowed to use in the benchmarking.
   '''-match''' ''pattern'':   Restricts the benchmarking to benchmarks whose description matches the [string match%|%glob%|%] pattern specified as the value  of this option. If both '''-match''' and '''-rmatch''' are specified benchmarks have to fulfill both conditions to be executed.
   '''-rmatch''' ''regexp'':   Restricts the benchmarking to benchmarks whose description matches the [regexp] pattern specified as the value of this option. If both '''-match''' and '''-rmatch''' are specified benchmarks have to fulfill both conditions to be executed.
   '''-normalize''' ''version'':   Takes a version number as value and causes the application to normalize the timing values to the value of the given version.
   '''-notcl''':   Forces the application to skip all tcl related benchmarks. Takes no additional value.
   '''-notk''':   Forces the application to skip all tk related benchmarks. Takes no additional value.
   '''-output''' ''style'':   Declares the style to use when generating the output for the benchmarks. Accepts '''text''', '''csv''', '''list''' and '''wiki'''. '''text''' is the default.
   '''-threads''' ''count'':   Number of threads to use for running benchmarks (where possible).
   '''-paths''' ''list'':   Accepts a list of paths to search for tcl interpreters. If no paths are specified the paths in the [env%|%environment variable%|%] PATH are used.
   '''-throwerrors''':   Causes errors in benchmark files to be propagated out.
   '''-verbose''':   Activates the output of interim status info. Takes no additional value

<<discussion>>
[Kroc] - 23/06/2004 : I compared two recents 8.4.6 [Tclkit] versions, a standard and an '''upx''' one, and an older 8.4.0. The
bench were launched on three differents computers, running nothing else. 

   * Laptop with Crusoe 600 MHz processor, 192 Mb RAM running linux 2.6.6 :

                          TCL tests :           TK tests :            Total :               Diff. :
 Tclkit 8.4.0     ->      08 min 48 sec         00 min 41 sec         10 min 12 sec         + 24 %
 Tclkit 8.4.6     ->      07 min 33 sec         00 min 40 sec         08 min 13 sec         +  0 %
 Tclkit 8.4.6 upx ->      09 min 03 sec         02 min 36 sec         11 min 39 sec         + 42 %

   * PC with Celeron 600 MHz processor, 128 Mb RAM running Windows 98 SE :

                          TCL tests :           TK tests :            Total :               Diff. :
 Tclkit 8.4.0     ->      08 min 01 sec         01 min 35 sec         09 min 36 sec         +  6 %
 Tclkit 8.4.6     ->      07 min 32 sec         01 min 31 sec         09 min 03 sec         +  0 %
 Tclkit 8.4.6 upx ->      10 min 06 sec         05 min 32 sec         15 min 38 sec         + 72 %
 
   * PC with Celeron 2400 MHz processor, 512 Mb RAM running Windows XP Family :

                          TCL tests :           TK tests :            Total :               Diff. :
 Tclkit 8.4.0     ->      06 min 16 sec         01 min 03 sec         07 min 19 sec         + 33 %
 Tclkit 8.4.6     ->      04 min 31 sec         01 min 00 sec         05 min 31 sec         +  0 %
 Tclkit 8.4.6 upx ->      05 min 05 sec         02 min 04 sec         07 min 09 sec         + 30 %

First thing: a recent [Tclkit] version is always fastest than an older one, whatever platform and computer 
running it. So we could say [TCT] do their job ;^) [jcw] explained me '''upx''' results are bad because it 
is slower on startup. Anyway, the smaller the computer is, the slower '''upx''' was compared to standard
[Tclkit].
----
[ferrieux] - 20/10/2010: measuring perf effects in TEBC et al, I found it necessary to do some extra work to _denoise_ tclbench output:

     * run it 10 times, and look at min, med, avg, max for each value. Verify that (med-min)/min stays small, so that there's strong accumulation on the min.

     * vary spacing between funtions in libtcl.so. A 16-byte "void foo(void){}" at the beginning of regcomp.c has a large impact on the recursive variant of STR.match.

     * use the new autoscale feature, so that no test lasts below 0.1s. There may be undershoot due to clock granularity on the first measurement, so I added an extra step of re-evaluation of iteration count (with margin) in libbench.tcl:

                    if {$res*$iter<100000} {
                        set iter [expr {int(200000.0/$res)}]
                        set code [catch {uplevel \#0 [list time $opts(-body) $iter]} res]
                        set res [lindex $res 0]
                    }

The effect of these 3 tricks is a much greater stability. Remaining noise no longer occurs from one run to another, but rather exposes sensitivity to function offsets, instruction cache associativity, etc. Here again, other "spacers" sprinkled over the library have drastic effects. However, optimizing such things across processors and compilers is a daunting task. The bottom line is that tclbench can be used as a reliable speed comparison tool, but that an observed slowdown or speedup may be a very indirect consequence of code changes.



<<categories>> Package | Performance