Tcl Static Prime

Tcl Static Prime (TSP) is an experimental compiler for the Tcl language that produces C or Java code, which is then compiled on-the-fly during execution of a Tcl program. TSP is a currently a work-in-progress; its performance varies greatly depending the Tcl commands that are currently compiled.

TSP compiles a typed subset of Tcl. Proc definitions and variables are typed by comment-based annotations. Native types supported are boolean, int (64-bit integers), double, string, and var (TclObjects for lists, dicts, etc.)

TSP language restrictions include restricting all arithmetic expressions (expr, if, while, etc) to using boolean, int, double, and string data types. Additionally, expressions may not include array references or nested commands. TSP also assumes that builtin Tcl commands are not re-defined, as builtin commands are compiled to C or Java, or the native command implementation is invoked directly, bypassing the Tcl interpreter.

TSP is written entirely in Tcl, with support libraries written in C and Java.

TSP is written by Tom Poindexter (TP).

TSP development is hosted at Github: https://github.com/tpoindex/tsp

Very, very, very preliminary results (July, 2015) Timing on my local machine, 3.0ghz i7.

md5 - This is a slightly modified version of the md5 function included in Tcllib, version 1.4.4. Various size message are tested. Times are in seconds. Timings for both C/Tcl and JTcl.

langbench - Larry McVoy's langbench, using slightly modified versions of the Tcl programs found in the benchmark. Several other notable changes to how the benchmark was prepared and run should be reviewed . Times are in seconds.

Notes - I've been working on this off and on for over two years, and thought it was about time TSP escaped out of the lab. TSP currently has about 9,700 lines of Tcl, 2,400 lines of C, and 750 lines of Java. All of my development has been on Linux, but since it's nearly all written in Tcl, there should be some degree of portability to OSX & Windows.

I haven't been able to find much time lately to work on it, so I'm hoping posting it now will spur some interest in others who might like to help out, test new ideas, work on code generation, etc. Interested in contributing? Start by forking the Github repository. I'm often logged into the Tcl Chatroom most weekdays for quick questions, or feel free to email.

aspect: this is fantastic! I made a little chart with ukaz using the test_fib example to get my feet wet. I stopped benchmarking each when a computation took more than 1s: interpreted got fib(31) = 1346269, compiled to fib(36) = 14930352.

Reading the implementation is going to be a lot of fun!

The benchmark script is hidden below this discussion tag, in the unlikely case it's useful to derive more:

package require tsp
package require ukaz

tsp::proc tsp_fib {n} {
    #tsp::procdef int -args int
    if {$n < 2} {
        return $n
    }
    #tsp::int n1 n2
    incr n -1
    set n1 [tsp_fib $n]
    incr n -1
    set n2 [tsp_fib $n]
    set n [expr {$n1 + $n2}]
    return $n
}

proc fib {n} {
    if {$n < 2} {
        return $n
    }
    incr n -1
    set n1 [fib $n]
    incr n -1
    set n2 [fib $n]
    set n [expr {$n1 + $n2}]
    return $n
}

proc elapsed {cmd} {
    set s -[clock microseconds]
    uplevel 1 $cmd
    incr s [clock microseconds]
    set s [expr {$s / 1e6}]
    return $s
}


set ctimes {0 0}
set itimes {0 0}

set maxtime 1   ;# stop measuring once an iteration takes this long
set cok 1
set iok 1

pack [ukaz::graph .l] -expand yes -fill both
set cg [.l plot $ctimes with lines color red title "Compiled fib"]
set ig [.l plot $itimes with lines color blue title "Interpreted fib"]
.l set xlabel "N"
.l set ylabel "Seconds"

for {set i 1} {$i < Inf} {incr i} {
    if {$cok} {
        set ctime [elapsed {set cr [tsp_fib $i]}]
        .l update $cg data [lappend ctimes $i $ctime]
        set cok [expr {$ctime < $maxtime}]
        update
    }
    if {$iok} {
        set itime [elapsed {set ir [fib $i]}]
        .l update $ig data [lappend itimes $i $itime]
        set iok [expr {$itime < $maxtime}]
        update
    }
    if {!($cok || $iok)} {break}
    if {($cok && $iok)} {
        if {$cr ne $ir} {
            puts "ERROR: results differ for $i ($cr vs $ir)"
            break
        }
    }
}

Benchmarks comparing TSP with CriTcl vs Tcc4tcl

Below are outputs from benchmark scripts included in the TSP repo, run with pure Tcl, compiled using CriTcl, then compiled using tcc4tcl

tcl 8.6.5-----------------------------------------------------------------
 msg size:  ---32 bytes---  ---4k bytes---  --64k bytes---  --256k bytes--
 avg secs:       0.0001889       0.0123275       0.1924361       0.7570042 
--------------------------------------------------------------------------

tcl 8.6.5+critcl----------------------------------------------------------
 msg size:  ---32 bytes---  ---4k bytes---  --64k bytes---  --256k bytes--
 avg secs:       0.0000435       0.0001827       0.0015311       0.0042171 
--------------------------------------------------------------------------

tcl 8.6.5+tcc4tcl---------------------------------------------------------
 msg size:  ---32 bytes---  ---4k bytes---  --64k bytes---  --256k bytes--
 avg secs:       0.0000182       0.0001161       0.0015570       0.0061088 
--------------------------------------------------------------------------

lang    cat     grep    hash    loop    proc    fib     sort    wc
ctcl    0.78    1.98    0.86    0.03    0.29    2.58    2.00    12.72   

lang    cat     grep    hash    loop    proc    fib     sort    wc 
c+crt   0.44    1.73    0.61    0.00    0.08    0.26    1.57    16.28

lang    cat     grep    hash    loop    proc    fib     sort    wc   
c+tcc   0.52    1.88    0.75    0.00    0.01    0.09    1.55    17.22

ak - 2015-07-28 18:54:43

Discussion on HN: https://news.ycombinator.com/item?id=9962806

MiR 2022-04-28 Great tool! I got this up and running under TCL 8.6.6 using tcc4tcl last night and it seems to work after patching tcc4tcl and TSP. Just one thing made me puzzle: I had to comment out the usage of TCL_PushCallframe/TCL_PopCallFrame, since these are internals I don't want to use... and I don't know if this workaround has any side effects on variables (esp. namespace and upvar things)... I tested some of the benchmark procs (md5, cat, loop, hash, fib), worked. Update: Meanwhile 'I was able to get tclInt.h up and running under tcc (wich meant to cripple winsock2), patched tclIntStubsptr into tcc4tcl, add the stublibs to tcc (but converted to elf without underscore) and thus was able to get Push/PopCallframe back running. Had to patch some parts of TSP, esp. string comparison, which did expect TCL_DString, but got char* from tsp. I'm not a coder by profession, so I wonder if and how to contribute part of the patches back into TSP/TCC4TCL. I think this combination has great potential, if you don't want to compile whole apps, but only bottleneck-routines. Thank you a lot TP!

SEH 2022-05-21 - I would be interested in seeing your changes and giving the package a try myself. You could try forking the project on Github, adding your changes, then sending a pull request to the original project repository. Do you have a Github account?

TP 2022-06-22 - Happy to see some interest and work on TSP! MiR feel free to keep your own fork running, I haven't had time or motivation to work on TSP for quite a while. Using TCC as a compiler was a goal of mine too, gcc was just easier to get things up and running with critcl.

MiR 2022-06-23 - Yes, meanwhile I got TSP//tcc 0.9.27 running on win32 (Linux still some work to do), I patched some bugs in TSP and introduced some additions. Next goal will be to speed up runtime code in some special cases. Got great support from SEH!

Repo is here https://github.com/MichaelMiR01/tsp - latest branch is tsp4tcc.

Tclkit with batteries included is here https://github.com/MichaelMiR01/tccide

SEH 2022-07-28 - MiR's latest feature additions include:

• Previously, procs could only be created in the global namespace. Now you can define a project namespace; if defined, ALL procs will be rewritten to this namespace

    set tsp::PACKAGE_NAMESPACE pkg
    tsp::proc foo {} {        ;# will be rewritten pkg::foo
        #tsp::procdef void
        #tsp::var v
        variable v            ;# will be connected to $pkg::v
    }

• Inline C can be intermixed directly in with the Tcl code to be compiled. So it is possible to e.g. call other c-procs directly.
• For each line of inline C, a fallback pure-Tcl line can be specified so that a proc can still run in cases where a compiled library can't be made available.

set handle $::tsp::TCC_HANDLE
$handle cproc t1 {int i} double {;#define a simple c procedure here -- it will be compile as c_t1 later
    return i*0.5;
}
tsp::proc test {} {
    #tsp::procdef double
    #tsp::int i
    #tsp::double l k
    set k 1.0
    for {set i 0} {$i<100000} {incr i} {;# $i will be decorated as __i in TSP, $l as __l and so on
        #tsp::inlinec __l = c_t1(__i);
        #tsp::altTCL set l [t1 $i]
        #tsp::inlinec __k = __k+sqrt(__l*__l*0.33);
        #tsp::altTCL set k [expr {$k+sqrt($l*$l*0.33)}]
    }
    return $k
}

• Optional package generation utilities will generate a shared library binary from all TSP-compiled Tcl procedures, plus a fallback pure-Tcl file for when a compiled library is not available, and a pkgIndex.tcl file to check and load the correct choice; as well as a file of the C code that was compiled accompanied by standard compiler command line flags, so that the library can later be compiled with a full-featured compiler like gcc. The utilities also allow inclusion of pure Tcl code not intended to be compiled, to be sourced at package load time.