Using sugar for forward-compatible {*}

A/AK: Since I've liked the sugar macroprocessor and decided to use it constantly, I tried to implement a syntax macro that would expand {*} for pre-8.5 Tcl's in a forward-compatible way.

The ease of implementing it with sugar was a great surprise for me. Feel free to use the code below (don't forget to install sugar before testing).

 # Provides {*} support in sugar::proc's for pre-8.5 Tcl
    
 package require sugar 0.1
    
 # Let's test if we already have {*}
 if {[[format catch] {list {*}list}]} {
     # Tcl can't expand, so we define a syntax macro that will 
     # replace commands with {*}ed arguments with eval
     sugar::syntaxmacro expand-8.5 args {
         # The first thing we check: is there anything to expand?
         if {[lsearch $args {{\*}?*}]==-1} {
             # and if there is none...
             return $args
         } else {
             # We turn the whole command into 'eval'
             set evalCmd eval
             foreach token $args {
                 # If the arg is expanded
                 if {[string match {{\*}?*} $token]} {
                     # we put [lrange] to the eval's argument list
                     # to (1) imitate {*}'s pure-list semantics,
                     #    (2) probably gain some speed on pure-list eval
                     set whattoexpand [string range $token 3 end]
                     lappend evalCmd "\[lrange $whattoexpand 0 end\]"
                 } else {
                     # we append a one-element [list] 
                     # to the eval's argument list
                     lappend evalCmd "\[list $token\]"
                 }
             }
             return $evalCmd
         }
     }
 }

 if 1 {
     # Simple proc that takes a list value and appends other args twice
     sugar::proc testingExpand {lst args} {
         lappend lst {*}$args
         lappend lst {*}$args
     }
     # Just to compare: the pre-8.5 traditional naive approach to 'eval'
     sugar::proc testingNoExpand {lst args} {
         eval lappend lst $args
         eval lappend lst $args
     }
     # Let's test it
     puts "Your tcl is [info patchlevel]."
     puts "Testing the proc that uses {*} (real or emulated)"
     puts [time {testingExpand {a b c} d e f} 20000]
     puts "Testing the proc using a 'naive' eval"
     puts [time {testingNoExpand {a b c} d e f} 20000]
 }

After preparing this "toy", "prototype" version of a macro I wanted to know whether it is usable for real life - is there any performance hit introduced by placing literal arguments for eval into list and by placing pure-list expanded arguments into lrange.

Timing results were again of a great surprise:

    Your tcl is 8.4.9.
    Testing the proc that uses {*} (real or emulated)
    29 microseconds per iteration
    Testing the proc using a 'naive' eval
    36 microseconds per iteration

We'd actually gained run-time performance from using {*} in Tcl 8.4! Though it is emulated with sugar, it's a little faster than a straightforward approach to eval (of course, the compile time will increase much).

Of course I do know that eval runs faster with pure-list arguments, but it was rather unobvious to me that performance may increase either by replacing $args with [lrange $args 0] (when $args is always a pure list already) or by replacing lappend with [list lappend] (seems less probably that it's the case here. Someone interested may test it).

Conclusion: though more performance testing is needed, it seems that with sugar, we may use Tcl-8.5's {*} in Tcl 8.4 without any runtime overhead.


glennj You would get better performance out of the naive eval proc with:

     sugar::proc testingNoExpand2 {lst args} {
         eval [linsert $args 0 lappend lst]
         eval [linsert $args 0 lappend lst]
     }

My results:

 % time {testingNoExpand {a b c} d e f} 20000
 18.4102 microseconds per iteration
 % time {testingNoExpand2 {a b c} d e f} 20000
 10.3229 microseconds per iteration

A/AK It's exactly what I don't call naive eval. The macro above expands into eval [lrange ...] [list...] which allows pure-list optimization and is almost as good as using linsert (I've not used linsert in macro expansion because it breaks expected order of parameter substitution).

There is still a lot of code using the eval commandName $args style, which slows the thing down (or used to slow them down) because it doesn't trigger pure-list optimization.

All this thing is not too useful now, as more and more people are migrating to 8.5. The reason to create this page was that I seldom see script-only forward compatibility tricks not leading to slowdown (or leading to speedup).

2009-10-12 I edited the script to provide {*} instead of {expand}, as in the final 8.5 syntax. It was {expand} only on TCL 8.5 alphas.


Here are the timings for current Tcl's CVS HEAD, where the native {*} is used (these results are close to what I've expected):

    Your tcl is 8.5a4.
    Testing the proc that uses {*} (real or emulated)
    13.03415 microseconds per iteration
    Testing the proc using a 'naive' eval
    33.1472 microseconds per iteration

AMG: Heh, we don't have syntax! :^)

AH: We do now. :D

TP: Minor fix for the current sugar version found on the Sugar page.