code is data is code

Lars H, 2008-08-29: Having been asked to further clarify proof of concept: suspend and resume, I now see myself forced to take data is code one step further, and devise a way to turn Tcl scripts into something that can both be conveniently edited as data, and run as code — much like SEXPs are for LISP (but with simpler syntax, since this is Tcl). Technically, this is an appendix to parsetcl, and the code below depends on that package.

Concretely, what I'll aim for is:

every script is a command, and every command is a list.

Obviously, this is not the case in Tcl in general, since scripts are sequences of zero or more commands, so we'll have to introduce a new control structure to represent sequences of commands: cmdseq. This can be defined as follows:

 proc cmdseq {args} {
    set res ""
    foreach cmd $args {set res [uplevel 1 $cmd]}
    return $res

This way, every script can be turned into a command; for example

  set a 3
  incr a 2
  append a x

can be expressed as

  cmdseq {set a 3} {incr a 2} {append a x}

The other identification — to make every command a list, which amounts to getting rid of explicit substitution — is trickier, but quite doable. What one needs is a control structure that does command substitution on the arguments of a command, and then evaluates the command as a whole. Concretely, I chose

 proc gamma {prefix args} {
    foreach cmd $args {lappend prefix [uplevel 1 $cmd]}
    uplevel 1 $prefix

(Why name this gamma? Well, since it is a variadic composition operation, I immediately associate it to the structure map of an operad, and this is usually denoted gamma.) A basic example is that the command with substitution

  A \b $c [D]

can be turned into the command&list

  gamma {A \b} {set c} D

For more complicated examples however, doing it by hand is not very reliable, so I've implemented a command that does the transformation for me — from ordinary Tcl script to one which is also a single command and list:

 % parsetcl::cidic {list $a [B $c] [D][E] $f[G $h] $i($j,[K]); error "Whatever: $l" "Not tonight"}
 cmdseq {gamma list {::set a} {gamma B {::set c}} {gamma {::format %s%s} D E} {gamma {::format %s%s} {::set f} {gamma G {::set h}}} {gamma ::set {gamma {::format i(%s)} {gamma {::format %s,%s} {::set j} K}}}} {gamma error {gamma {::format {Whatever: %s}} {::set l}} {::return -level 0 {Not tonight}}}

String concatenation could probably have been done slightly more elegantly using the cconcat command, but format works too.

So, how is the parsetcl::cidic that can output such monstrosities implemented, then? Thus:

 package require parsetcl 0.1 ; # See
 namespace eval ::parsetcl::constant {}

 proc ::parsetcl::constant::Lr {interval text args} {return $text}
 proc ::parsetcl::constant::Lb {interval text args} {return $text}
 proc ::parsetcl::constant::Lq {interval text args} {return $text}

 proc ::parsetcl::constant::Sb {interval text args} {return $text}

 proc ::parsetcl::constant::Sv {interval text args} {error varsub}
 proc ::parsetcl::constant::Sa {interval text args} {error varsub}
 proc ::parsetcl::constant::Sc {interval text args} {error cmdsub}

 proc ::parsetcl::constant::Mr {interval text args} {
     set res {}
     foreach a $args {append result [eval $a]}
     return $res

 proc ::parsetcl::constant::Mq {interval text args} {
     set res {}
     foreach a $args {append result [eval $a]}
     return $res

 proc ::parsetcl::constant::Cd {interval text args} {error cmd}
 proc ::parsetcl::constant::Rs {interval text args} {error root}

 namespace eval ::parsetcl::cidic {}
 # Commands in this namespace return a command (list of words) 
 # corresponding to the node they represent. For nodes that are 
 # words or pieces of words, that means the return value of that 
 # command is what the corresponding word became after substitution.

 interp alias {} ::parsetcl::cidic::constant {} namespace inscope ::parsetcl::constant
 proc ::parsetcl::cidic::Lr {interval text args} {
     list ::return -level 0 $text
 interp alias {} ::parsetcl::cidic::Lb {} ::parsetcl::cidic::Lr
 interp alias {} ::parsetcl::cidic::Lq {} ::parsetcl::cidic::Lr

 proc ::parsetcl::cidic::Sb {interval text args} {
     list ::return -level 0 $text

 proc ::parsetcl::cidic::Sv {interval text name args} {
     list ::set [constant $name]

 proc ::parsetcl::cidic::Sa {interval text name index args} {
     Cd {} {} {Lr {} ::set} [list Mr {} {} $name {Lr {} (} $index {Lr {} )}]

 interp alias {} ::parsetcl::cidic::Sc {} ::parsetcl::cidic::Rs

 proc ::parsetcl::cidic::Mr {interval text args} {
     set parts {}
     set const {}
     foreach a $args {
         if {![catch {
             append const [constant $a]
         }]} then {continue}
         lappend parts $const
         set const {}
         lappend parts [eval $a]
     if {![llength $parts]} then {
         return [list ::return -level 0 $const]
     set res [list gamma {::format fstr}]
     set fstr {}
     foreach {str cmd} $parts {
         append fstr [string map {% %%} $str] %s
         lappend res $cmd
     append fstr [string map {% %%} $const]
     lset res 1 1 $fstr
     return $res
 interp alias {} ::parsetcl::cidic::Mq {} ::parsetcl::cidic::Mr

 proc ::parsetcl::cidic::Cd {interval text args} {
     set res [list gamma]
     set prefix {}
     catch {
         foreach a $args {
             lappend prefix [constant $a]
     lappend res $prefix
     foreach a [lrange $args [llength $prefix] end] {
         lappend res [eval $a]
     if {[llength $res]==2} then {
         return $prefix
     } else {
         return $res
 proc ::parsetcl::cidic::Nc {args} {return -code continue}

 proc ::parsetcl::cidic::Rs {interval text args} {
     set res [list cmdseq]
     foreach a $args {lappend res [eval $a]}
     if {[llength $res]==2} then {return [lindex $res 1]}
     return $res

 proc ::parsetcl::cidic {script} {
     namespace eval [namespace current]::cidic [
         basic_parse_script $script

With acknowledgements to CMcC, whose contribution on the parsetcl page showed me how to conveniently turn parse trees back into scripts, and was an important starting point for data is code.