[Richard Suchenwirth] 2013-11-30 - [dis2asm], the converter from the language produced by tcl::unsupported::'''disassemble''', to the one accepted by tcl::unsupported::'''assemble''', [TAL], had a troublesome issue: it could do [for] and [while] loops, but not [foreach]. Disassembling a foreach loop came out as
 % aproc f x {foreach j $x {puts $j}} 
 ByteCode 0x0x9eaad68, refCt 1, epoch 16, interp 0x0x9e057a0 (epoch 16)
  Source "foreach j $x {puts $j}"
  Cmds 2, src 22, inst 29, litObjs 2, aux 1, stkDepth 2, code/src 0.00
  Proc 0x0x9ebb820, refCt 1, args 1, compiled locals 4
      slot 0, scalar, arg, "x"
      slot 1, scalar, temp
      slot 2, scalar, temp
      slot 3, scalar, "j"
  Exception ranges 1, depth 1:
      0: level 0, loop, pc 17-22, continue 10, break 26
  Commands 2:
      1: pc 0-27, src 0-21        2: pc 17-22, src 14-20
  Command 1: "foreach j $x {puts $j}"
    (0) loadScalar1 %v0         # var "x"
    (2) storeScalar1 %v1         # temp var 1
    (4) pop 
    (5) foreach_start4 0 
                [data=[%v1], loop=%v2
                 it%v1        [%v3]]
    (10) foreach_step4 0 
                [data=[%v1], loop=%v2
                 it%v1        [%v3]]
    (15) jumpFalse1 +11         # pc 26
  Command 2: "puts $j"
    (17) push1 0         # "puts"
    (19) loadScalar1 %v3         # var "j"
    (21) invokeStk1 2 
    (23) pop 
    (24) jump1 -14         # pc 10
    (26) push1 1         # ""
    (28) done 
and in [TAL] there are no instructions ''foreach_start'' (see (5)) or ''foreach_step'' (see (10)). But I wanted them so badly...

Luckily it was weekend, and I had some time on my hands - so I made this my fun project of the day. At first, I manually coded a working solution in TAL:
 proc foreachtest {x {_i ""} {_l ""}} {asm {
    load x
    listLength
    store _l
    push -1
    store _i

    label loop;
    push puts
    load x
    load _i
    listIndex
    invokeStk 2
    pop
    incrImm _i +1
    load _l
    lt
    jumpTrue loop

    pop
    pop
    push {}
 }}
 foreachtest {a b c d}
This uses two local variables, _i as index into the input list, and _l to hold the length of the list. The code at the [puts] invocation did however not match the disassembly above. So as next step, I restructured the code to look exactly like the disassembly, except for extra instructions to replace ''foreach_start'' and ''foreach_step'' - I considered them macros. So here is the second version. that meets that expectation:
 proc foreachtest {x {i ""} {2 ""} {1 ""}} {asm {
    load x
    store 1
    pop
    #foreach_start ;#
    push -1        ;#
    store 2        ;#
    pop            ;#
 label L10
    #foreach_step ;#
    incrImm 2 +1  ;#
    load 1        ;#
    load 2        ;#
    listIndex     ;#
    store i       ;#
    pop           ;#
    load 1        ;#
    listLength    ;#
    lt            ;#
    jumpFalse L26
    push puts
    load i
    invokeStk 2
    pop
    jump L10
 label L26
    push {}
 }}
 foreachtest {e f g h}
The macros are comments, followed by their expansions also marked by a trailing comment. In the local variables, I accepted the numeric names 
   * "1" (for a local copy of the list to be iterated over, in case it was changed in the loop) and 
   * "2" (for the index into the list), 
which are valid in Tcl as well. "i" now holds the current list element, like it should. The [llength] can be checked at runtime, no variable needed for that. There still was the problem that inside the expansion, "i" was mentioned (in the "store i" instruction), and could of course have any other name in practice (my counter-test was to use "j" instead ;^) So the macro had to be parametrized. Looking at the disassembly again, the local variable "slots" are listed:
      slot 0, scalar, arg, "x"
      slot 1, scalar, temp
      slot 2, scalar, temp
      slot 3, scalar, "j"
The sequence is not the same as in the argument list, but that doesn't matter - the "temp" ones can use the slot number as their name. I save the slot info in a temporary [array] indexed by slot number. Now, the ''foreach_*'' macros each span three lines:
    (5) foreach_start4 0 
                [data=[%v1], loop=%v2
                 it%v1        [%v3]]
    (10) foreach_step4 0 
                [data=[%v1], loop=%v2
                 it%v1        [%v3]]
and this is a little troublesome, as [dis2asm] processes one line at a time. Clearly, %v3 indicates the loop variable, so I can parse it out of there. Luckily, the foreach_start has this information in its third line, so I can look up slot(3) to retrieve the real name ("j" in this case), before the foreach_step arrives and needs to be substituted.

Proof of the pudding: disassemble, reassemble, run:
======
% aproc f x {foreach j $x {puts $j}} -x
proc f {x {1 {}} {2 {}} {j {}}} {asm {
   load x                   ;# (0) loadScalar1 %v0         # var "x"
   store 1                  ;# (2) storeScalar1 %v1         # temp var 1
   pop                      ;# (4) pop
   push -1; store 2;pop     ;# (5) foreach_start4 0
 label L10;
   incrImm 2 +1;load 1;load 2
   listIndex;store j;pop
   load 1;listLength;lt     ;# (10) foreach_step4 0
   jumpFalse L26            ;# (15) jumpFalse1 +11         # pc 26
   push puts                ;# (17) push1 0         # "puts"
   load j                   ;# (19) loadScalar1 %v3         # var "j"
   invokeStk 2              ;# (21) invokeStk1 2
   pop                      ;# (23) pop
   jump L10                 ;# (24) jump1 -14         # pc 10
 label L26;
   push {}                  ;# (26) push1 1         # ""
                            ;# (28) done
}}
% f {T A L}
T
A
L
======
Boy, was I happy! This is still far away from a real, decent macro assembler, with the tests and the substitutions hard-wired in the code, but it is a good result for the 3 hours it took me to investigate and hack... The '''add_locals''' proc was extended to also pick up nameless temp variables;
======
proc add_locals {argl disasm} {
    foreach line [split $disasm \n] {
        if [regexp {slot.+scalar, "(.+)"} $line -> local] {
            lappend argl [list $local ""]
        } elseif {[regexp {slot (\d+), scalar, temp} $line -> temp]} {
            lappend argl [list $temp ""]
        }
    }
    return $argl
}
======
The [dis2asm] proc was extended in various ways. The lengthy expansion for ''foreach_step'' was put in a variable ''fstep'' with a place-holder @ to be substituted with the name of the loop variable.
======
proc dis2asm body {
    set fstep " incrImm 2 +1;load 1;load 2
   listIndex;store @;pop
   load 1;listLength;lt    "
    set loopvar i
    set res ""
    set jumptargets {}
    foreach line [split $body \n] {
        if [regexp {\# pc (\d+)} $line -> pc] {lappend jumptargets $pc}
    }
    foreach line [split $body \n] {
        set line [string trim $line]
        if {$line eq ""} continue
        set code ""
        if {[regexp {slot (\d+), (.+)} $line -> number descr]} {
            set slot($number) $descr
        } elseif {[regexp {it%v\d+.+\[%v(\d+)\]} $line -> number]} {
            set loopvar [lindex $slot($number) end]
        } elseif {[regexp {\((\d+)\) (.+)} $line -> pc instr]} {
            if {$pc in $jumptargets} {append res "\n label L$pc;"}
            if {[regexp {(.+)#(.+)} $instr -> instr comment]} {
                set arg [lindex $comment end]
                if {$arg eq ""} {set arg "{}"}
                if [string match jump* $instr] {set arg L$arg}
            } else {set arg ""}
            set instr0 [normalize [lindex $instr 0]]
            switch -- $instr0 {
                invokeStk {set arg [lindex $instr end]}
                incrImm   {set arg [list $arg [lindex $instr end]]}
            }
             set code [format " %-24s" "$instr0 $arg"]
            switch -- $instr0 {
                startCommand  {set code ""}
                foreach_start {set code " push -1; store 2;pop    "}
                foreach_step  {set code [string map [list @ $loopvar] $fstep]}
            }
            append res "\n  $code ;# $line"
        }
    }
    append res \n
    return $res
}
======
Now to look around for the next challenge, I have only used 25% of the weekend yet...

<<categories>>Example