foreach little friends

iu2 - started on Jan 7 2007


1. Enjoying coding: Pressing letter keys, not arrow keys

There is often a need to introduce a new variable inside a foreach loop. For example, given:

foreach x $list {
  puts $x
}

in order to count the list's items during the loop I may do:

set c -1
foreach x $list {
    incr c
    puts "$c. $x"
}

While not so bad I still prefer not introducing new variables because

  • Having to go backwards to set new variables doesn't feel very pleasant. Coding feels good when it flows forward, especially inside loops. By coding I mean typing the code.
  • If there is a need for more than one variable, it begins to feel messy.
  • It makes me feel like coding in C... from tcl I would expect something else.

I think this is what mapping lists, filtering lists, lists comprehension and lambdas are all about: Make code typing flow forward. They also make the code clearer, because understanding syntax is easier than exploring algorithms.

This is a valid syntax (in python)

res = [2*x for x in lis]

but this is an algorithm:

res = []
for x in lis:
  res += [2*x]

and it remains an algorithm even if it is written like this

set res {}; foreach x $lis {lappend res $x}

This is better

proc mult2 x {expr $x*2}
set res [struct::list map $res mult2]

but not as good as this one

set res [struct::list map $res {apply {x {expr $x*2}}}

LV Better in what sense? I look at this last item and have no idea what its purpose is, while if I look at the first tcl foreach example I have a clear idea of what it is doing... except that it isn't doing what the rest of the items is doing (there's nothing about multiplying anything by 2 in the foreach example...).

iu2 Oops,I should have written the first example like this

set res2 {}; foreach x $res {lappend res2 [expr {2*$x}]}

This form has three disadvantages:

  • One has to read until the end of the line to get the idea of what this line is doing
  • The list variable itself res cannot be reused, so res2 must be introduced
  • I feel that stacking up two commands in one line is a less favorable coding pattern...

while the last pattern has these three advantages:

  • On can tell what's happening in one glimpse at the beginning of the line - a list is formed out of another list. In order to understand exactly how it is formed - the 'apply' part is right there at the end.
  • The list variable itself can be reused, as shown in the example.
  • One command in the line

I guess this is why there are so many pages here about looping, lambdas, etc.. Well, this is another one.. ;-)

Stu Jan 8 2007 - A proc facilitating the use and cleanup of 'temp' vars, counters, etc.

proc with {vars body} {
    foreach v $vars {
        foreach {name default} $v { break }
        uplevel 1 [list set $name $default]
    }
    uplevel 1 $body
    foreach v $vars {
        foreach {name default} $v { break }
        uplevel 1 [list unset -nocomplain $name]
    }
}

Your example above, using with:

with {{c -1} x} {
    foreach x $list {
        incr c
        puts "$c. $x"
    }
}

2. Introducing a counter inside a foreach

We start straight from eliminating set c -1

set list {a b c d e f g}

foreach x $list c [struct::list iota [llength $list]] {
    puts "$c. $x"
}

With a little help from

proc counters list {
    for {set c 0} {$c < [llength $list]} {incr c} {
        lappend res $c
    }
    return $res
}

we can go

foreach x $list c [counters $list] {
    puts "$c. $x"
}

This is the Python way

proc enumerate list {
    set c 0
    foreach x $list {
        lappend res $c $x
        incr c
    }
    return $res
}

foreach {c x} [enumerate $list] {
    puts "$c. $x"
}

but I prefer the previous one, which is more foreach-y.

3. First iteration commands

This code

foreach x $list first 1 {
    if {$first == 1} {set c 0} else {incr c}
    puts "$c. $x"
}

introduces the variable first, which is set to 1 in the first iteration and then becomes "" for all the rest. Since first is in foreach's arguments list, it doesn't realy seem like going back and setting a new variable.

This code sums up a list

set numbers {1 2 3 4 5 6 7 8 9 10}

foreach x $numbers first 1 {
    if {$first == 1} {set sum $x} else {incr sum $x}
}
puts $sum

Result:

55

and this one finds the maximum

set numbers2 {-2 -4 -1 -3 0 10 3}

foreach x $numbers2 first 1 {
    if {$first == 1 || $x > $max} {set max $x}
}
puts $max

Result:

10

4. Little functions giving a Common Lisp flavour

Instead of first, let's call that variable enablecl, standing for Enable Common Lisp.

proc incrementing {var} {
    uplevel 1 [list if {$enablecl == 1} [list set $var 0] else [list incr $var]]
}

# test
foreach x $list enablecl 1 {
    incrementing c1
    puts "$c1. $x"
}

Let's add more functions like that

proc summing {exp into var} {
    uplevel 1 [list if {$enablecl == 1} [list set $var $exp] else [list incr $var $exp]]
}

proc counting {cond into var} {
    uplevel 1 [list if {$enablecl == 1} [list set $var 0]]
    uplevel 1 [list if $cond [list incr $var]]
}

and give them a try

set numbers {1 2 3 4 5 6 7 8 9 10 11}

# test
foreach x $numbers enablecl 1 {
    summing $x into sum1
    summing [expr 2*$x] into sum2
    counting 1 into count
    counting {int($x)/2*2 == $x} into even
    counting "int($x)/2*2 != $x" into odd
}
puts "$sum1, $sum2, $count steps, $even evens, $odd odds"

Result:

66, 132, 11 steps, 5 evens, 6 odds

Nice.

Maximum and minimum took me a while to figure out...

proc maximizing {exp into var} {
    uplevel 1 [list if {$enablecl == 1} [list set $var $exp]]
    uplevel [list if [concat $exp > $$var] [list set $var $exp]]
}

proc minimizing {exp into var} {
    uplevel 1 [list if {$enablecl == 1} [list set $var $exp]]
    uplevel [list if [concat $exp < $$var] [list set $var $exp]]
}

set numbers2 {-2 -4 -1 -3 0 10 3}

# test
foreach x $numbers2 enablecl 1 {
    maximizing $x into max
    minimizing $x into min
    puts $x,$max,$min
}
puts $max,$min

We advance towards list comprehension by introducing appending

proc appending {exp into var} {
    uplevel 1 [list if {$enablecl == 1} [list set $var [list $exp]] else [list lappend $var $exp]]
}

# test
foreach x $numbers enablecl 1 {
    appending $x into nums
    appending "Number $x" into strings
    if {$x > 5} {appending [expr 2*$x] into twice}  ;# a list comprehension...
    if {[set res [expr $x+1]] > 5} {appending $res into plus1} ;# and an improvement
}
puts [join $nums ,]
puts [join $strings \n]
puts [join $twice ", "]
puts [join $plus1 ", "]

Adding a bit more syntax to appending makes it more interesting

proc appending {exp into var {if if} {cond 1}} {
    uplevel 1 [list if {$enablecl == 1} [list set $var {}]]
    set res [uplevel 1 [list subst $exp]]
    uplevel 1 [regsub -all -- {%\yr\y} [list if $cond [list lappend $var $res]] $res]
}

# test
foreach x {1 2 3 4} y {5 6 7 8} enablecl 1 {
    appending [expr $x+$y] into sums2 if {%r > 8}
}
puts [join $sums2]

More stuff

proc toggling {into var} {
    uplevel 1 [list if {$enablecl == 1} [list set $var 0]]
    uplevel 1 [list if {$enablecl != 1} [list set $var [uplevel 1 [list expr 1-$$var]]]]
}

# test
foreach x {1 2 3 4} enablecl 1 {
    toggling into togl
    puts "$x - $togl"
}

This is a general form of updating a variable each iteration

proc updating {init exp into var} {
    uplevel 1 [list if {$enablecl == 1} [list set $var $init]]
    uplevel 1 [list set $var [uplevel 1 $exp]]
}

# test
foreach x {1 2 3 4 5} enablecl 1 {
    updating 0 {expr $count+1} into count
    updating 0 {expr $sum2+$x} into sum2
    updating 1 {expr $mult*$x} into mult
    updating {} {lappend mult2 [expr 2*$x]} into mult2
}

# print result
foreach x {count sum2 mult mult2} {puts "$x: [set $x]"}

Result:

count: 5
sum2: 15
mult: 120
mult2: 2 4 6 8 10
# another test
foreach x {1 2 3 4 5 6 7 8 9 10 11 12} enablecl 1 {
    updating -1 {expr ($cyc+1)%3} into cyc
    puts "x: $x, cyc: $cyc"
}

The last test leads to another idea

proc cycling {exp into var} {
    uplevel 1 [list if {$enablecl == 1} [list set $var -1]]
    uplevel 1 [list set $var [uplevel 1 [list expr ($$var+1)%$exp]]]
}

# test
foreach x {1 2 3 4 5 6 7 8 9 10 11 12} enablecl 1 {
    cycling 3 into cyc
    puts "x: $x, cyc: $cyc"
}

I often write lists as rows, each row having col_count items, so

foreach number $list enablecl 1 {
    puts -nonewline "$number "
    cycling $col_count into cyc
    if {$cyc == $col_count-1} {puts ""}
}

because of that last example, I just can't resist extending cycling a little bit...

proc cycling {exp into var args} {
    uplevel 1 [list if {$enablecl == 1} [list set $var -1]]
    uplevel 1 [list set $var [uplevel 1 [list expr ($$var+1)%$exp]]]
    set condcount 0
    foreach {on list what} $args {
        # if {[uplevel 1 [list set $var]] in $list} tcl 8.5
        if {[lsearch $list [uplevel 1 [list set $var]]] > -1} {uplevel 1 $what; incr condcount} else {
            if {$on eq "else" && $condcount == 0} {uplevel 1 $list}
        }
    }
}

# test
foreach x {1 2 3 4 5 6 7 8 9 10 11 12} enablecl 1 {
    puts -nonewline "$x "
    cycling 3 into cyc on 2 {puts ""}
}

# or even this
foreach x {1 2 3 4 5 6 7 8 9 10 11 12} enablecl 1 {
    puts -nonewline "$x"
    cycling 3 into cyc on 2 {puts ""} else {puts -nonewline " "}
}

# another test
foreach x {1 2 3 4 5 6 7 8 9 10 11 12} enablecl 1 {
    cycling 4 into cyc on 0 {puts "start: cyc=$cyc"} on 3 {puts "last: cyc=$cyc"} on {1 2} {puts "middle: cyc=$cyc"}
    puts $cyc
}

5. Helper functions with a helper variable

The following example uses another variable besides var

proc cyclelist {hlpvar list into var} {
    uplevel 1 [list if {$enablecl == 1} [list set $hlpvar -1]]
    set len [llength $list]
    set index [uplevel 1 [list set $hlpvar]]
    set index [expr ($index+1)%$len]
    uplevel 1 [list set $hlpvar $index]
    uplevel 1 [list set $var [lindex $list $index]]
}

# test
foreach x {1 2 3 4 5 6 7 8 9 10 11 12 13 14} enablecl 1 {
    cyclelist cyc1 {one two three four} into cyc2
    puts "$x $cyc2 ($cyc1)"
}

6. Summary

  • With these little helpers foreach-ing can be more readable, more fun and more press letter keys, not arrow keys ;-)
  • I'm sure more functions of this type can be introduced
  • There still may be bugs in them, so bug fixes are welcome
  • I'm not sure the uplevel 1 [list if... way I use is the best. Shorter or more readable formats will be appreciated.

LES on same day:

set res [struct::list map $res {apply {x {expr $x*2}}}

You call that "readable"?


iu2 well, yes, beacuse I recognize a structure: map-list-apply.


RS Due to the path structure, the first part is a bit cluttered. If you seriously use map, it might help to

interp alias {} map {} struct::list map

and then code (remember to brace your expr-essions :^)

set res [map $res {apply {x {expr {$x*2}}}}]

NEM: And adding a constructor for lambdas helps a lot:

proc func {params body} { list ::apply [list $params [list expr $body] ::] }
map $res [func x {$x*2}]

Unfortunately, the tcllib versions of map, filter and fold take arguments in an inconvenient order for interp alias, so e.g. we have to do:

proc ldouble xs { map $xs [func x {$x*2}] }

rather than simply:

def ldouble = map [func x {$x*2}]

aspect -- frankly, I find the use of list map and apply awkward in the above example. Using lmap it simplifies (without the need for lambda) to:

set res [lmap i $res {expr {$i*2}}]

.. or, using the version (which I prefer for succinctness, even if it is a little less flexible) from list map and list grep you get:

set res [lmap $res {expr {$_*2}}]

NEM I'm not sure which uses you find awkward? The func constructor I provide is a simple single-line proc, and the resultant map is just as tiny as lmap:

proc map {f xs} {
    set ys [list]
    foreach x $xs { lappend ys [{*}$f $x] }
    return $ys
}
map [func i {$i*2}] $res

It can also be used directly with existing commands. For instance, if we want to convert all words in a list to upper-case, we can simply do:

map {string toupper} {this is a list of words}

This is just one of the benefits of properly factoring out the construction of a callback from the control construct that uses it. Others are that we can use multiple different constructors in different situations: e.g. a plain lambda, a version that uses expr, a version that captures a closure, etc. Not to mention the benefits of byte-compilation that come from having the lambda as a single argument.


FM : sometimes I like to use this proc

proc except cond {
    uplevel [list if $cond {return -code continue}]
}

set L [list 0 1 2 3 4 5 6 7 8 9]

foreach e $L {
    except {$e%2==1 || $e==0}
    lappend Res $e
}
set Res ; # 2 4 6 8