Version 71 of Closures

Updated 2010-02-06 15:25:34 by nem

These references are standard for the abstract notion of "closure": [L1 ] [L2 ] [L3 ].

From the Tcl chatroom:

dkf Aliases are sort-of poor-mans closures. If we had real closures, we'd use them.

suchenwi What prevents us from having them?

dkf Lifespan issues mainly.

am Why are aliases not closures - sorry, I think I have asked this before, but I do not fully grasp the concept of a closure

dkf If we could be sure that the closure would go away as soon as it was no longer needed, they'd be trivial to add. A closure is really a piece of stack frame that is not on the stack. Or something like that. (The really strange things are continuations, but we're not dealing with those now.) Now it would be pretty easy to create a new closure object and commands to manipulate that, but ensuring that it doesn't go away at the wrong moment is not entirely trivial.

suchenwi Could static variables be called atomar closures? I meant in the sense that they're local to a function, but persistent between calls.

dkf C's static variables? Sort of. Not that they're exactly that. Closures are really a bit antithetical to C anyway.

Lars H, 2010-02-05: It seems Apple is bent on adding some kind of closures to C, though [L4 ].

dkf However, I could envision a closure/stack-frame object that you'd access through upvar A proper closure keeps track of all enclosing stack frames, but that's probably more than is needed for most purposes. Then you could also store the object as an argument to the proc using aliases... And that'd work. So long as the closure doesn't get disposed of too early. We really need to extend Tcl_ObjType when we come to Tcl9.


glennj It took me a while to figure out how aliases could work like closures in the above discussion. I think I came up with an example: a procedure returns the number of times it's been invoked:

  # the first time, send "0" to the aliased proc
  interp alias {} countMe {} _countMe 0

  proc _countMe {count} {
      # for the next invocation, reset the alias
      interp alias {} countMe {} _countMe [incr count]
      return $count
  }

  puts [countMe] ;# ==> 1
  puts [countMe] ;# ==> 2
  puts [countMe] ;# ==> 3

Another attempt at Emulating Closures in Tcl

Todd Coram - While playing with PostgresSQL, I got tired of passing around the connection id to each postgres tcl command (plus there was some other contextual info I wanted to keep around with each connection).

Objectifying the postgres interface seemed over-kill and keeping state in a single namespace prevented me from keeping more than one database connection open at a time. I could have used the classic objectifying technique of keeping the instance info in array, but this felt clumsy. I could have curried the connection id to postgres commands, but that felt too restricting.

Closures are neat because they give you a way to let functions carry a bit of state with it. Tcl Namespaces give you most of what you need, but you start to bleed into object land when you try to create more than one exclusive set of variables.

So, thus was born this late night hacked attempt at emulating closures in Tcl:

 # Create a proc named 'name' that will create a closure over the supplied
 # 'variables' definition for the 'lambda_proc'. Any supplied variables in
 # 'arglist' can be used to initialize 'variables' during the closure
 # definition.
 #
 proc make-closure-proc {name arglist variables lambda_proc} {
    set invoke_context [uplevel namespace current]
    set name_context ${invoke_context}::$name

    # Create a namespace called $name_context to hold auto_cnt
    #
    namespace eval $name_context {
       if {![info exists auto_cnt]} { variable auto_cnt -1}
    }

    # Now, build a proc in invocation context that will create
    # closures. We do this by substituting all of the passed
    # parameters (name, arglist, variables, lambda_proc) and the
    # $name_context.
    #
    # The resulting proc will:
    # 1. Accept $arglist as initializers for the closures.
    # 2. Create a unique closure_name from the auto_cnt variable.
    # 3. Create a namespace for the closure.
    # 4. Evaluate the $variables (optionally evaluating them with
    #    $arglist).
    # 5. Create an alias called 'dispatch' for the lambda_proc.
    # 6. Return the alias.
    #
    namespace eval $invoke_context \
       [subst -nocommands -nobackslashes {
           proc $name {$arglist} { 
                set closure_name \
                ${name_context}::$name[incr ${name_context}::auto_cnt]
                eval [subst {
                    namespace eval [set closure_name] {
                        $variables
                    }
                }]
                namespace eval [set closure_name] {
                    # Curry a dispatcher for the lambda_proc.
                    #
                    curry [namespace current]::dispatch [$lambda_proc]
                } 
                return [set closure_name]::dispatch}
            }]
 }

 proc delete-closure {name} {
     namespace delete [namespace qualifiers $name]
 }

I used a curry proc to create the dispatcher (I really don't create a curry and perhaps namespace code could accomplish a similiar thing, but I had a curry proc handy, so there):

 proc curry {new args} {
    uplevel [list interp alias {} $new {}] $args
 }

And a lambda proc to pass to the make-closure-proc:

 proc lambda {arglst body} {
    set level [info level 0]
    set name [string map {\n _ \t _ \" _ " " _ \; _ $ _ : _ \{ _ \} _ \[ _ \] _} $level]
    set invoke_context [uplevel namespace current]
    proc ${invoke_context}::$name $arglst $body
    return ${invoke_context}::$name
 }

Here is a (contrived) example of how to create and use the closures:

 make-closure-proc make-logputs {_level _filename} {
    variable fd [open $_filename w]
    variable filename $_filename
    variable level $_level
 } {
    lambda {cmd {str ""}} {
        variable fd; variable filename; variable level
        switch -- $cmd {
            puts { 
                puts "Writing ($level to $filename) $str"
                puts $fd "([clock format [clock seconds]] - $level) $str"
            }
            close { 
               close $fd 
            }
        }
    }
 }

 set info [make-logputs INFO info.out]
 set warn [make-logputs WARN warn.out]

 $info puts "Some info: hello world"
 $info puts "blech"
 $warn puts "You have been warned!"

 $info close
 $warn close

 delete-closure $info
 delete-closure $warn

GN There is a strong relation between objects and closures. a classical paper is from Uday Reddy "Objects as closures: abstract semantics of object-oriented languages", who defines objects as an "message environment" (binding messages to methods) with an hidden local environment (binding instance variables to values).

This is pretty close to the notion of objects in XOTcl; the example above can be written much simpler with XOTcl (certainly with other oo languages as well):

 package require XOTcl; namespace import -force xotcl::*

 Class Logger -parameter {level filename}
 Logger instproc init {} {
   my instvar fd filename
   set fd [open $filename w]
 }
 Logger instproc puts {str} {
   my instvar level filename fd
   puts "Writing ($level to $filename) $str"
   puts $fd "([clock format [clock seconds]] - $level) $str"
 }
 Logger instproc close {} {
   my instvar fd
   close $fd
 }

 set info [Logger new -level INFO -filename info.out]
 set warn [Logger new -level WARN -filename warn.out]

 $info puts "Some info: hello world"
 $info puts "blech"
 $warn puts "You have been warned!"

 $info close
 $warn close

 $info destroy
 $warn destroy

Category XOTcl Code


TV Isn't closure like in physics where you'd have a mathematical construct which is 'closed', which is usually pretty darn hard and often quite useless except probably in some essential fundamental cases.

DKF: That's essentially unrelated. (And that's a maths/topology concept which physics has borrowed.)


See custom curry and the references there for an explanation of curry and currying function.


RS asks: Could one say the following are (rudimentary) examples of closures?

1. Default arguments (in danger to be overwritten by caller);

 proc foo {a b {c 1} {d 2} {e 3}} {...}

2. Name-value map in an explicit argument

 set closure [list c 1 d 2 e 3]
 ...
 proc bar {a b closure} {
     foreach {name value} $closure {set $name $value}
     ...
 }

3. Closure values passed in over an alias:

 proc _grill {c d e a b} {...}
 ...
 interp alias {} grill {} _grill 1 2 3

Zarutians first attempt at closures:

 package require Tcl 8.4
 proc closure_var args {
   set varname [lindex $args 0]
   set value   [lindex $args 1]
   upvar $varname breyta
   set breyta $value
   trace remove variable breyta {write} update_closure_var ; # to prevent multiple trace clones
   trace add variable breyta {write} update_closure_var
 }
 proc update_closure_var {varname1 varname2 op} {
   if {$varname2 != ""} {
     set varname "[set varname1]([set varname2])"
   } else {
     set varname $varname1
   }
   upvar $varname newvalue
   debug [info level 1] $varname $op $newvalue
   # redefine the calling proc to reflect the variables new value

   set procsName [lindex [info level 1] 0]
   set procsArgs [info args $procsName]
   set procsBody [info body $procsName]

   debug $procsName $procsArgs
   debug $procsBody

   set temp "closure_var $varname"
   set insertLoc [expr [string last $temp $procsBody] + [string length $temp]]
   set firstHalf [string range $procsBody 0 $insertLoc]
   set lastHalf  [string range $procsBody $insertLoc end]
   set procsBody "[set firstHalf]\{[set newvalue]\}[set lastHalf]"

   debug $procsBody

   proc $procsName $procsArgs $procsBody
 }
 proc debug args { puts "$args" }
 proc someFunction {} {
   closure_var blu 21
   set blu 234
 }

The above implemention doesnt handle that when other sub-procs upvar a closure_var . The next implemention of closures from Zarutian will have that fixed.

Zarutians second attempt at closures

 package require Tcl 8.4

 proc closure_var args {
   set varname [lindex $args 0]
   set value   [lindex $args 1]
   upvar $varname breyta
   set breyta $value
   trace remove variable breyta {write} [list update_closure_var $varname]; # to prevent multiple trace clones
   trace add variable breyta {write} [list update_closure_var $varname]
 }
 proc update_closure_var {originalVarname varname1 varname2 op} {
   if {$varname2 != ""} {
     set varname "[set varname1]([set varname2])"
   } else {
     set varname $varname1
   }
   upvar 1 $varname newvalue
   debug [info level 1] $varname $op $newvalue
   # redefine the calling proc to reflect the variables new value

   set procsName [lindex [info level 1] 0]
   set procsArgs [info args $procsName]
   set procsBody [info body $procsName]

   debug $procsName $procsArgs
   debug $procsBody

   set temp "closure_var $originalVarname"
   debug $temp
   set insertLoc [expr [string last $temp $procsBody] + [string length $temp]]

   set firstHalf [string range $procsBody 0 $insertLoc] 
   set lastHalf  [string range $procsBody $insertLoc end]
   set procsBody "[set firstHalf]\{[set newvalue]\}[set lastHalf]"

   debug $procsBody

   proc $procsName $procsArgs $procsBody
 }
 proc debug args { puts "$args" }
 proc someFunction {} {
   closure_var blu 21
   decr blu
 }
 proc decr {varname {delta 1}} {
   upvar 1 $varname var
   set var [expr $var - $delta]
 }

This implemention should work fine with other sub-procs upvaring closure_vars. But these two implementions have one catch/issue: namely that every previus value of a closure_var is saved too. I am trieing find out how it is possible to get the old value of a variable before it is updated to the new value so I can fix that catch/issue

Just to be sure: isnt a closure just a procedure with some state saved with it, no?


SS 30 Dec 2004: For the Tcl semantic, lexical binding is hard because it's not possible to tell from the source code what's a reference to a variable and what is not. Still it's possible to capture the context of a lambda's creation saving the current stack frame, thus allowing for real closures (procedures where if some variable can't be resolved in the local context, will try to access the stack frame that was active at the procedure creation time). Because of the way Tcl works and the fact it's so dynamic I'm not sure closures may significantly help Tcl programming, but when they are used as objects. It's a more interesting addition to have an object system with garbage collection. Because in most Tcl objects systems objects are "callable", like in [$o a b c], objects can be used to do most of the things you may like to do with closures, they are very similar to procedures but can take/alter their internal state. Of course this is possible to do even with OOP systems not having garbage collection, but the programming style that closures make possible is often about to capture some information to semplify/specialize the next uses of a given procedure, it's hard to think at this programming style as comfortable if you have to clean by hand the objects.


Zarutian wrote on 2. jan 2005: append this to my second attempt at closures:

 proc closure {name args body} {
   set locals [uplevel 1 [list info locals]]
   foreach loco $locals {
     set body "[list closure_var $loco [uplevel 1 [list set $loco]]]\n[set body]"
   }
   proc $name $args $body
 }

and I think you have got something like that SS described above.


SS 3 Jan 2004: Yeah this is a smart idea, to capture the context. Still there are a few things to fix, and many other things that we can't fix at all. What should be fixed is that this way you set the locals variables even if the name is the same as one of the procedure arguments, that's not correct: if there is an argument with that name, the symbol should be bound to it. Second, in real closures, many closures can *share* the environment, and can modify it (and every other closure bound to the same external variables will see the change). We can't get that far with Tcl of course, but still your exercise is very interesting and valuable IMHO.

DKF: I use something very much like closures in my oo2 code. Basically, dict's update and with subcommands (together with some wrapping scripts) allow for closure-like behaviour very easily. The complexity is that the closure is actually stored in a "global" variable, but as long as you ignore the man behind the curtain, you're OK... ;^)

Zarutian 4. jan 2005: SS are you speaking from the past? ;-) Because I hope you meant 3 Jan 2005.

Anyway, I should get to the point. As I understand it Tcl procedure's enviroment is three leveled: global, local to procedure's namespace, local to procedure's level/callstack.

Then when variable lookup is performed (either because of Tcl's seventh substitution rule or procedure set is invoked) inside a procedure's body (the procedure was called) the interpreter first looks for the variable in local callstack scope then in procedure's namespace (if that variable name was marked by calling the procedure 'variable') and at last in the global namespace (likewise if the variable name was marked by calling the procedure 'global').

Where other scripting languages like Lua and Lisp variants use linked tables/hashmaps/dictionaries and lookup the variable's value in the parent table of the current variable|value binding table if not found in current until the top most variable|value binding table is reached. (Hmm... the preceeding sentence may be unclear because of my lack of skill writing English)

Here above I have descriped two possible ways to look up values for variables. So I ask what are the 'pros' and 'cons' of each?

In my opinion I think the latter method enables more flexibility of variable lookup for value. {IS: Að mínu áliti held ég að síðari aðferðin gefi kost á meiri liðleika til að fletta upp gildi breytu.}


SS 23Feb2005:

This is a proposal for closures I made in comp.lang.tcl:

From time to time there is this discussion about adding closures to Tcl, but usually the idea is to make they similar to languages like Scheme, i.e. closures with lexical scoping. My opinion is that this is very hard to do with Tcl, and even not in the spirit of the language. In a Tcl program, what part of a procedure body is a variable is not defined until execution, for example:

 set a 10
 closure {x} {
    incr a
 }

Then I define 'incr' to be like 'puts', and 'a' is no longer a variable. It was already suggested that in Tcl closures should not have any kind of creation-time resolution rule, but that the context where they are defined should be captured as a whole, and used when the closure is running to resolve "unbound symbols" (that are better referred as variables and procedures not otherwise defined during execution in Tcl slang). Still there is the problem that to similuate the lexical scoping you have to take *references* to the shared environment, so that:

 set a 10
 set foo [closure {x} {
    incr a $x
 }]
 set bar [closure {x} {
    incr a $x
 }]
 $foo 1;# -> 11
 $bar 1;# -> 12

foo and bar will share the same 'a'.

I think this is a all to complex and UnTclish, so I tried to design a new semantic for closures.

The semantic is based on a single command closure, that can set/get closure variables in the scope of the current procedure, together with a minimal change in the proc and lambda command (I know we don't have lambda... but still there is a way Tcl users already think about it and should change).

Basically, to set a closure variable, there is to write the following command:

 closure set x 10

this will set 'x' inside the closure of the current procedure. This 'x' will be persistent accross different calls of this procedure, and all this environment will be destroied once the procedure itself is destroied. The procedure can test for the existence of a closure variable with [closure exists x], can get the value with [closure get x] and so on.

The change required to the proc command, is that it can take an optional further argument, that's a Tcl list of key/value pairs used to inizialize the closure at procedure creation time.

so:

 proc foo {} { .... } {x 10 y 20}

will create the procedure foo with x=10 and y=20 inside the closure. The same for lambda.

The following is an example of procedure that returns a progressive integer number every time it's called:

 proc counter {} {
    if {![closure exists x]} {
         set x 0
    }
    set x [closure get x]
    closure set x [expr $x+1]
    return $x
 }

Of course it's better to use the optional argument of proc and write it as:

 proc counter {} {
    set x [closure get x]
    closure set x [expr $x+1]
    return $x
 } {x 0}

Of course for this to be very useful, lambda it's needed. This version of lambda, like proc, should accept the optional argument to initialize the closure. This is an example:

 proc createAdder {x} {
    lamba {y} {expr $y+[closure get $x]} [list x $x]
 }

 set f [createAdder 5]
 $f 10 ;# -> 15

I think that this design can do everything lexical scoping is able to do, but with a command-based interface that plays better with Tcl.

This kind of closures will be added into the Jim interpreter, a small footprint Tcl interpreter I'm writing. This interpreter is already working and I'll be happy to send a preview tar.gz to interested people (the license is the APACHE2, so it's possible to use Jim in commercial projects if needed).

 Regards,
 Salvatore 

and then I replyed to myself with this message:

I forgot to include a couple of points.

The closure command can also access closures of external procedures:

 [closure in $procName get/set x ...] and so on.

This makes possible to share the environment if really needed. Example:

 proc createAdder {x} {
    set a [lamba {y} {expr $y+[closure get $x]} [list x $x]]
    set b [lamba {incr} {closure in $a set x $incr} [list a $a]]
    list $a $b
 }

this returns one closure to add and one to change the increment performed by the first closure.

Also it is worth to note that this way to do closures don't play well with functions as values nor with lambas based on auto expansion, because full closures need to be able to modify their environment so they can not be immutable values.

 Ciao,
 Salvatore 

NEM offers the following as a simple closure-like thing:

 proc closure {name arglist body} {
    if {![string match ::* $name]} {
        # Not fully qualified
        set ns [uplevel 1 namespace current]
        set name ${ns}::$name
    } else {
        set name $name
    }
    set sname [namespace tail $name]
    # Just create a proc as usual, but a namespace along with it
    namespace eval $name [list proc $sname $arglist $body]
    interp alias {} $name {} ${name}::$sname
 }

and a test:

 closure make-counter {start} {
     variable id
     if {![info exists id]} { set id 0 }
     incr id
     closure counter-$id {{amount 1}} [string map [list %S $start] {
         variable counter
         if {![info exists counter]} { set counter %S }
         incr counter $amount
     }]
     return [namespace current]::counter-$id
 }

Could do with a bit of tarting up, and it's not as powerful as closures in Scheme, which get a lot of their power from lexical scoping. Still, it goes quite a long way, I think. Oh, and you can do drop-dead simple ensembles too:

 (tile) 57 % closure MyEnsemble {method args} {

   closure say {message} { puts "Say: $message" }
   closure sing {message} { puts "Sing: $message!" }

   $method {expand}$args
 }
 ::::MyEnsemble
 (tile) 58 % MyEnsemble
 wrong # args: should be "::::MyEnsemble::MyEnsemble method ..."
 (tile) 59 % MyEnsemble say Hello
 Say: Hello
 (tile) 60 % MyEnsemble sing "Tra-la-la"
 Sing: Tra-la-la!

Sarnold 2010-Feb-5: Here is my own implementation of simple closures. To use them you will have to store them in a variable and pass its name to a custom applyc (apply-closure) proc, which saves the environment in the variable holding the closure. The closure proc creates a closure (a list) by passing to it a variable list for the environment. (this is AFAIK the way you create closures in PHP)

package require Tcl 8.5

proc closure {arglist args} {
        # returns a list of 4 elements:
        # - the argument list of apply
        # - the closure var list
        # - the closure vars initial values (to be updated by applyc)
        # - the body
        switch -- [llength $args] {
                1 {return [list $arglist "" "" [lindex $args 0]]}
                2 {lassign $args varlist body}
                default {error "bad arguments to proc closure"}
        }
        if {[llength $varlist]==0} {return [list $arglist "" "" $body]}
        foreach var $varlist {
                lappend data [uplevel 1 set $var]
                set body "upvar 1 closure_$var $var\n$body"
        }
        list $arglist $varlist $data $body
}

proc applyc {name args} {
        upvar 1 $name _closure
        lassign $_closure arglist varlist data body
        foreach var $varlist val $data {
                set closure_$var $val
        }
        # call the lambda
        set res [apply [list $arglist $body] {*}$args]
        # variable updates are saved in local vars
        foreach var $varlist {
                # retrieve them
                lappend values [set closure_$var]
        }
        # save them as third element of _closure
        lset _closure 2 $values
        set res
}
                
        

set x 1
set accu [closure {num} {x} {incr x $num}]
puts $accu
puts [applyc accu 2]
puts [applyc accu 3];#should increase value
proc lam {lam} {applyc lam 2}
puts [lam $accu]
puts [lam $accu];#should print the same

[NEM] 2010-02-06: This is essentially how [dictutils] works, except that in that implementation the lambda expression
and the variable environment (a [dict]) are separate arguments to the apply command. Both have the nice property that
updates are only committed to the environment if no error is thrown in the body of the lambda. Both however need a
little work to ensure that the lambda expression can be byte-compiled properly.

AMG: I'm pretty sure that sproc provides closures.


See also Simple Closures and Objects