Version 22 of DISCUSSION: Argument Expansion Syntax

Updated 2003-08-02 00:27:56

Note that

 eval exec $program $args

is not an adequate substitute for

   eval [linsert $args 0 exec $program]

or

  exec $program {}$args

since it breaks when $program contains spaces.

Option 4: leading @: Larry Smith If you favor a leading character then "@" is easy to see and read, much more so than `.

Option 4: $(<whatever you want expanded>): Larry Smith Personally, I prefer marking a particular string or set of strings for an additional parsing/substitution pass using something that is reminicent of the current syntax. $([cmd]) or $([cmd1] [cmd2]) marks things clearly as being expanded. After all, it is the $ that signifies substitution, not the {}.

  • $([cmd]) is already legal syntax within variable substituition and shouldn't be mixed up with expansion.

Jacob Levy July 28, 2003: What the example above tries to achieve seems to be possible without any new syntax, using a procedure called apply, which behaves as follows:

 set l {a b c d e}
 apply list $l $l
 ==> {a b c d e a b c d e}

The definition of apply is that its first argument is a function that is applied to rest of the arguments which are evaluated and their values are collected into a list.

The definition of apply is probably something like:

 proc apply {fn args} {
     set l ""
     foreach a $args {
         set l [lconcat $l $a]
     }
     return [eval $fn $l]
 }

 proc lconcat {pre el} {
     set l $pre
     foreach e $el {lappend l $e}
     return $l
 }

Now there may be more to TIP #144 that is not covered by this; however, at least for the given example, no new syntax is needed for implementing the functionality in Tcl.

DGP For this specific example, [apply] would be enough, yes.

When you extend things to get a command that can solve all of the problems addressed by the proposed syntax, you end up with the [expand] command proposed in TIP 103.

TIP 103 was considered and rejected.

Can we move on now please?

DGP For that matter, [apply] will not even work for the [exec] example above (when $program contains spaces).

Jacob Levy Fixed the $args handling, above. To handle $prog with spaces in it, you'd need to pass [list $prog] to the invocation of apply.

TP I like the apply option best. I would suggest that the apply command could have it's own mini-language (ala format, regexp, etc) using the {} or ` formatting. E.g.

   apply {somecommand {}$expand_this $but_not_this}

DGP The "special command with its own mini-language" approach was TIP 103.

TP I went back to read TIP 103 and the wandering discussion in the TCT mail archive. I'd still vote against including some special syntax in Tcl 8.5, 9.0, 10.0, ..., where some command or modified eval could do the job. I just don't favor including some hack that was previously illegal syntax ( {}$foo or `[bar] ) in the base language, or any other non-backward compatible syntax feature. Just call me an OF [L1 ].


KBK (2003-07-28) I know that I have code that will break with the backquote syntax.


Setok I know I probably bore people already with my ramblings on the caret expansion syntax but I find it pleasing. Note that expr does not allow double-carets (no logical XOR). So:

  file join ^^$FilePath

or

  exec $program ^^$args

The reason I find it nice is because it actually looks like the operation it is doing. It, to me, feels like real syntax instead of a hack.

The other option I suggested a couple of times was the ability for a command to affect its calling command line. This is somewhat akin to upvar and uplevel in how dangerous it can be, but allows macro-style functionality into Tcl. Other benefits are that no new syntax is added -- only a new capability provided to commands and procedures. I do realise it's somewhat complex.


(Continuing from a thread on comp.lang.tcl)

I think that paired backquotes is a good marker for argument expansion. The desired operation is to unlist something, so having a paired operator seems fitting. In sh, a backquoted string is not a grouping mechanism, but an expansion mechanism. In fact, to be completely analogous to shell programming, a backquoted string would mean to evaluate the backqoted string and retokenize the result. Hmm...I like this. Essentially, the backquotes would be markers more like square braces than like a dollar sign.

Fleshing this out, the examples from Tip #144 would become:

   destroy `winfo children .`
   button .b `subst $stdargs` -text $mytext -bd $border
   exec $prog `set opts1` `getMoreopts` $file1 $file2

(Note that there's a couple of ways to get an expanded variable dereferences. Good thing or bad thing?)

I think this would be very understandable for anybody with a shell programming background, and tcl certainly has a lot of shell heritage. And IMO, it's *a lot* cleaner. Given that tcl is built around the notion of command evaluation, extending that concept seems like a good thing. I mean, the language doesn't even have infix math operators, and I see proposals for extending the {} syntax with indexes, ranges, etc. That's so far removed from my conception of "the Tcl Way" that it's hard to fathom. After all, wasn't even the dollar sign variable dereference syntax controversial back in the day? "Why do you need a special syntax? You should just use the set command!"

Scott Gargash


DKF: My opinion is that adding a rule to the Endekalog is the way forward (it adds a new class of capability and #144 is about right on the semantic behaviour) but I'm not 100% happy with either {} (too easy to generate from a typo) or backtick (not compatable.)


Joe Mistachkin: For the record, I think adding new global syntax to Tcl for this type of feature is a big mistake. I favor something along the lines of lconvert, [apply], [expand], or [return -code expand].

Here is some Tcl code that implements the required functionality (I whipped this up really quickly):


 ## start of expand.tcl
 #
 # prototype expand command
 # Joe Mistachkin
 # 7/31/2003

 set expand_argument_error "wrong # args: should be \"expand ?-expand expand_list? arg ?arg ...?\""
 set expand_list_error "malformed expansion list"

 proc expand_processExpansionList { array_name error_name expand_list argument_count } {
   # get variable to put error message into...
   upvar "1" $error_name local_error

   if {$argument_count > "0"} then {
     # we are going to put the results into the caller's array.
     upvar "1" $array_name local_array

     # set all arguments to default state (not expanded)
     set index "0"
     while {$index < $argument_count} {
       set local_array($index) "0"
       incr index
     }

     # assume success until we find an actual error.
     set result "0"

     # are we processing a "to" range?
     set range(from) "-1"
     set range(to) "-1"

     set last_item ""
     foreach this_item $expand_list {
       if {[string is integer -strict $this_item] != "0"} then {
         if {$range(from) != "-1"} then {
           # mark the end of the range...
           set range(to) $this_item

           # make sure it's a valid range...
           if {$range(from) <= $range(to)} then {
             # mark entire range to be expanded...
             set index $range(from)
             while {$index <= $range(to)} {
               set local_array($index) "1"
               incr index
             }

             # reset range positions...
             set range(from) "-1"
             set range(to) "-1"
           } else {
             # nope, not a valid range.
             set local_error "not a valid range"
             set result "-1"
             break
           }
         } else {
           if {(($this_item >= "0") && ($this_item < $argument_count))} then {
             # mark this argument for expansion...
             set local_array($this_item) "1"
           } else {
             # out of bounds.
             set local_error "argument index out of bounds"
             set result "-1"
             break
           }
         }
       } else {
         switch -exact -- $this_item {
           "to"    {
             if {$range(from) == "-1"} then {
               if {[string is integer -strict $last_item] != "0"} then {
                 set range(from) $last_item
               } else {
                 # invalid start of range...
                 set local_error "invalid index for start of range"
                 set result "-1"
                 break
               }
             } else {
               # already started a range...
               set local_error "range already started"
               set result "-1"
               break
             }
           }
           "end"   {
             # mark last argument to be expanded...
             set local_array([expr {$argument_count - "1"}]) "1"
           }
           default {
             # unknown argument processing directive.
             set local_error "unknown argument processing directive"
             set result "-1"
             break
           }
         }
       }

       # set last item processed to current item...
       set last_item $this_item
     }
   } else {
     # NOTE: not really an error.
     set local_error "nothing to process"
     set result "0"
   }

   return $result
 }

 proc expand { args } {
   global expand_argument_error
   global expand_list_error
   #
   # check for list of things to expand...
   # (default is to expand nothing)
   #
   if {[llength $args] > "0"} then {
     switch -exact -- [lindex $args "0"] {
       "-expand" {
         # we want to expand the specified arguments.
         set expand "1"
         # next, there should be a list of things to expand.
         set index "1"
         # the option and expansion list are NOT included in the argument count.
         set argument_count [expr {[llength $args] - ($index + "1")}]
       }
       "-noexpand" {
         # we want to NOT expand the specified arguments.
         set expand "0"
         # next, there should be a list of things to NOT expand.
         set index "1"
         # the option and expansion list are NOT included in the argument count.
         set argument_count [expr {[llength $args] - ($index + "1")}]
       }
       default {
         # don't expand by default.
         set expand "0"
         # there is no list of things to expand.
         set index "-1"
         # all arguments are passed along.
         set argument_count [llength $args]
       }
     }

     if {(($index == "-1") || ([llength $args] > ($index + "1")))} then {
       if {$index != "-1"} then {
         # get list of args to modify.
         set expand_list [lindex $args $index]

         set expand_error ""
         array set expand_array {}
         if {[expand_processExpansionList expand_array expand_error $expand_list $argument_count] != "0"} then {
           error "$expand_list_error: $expand_error"
         }
       } else {
         # no arguments to modify.
         set expand_list [list]
       }

       # start with an empty list as the command string to evaluate.
       set result [list]
       # start just beyond the end of the expansion list (if any).
       set this_index [expr {$index + "1"}]
       while {$this_index < [llength $args]} {
         # get the translated argument index...
         if {$index != "-1"} then {
           set argument_index [expr {$this_index - ($index + "1")}]
         } else {
           set argument_index $this_index
         }

         if {[info exists expand_array($argument_index)] != "0"} then {
           # is the argument targeted for expansion?
           if {$expand_array($argument_index) != "0"} then {
             set do_expand [expr {$expand ? "1" : "0"}]
           } else {
             set do_expand [expr {$expand ? "0" : "1"}]
           }
         } else {
           set do_expand [expr {$expand ? "0" : "1"}]
         }

         if {$do_expand != "0"} then {
           # add the expanded list to the command string.
           set result [concat $result [lindex $args $this_index]]
         } else {
           # add the unmodified argument to the command string.
           lappend result [lindex $args $this_index]
         }

         incr this_index
       }
     } else {
       # not enough arguments.
       error $expand_argument_error
     }
   } else {
     #
     # NOTE: potential for design change here.
     #
     # return empty string when given no arguments...?
     set result ""
   }

   uplevel "1" $result
 }

 proc expand_test_proc { args } {
   puts stdout "number of args = [llength $args]"
   puts stdout "args = \{ $args \}"
 }

 ## end of expand.tcl

Try these examples:

 expand -expand {1 to 3 end} expand_test_proc [list argument 1] [list argument 2] [list argument 3] [list argument 4] [list argument 5]
 expand -expand {1} expand_test_proc [list this is a test.]
 expand -noexpand {1} expand_test_proc [list this is a test.]
 expand expand_test_proc [list this is a test.]

RS gets a bit tired of how arguments expand on this page ;)