Version 10 of Tcl Quoting

Updated 2013-05-02 17:23:50 by pooryorick

Summary

If you think quoting in Tcl is hard or complicated, you're probably doing it wrong! We've seen over and over again that people resort to unnecessarily complicated quoting schemes. Forget what you know. Tcl is not sh and it is not C. Likewise, it is not Lisp, Perl, or Python. The most common cause of Tcl quoting confusion is assumptions that are carried over from other languages. Read on...

See Also

Tcl Syntax
EIAS
Tcl_Merge
eval
Frequently-Made Mistakes
Is white space significant in Tcl
covers much of the material pertinent to "Quoting hell".
Move any widget
which contains an over-quoted script and a fixed version
Toplevel Watermark
which contains an over-quoted script and a fixed version
Quoting Hell
you know you're a sinner when...
Quoting Heaven!
Can't have one without the other
[L1 ], by Brent Welch ,1993

Description

In a Tcl script, most characters represent their literal value. A handful of characters, $, [ , {, \, and white space, have syntactic meaning, but can be quoted so that they lose their syntactical meaning and represent their literal value instead.

Quoting Hell is the trap that Tcl newbies fall into when they don't know how to not overquote things in Tcl. You know you're in quoting hell when your Tcl code begins to look like Perl. Thankfully, unlike, some other languages, quoting hell can be mostly avoided in Tcl. To learn how, digest the contents of this page!

At the C level

When composing an individual command to be evaluated from inside C code, use Tcl_Merge, which composes a list from argv. The result can then be passed to Tcl_Eval. Tcl_VarEval does not take pains to compose a list from its arguments, but simply concatenates them together. If its arguments happen to not be lists, the results could be unexpected.

When Quoting is not Needed at All

Quoting is only necessary when there is a need to change the default interpretation of some character. For simple values, it can be avoided entirely:

puts Hello
set greeting Hello
puts $greeting

So far, so good. No quotes quotes necessary.

Command names are also just literal values. The only thing that makes them special is their position at the beginning of a line:

set cmd puts
set greeting Hello
$cmd $greeting

In the following example, variables are stacked against each other:

set prefix antidis
set root establishment
set suffix mentarianism
set entry $prefix$root$suffix

$prefix$root$suffix contains no literal whitespace, and $ carries its normal syntactic meaning, so no grouping is needed.

The following is an example of both variable subtitution and command substitution, with no grouping required:

set digits 245
puts 01[string replace $digits 1 1 34]67

output:

01234567

Since all characters in the script carried their normal syntactic meaning, no quoting was required. Note, however, that [ started a new command evaluation state where the whitespace served its normal purpose of delimiting words.

When a word includes literal whitespace, use {}

set greeting {Good morning}
puts $greeting

Variable substitution and command substitution are not performed within {}, so when those features are needed, " can be used instead of {}:

set name Bob
proc daystage {return Morning}
set greeting "Good [daystage], $name."
puts $greeting

output:

Good Morning, Bob.

Composing a Script for Evaluation

[uplevel] is used in the following examples, but the same principles hold for any command leading to [double substitution.

The best approach to this is probably just to avoid it by converting the script into a [[proc] and then composing a single command to invoke that proc. Besides sidestepping the need to compose a script, this also has the advantage that [[proc]'s are byte-compiled for better performance.

proc reputation_cb {msg msg2} {
    puts $msg
    puts $msg2
    set time [clock seconds]
    puts "The time is now [clock format $time]"
    set ::done 1
}

proc reputation {} {
    set msg {an idle and false imposition}
    set msg2 {got without merit} 
    uplevel #0 [list reputation_cb $msg $msg2]
}
reputation

One way to get the benefits of a [[proc] without actually creating another proc is to use [[apply]:

proc reputation {} {
    set msg {an idle and false imposition}
    set msg2 {got without merit} 
    set script {
        puts $msg
        puts $msg2
        set time [clock seconds]
        puts "The time is now [clock format $time]"
    }
    uplevel #0 [list apply [list {msg msg2} $script [namespace current]] $msg $msg2]
}
reputation

If you really want to compose a script for evaluation, read on.

When composing more than a single command (a script), the key is to make sure when substituting in values, each value is a well-formed list. There are various ways to do this.

One way to do this is to create a template, and then substitute in any desired values, making sure that the values are well-formed lists:

proc reputation {} {
    set msg {an idle and false imposition}
    set msg2 {got without merit}
    set script {
        puts ${msg}
        puts ${msg2}
        set time [clock seconds]
        puts "The time is now [clock format $time]"
    }
    set script [string map [list \${msg} [list $msg] \${msg2} [list $msg2]] $script]
    uplevel #0 $script
}
reputation

To avoid ambiguity when doing the replacement, the placeholders should be constructed with both a beginning and ending delimiter. Some people use @, but in the previous example, something that looked like a normal Tcl variable was used. If $msg and $msg2 had been used instead of ${msg} and ${msg2}, a subtle error would have occured as $msg2 would have become {an idle and false imposition}2 in the script.

Another way is to puts each substitution into a [[list] command, but the drawback is the need to quote every other special character, which can then often lead to all kinds of confusion as the code becomes less readable and newbie Tcl programmers start piling on the backquotes to try to get things to "work":

proc reputation {} {
    set msg {an idle and false imposition}
    set msg2 {got without merit}
    set script "
        puts [list $msg]
        puts [list $msg2]
        set time \[clock seconds]
        puts \"The time is now \[clock format \$time]\"
    "
    uplevel #0 $script
}
reputation

Composing a Command for Evaluation

double evaluation comes into play here. There are various ways to get Tcl to interpret a value as a script and evaluate it. [eval] is one of those ways. When using [variable substitution to manipulate a value which will then be evaluated as a Tcl script, use [[list] to keep the script well-formed:

proc reputation {} {
    set msg {an idle and false imposition}
    after 1000 [list puts $msg]
    after 2000 "puts [list $msg]"
}

$msg is local to the procedure, so after 1000 {puts $msg} would not have worked because puts $msg will be evaluated later.

set cmd "puts $msg" would have failed in this case, because $msg would have been expanded, and the script would have literally been puts an idle and false imposition, which is obviously too many arguments for [[puts].

list puts $msg is potentially better because "puts list $msg" can have worse performance due to internal list-string conversion.

Passing $args to Another Command

In the following script, $args are any additional options to [[button], e.g., -fg blue

#! /bin/env tclsh
package require Tk
proc mybutton { parent name label args} {
    if {$parent == "."} {
        set myname $parent$name
    } else {
        set myname $parent.$name
    }
    button $myname -text $label -command [list puts stdout $label] {*}$args
    pack append $parent $myname {left fill}
}

mybutton . hello whadda -bg aquamarine

{*}$args is the right way to do it, but prior to Tcl 8.5, it was necessary to use [[eval]:

eval {button $myname -text $label -command [list puts stdout $label]} $args

Alternatively, explicitly combine the lists first rather than relying on [[eval] to concatenate its arguments:

set cmd [concat {button $myname -text $label -command [list puts stdout $label]} $args]
eval $cmd

Misc

AMG: This article [L2 ] by Ian Lance Taylor discusses Tcl and how its EIAS philosophy is its downfall. He argues that EIAS requires very precise quoting in order to get anything nontrivial to work right: "Even then I recently wrote some Tcl code with seven consecutive backslashes, admittedly in a complex use case. That's too much for easy reasoning, and in practice requires trial and error to get right." Sounds like a case of Quoting Hell, alright. I wish I had the chance to see the code in question and suggest an alternative, since in my experience there's always been a safe, clean way to avoid Quoting Hell.

AMG: Here's pooryorick's response to the article linked above:

This post mis-characterizes most of the aspects of Tcl that it attempts
    to describe. In particular, the comment about seven consecutive backslashes
    is a strong hint that Ian didn’t grasp the elegance of Tcl quoting. This
    happens when people come to Tcl steeped in other language traditions, and
    attempt to apply those traditions to to Tcl, which is a different sort of
    creature. With all due respect to Ian, each of the criticisms in this
    article indicate that he just didn’t stick with Tcl long enough, or perhaps
    look into it deeply enough to resolve his misunderstandings of the
    language. More info at
    [http://www.ynform.org/w/Pub/ResponseToTclIanLanceTaylor20110331].

To Do