Tcl Quoting

Difference between version 32 and 33 - Previous - Next
!!!!!!
|[Beginning Tcl%|%previous : Learn Tcl]|[Contents%|%up: Contents]|[Learn to Program%|%next : Learn to Program]|
!!!!!!

'''[Tcl] quoting''' is elegant, minimal, and consistent.  However, it takes a
while to learn to see the multiple layers of interpretation that can be
involved with features like [eval%|%evaluation], [list] interpretation, or
other [domain-specific language%|%domain-specific languages] understood by
routines like `[expr]`, `[regexp]`, or `[string match]`.  Someone new to Tcl
almost always resorts initially to unnecessarily-complicated quoting schemes.
The most common cause of Tcl quoting confusion is assumptions that are carried
over from other languages.  Tcl is not [sh] and it is not [C].  It is also
distinctly different from [Lisp], [Perl], or [Python] or nearly any other
language.  Its design is a novel synthesis of features and ideas from [sh],
[C], [Awk], and [Fortran], among others.  To appreciate its syntax, be prepared
for something fundamentally different, minimal, and pure.



** See Also **

   [double substitution]:   How Tcl [dodekalogue%|%command evaluation] affects [routine%|%routines] that do their own interpretation.

   [escaping special characters]:   Provides routines to escape special characters in arguments to `[glob]`, `[string match]`, `[regexp]`, `[regsub]`, and `[subst]`.

   [Tcl syntax]:   

   [EIAS%|%everything is a string]:   The universal data interface provided by Tcl.

   [Tcl Minimal Escaping Style]:   Advocates using braces and quotes only when necessary.

   [http://www.tcl.tk/man/tcl/TclLib/SplitList.htm%|%Tcl_Merge]:   

   [eval]:   Discusses proper quoting of [script%|%scripts], particularly when they are programmatically composed.

   [FMM%|%Frequently-Made Mistakes]:   

   [list]:   The key to programatically substituting a properly-quoted word into a [script].

   [Is white space significant in Tcl%|%Is white space significant in Tcl?]:   Covers much of the material pertinent to Tcl quoting.

   [Move any widget]:   Contains an over-quoted script and a fixed version.

   `[string map]`:   The go-to routine for help generating properly-quoted Tcl code

   [Toplevel Watermark]:   Contains an over-quoted script and a fixed version.

   [Quoting Hell]:   You know you're a sinner when...

   [Quoting Heaven!]:   Can't have one without the other.

   [Quoting and function arguments]:   A pre-[Changes in Tcl/Tk 8.6%|%8.6] explanation of a few [gotcha%|%gotchas].

   [ftp://ftp.tcl.tk/pub/tcl/mirror/ftp.procplace.com/sorted/info/doc/README.programmer.gz%|%README.programmer], by Brent Welch, 1993:   When and how to use `[list]` to avoid [quoting hell].  Predates [{*}].



** Description **

In a [Tcl] script most characters represent their literal value.  There is a
small set of characters, `"`, `$`, `[[`, `{`, `\`,  and white space, that have
functional meaning, but that can be quoted so that they lose that meaning and
represent themselves instead.  Whereas in other languages quoting may specify
that a value is a string, in Tcl each value already [EIAS%|%is a string], so
quoting is just another mechanism to choose which characters are interpreted,
and which characters represent themselves.



** What Not to Do **

To stay out of trouble, follow this advice:

'''[Brace your expr-essions%|%Don't pass multiple arguments] to `[expr]`'''.
Passing multiple arguments slows things down considerably because a new
expression is effectively being generated and parsed each time instead a single
expression being [bytecode%|%byte-compiled] the first time.  Secondly it might
result in unintended [double substitution].

'''[list%|%Don't attempt to use braces] to turn a value into a [list] or a
[script]'''.   It [dodekalogue%|%doesn't work that way].  For example,
enclosing a value in braces doesn't turn it into a list:

======
set value "what does { do?"
set script "set var {$value}"

#Here comes an error:
eval $script
======

The value of `$script` is clearly incomplete:

======
set var {what does { do?}
======

To do it correctly use `[list]` instead of enclosing a value in braces:

======
set value "what does { do? "
set script "set var [list $value]"
eval $script
======

or even better:

======
set value "what does { do? "
set script [list set var $value]
eval $script
======



** Basic Quoting Techniques **

For simple values no quoting is needed:

======
puts Hello
======

======
set greeting Hello
puts $greeting
======

Substitutions can be performed on any word, including the name of the command:

======
set cmd puts
set greeting Hello
$cmd $greeting
======

A substitution is logically part of a bare string: 

======
set prefix antidis
set root establish
set suffix arian
set entry $prefix[set root]ment${suffix}ism
======

No quoting is needed in the example above because the value contains no literal
whitespace outside of the [script substitution], and `$` is intended as a
[variable substitution].

Perhaps, this example needs to be expanded. The ''$prefix[[set root]]ment${suffix}ism'' argument of ''set'' command is taken by Tcl as a whole, because it doesn't contain any spaces. The only apparent space is a part of ''[[set root]]'' command substitution and as such it isn't a real space that divides arguments of ''set'' command. So, firstly Tcl sees all of ''$prefix[[set root]]ment${suffix}ism'' as one argument, then Tcl performs all substitutions of it, then Tcl passes the resulting string to ''set'' command. We can freely use multiple words in variables' values of this example:

======
set prefix {me antidis.}
set root   { my establish}
set suffix {. my arian}
set entry $prefix[set root]ment${suffix}ism
puts $entry
# a bit longer example presenting the same one argument for Tcl:
set entry "$prefix[set root]ment${suffix}ism"
puts $entry
======

Another consideration is that, though all is done in one pass without recursions, the substitutions are performed from left to right. So, if a "left" command modifies a "right" variable's value, the result will include this new value:

======
set prefix {me antidis.}
set root   { my establish}
set suffix {. my arian}
set entry $prefix[set root { new Establish}]ment${suffix}ism:${root}ment!
======

[variable substitution] and [script substitution] can be used in a bare string:

======
set digits 245
puts 01[string replace $digits 1 1 34]67
======

output:

======
01234567
======

No quoting is required here because all characters in the script have their
normal syntactic meaning.  `[[` and `]]` indicate a [script
substitution%|%script substitution], and the whitespace in that script does not
affect the outer script.

When a word includes literal whitespace, use `{}`

======
set greeting {Good morning}
puts $greeting
======

[Variable substitution] and [script substitution] are not performed within `{}`,
so when those features are needed, `"` can be used instead of `{}`:

======
set name Bob
proc daystage {return Morning}
set greeting "Good [daystage], $name."
puts $greeting
======

output:

======
Good Morning, Bob.
======



** Composing a Script for Evaluation **

First, it's often preferable to compose a single command needing no substitutions:

======
proc reputation_cb {msg msg2} {
    puts $msg
    puts $msg2
    set time [clock seconds]
    puts [list {The time is now} [clock format $time]]
    set ::done 1
}

proc reputation {} {
    set msg {an idle and false imposition}
    set msg2 {got without merit}
    eval [list reputation_cb $msg $msg2]
}
reputation
======

This is likely to be more performant, as [proc%|%procedures] are byte-compiled.

The `[apply]` variant of this example is:

======
proc reputation {} {
    set msg {an idle and false imposition}
    set msg2 {got without merit}
    set script {
        puts $msg
        puts $msg2
        set time [clock seconds]
        puts [list {The time is now} [clock format $time]]
    }
    eval [list apply [list {msg msg2} $script [namespace current]] $msg $msg2]
}
reputation
======

To substitute a complete word into a script, use `[list]` to turn the word into
a list containing only the word.  This works because syntactically a [command]
containing no substitutions is a list, so each word in the command has the same
syntax as an item in a list.  If a value that is not a list is substituted into
a script but not quoted, the resulting script might be syntactically incorrect:

======
set msg {hello world}
set script "set msg2 $msg"
# Here comes an error:
eval $script
======

Here the value of `$script` is

======
set msg2 hello world
======

which results in the error message:

======
wrong # args: should be "set varName ?newValue?"
======

`[list]` correctly quotes the substituted value:

======
set msg {hello world}
set script "set msg2 [list $msg]"
eval $script
======

In this case the value of `$script` is what was intended:

======
set msg2 {hello world}
======

A small but important variation that is more common is to make the entire
command into a list:

======
set script [list set msg2 $msg]
======

This is preferred because even though [EIAS%|%every value is a string], Tcl is
careful not to generate an actual string representation for the value until
something actually requires it.  In this case `[eval]` can use the internal list
representation of the script directly, which has two advantages:  First, a
string representation of the script is not generated, which saves time and
memory.  Second, `[eval]` doesn't have to spend time parsing the string
representation of the script.  Instead, it uses the internal list
representation as the parsed representation of the script, which is a big win
in terms of efficiency.  This has a rather far-reaching implication: Even
though Tcl is known as a string-based language, it is really a
[lisp%|%list-processing] language, and the low-friction way to program in Tcl
is to embrace a list-oriented programming style.

`[string map]` can also be used to properly escape a word substituted into a
script:

======
set msg {hello world}
set script [string map [list @[email protected] [list $msg]] {
    set msg2 @[email protected]
}]
eval $script
======

To avoid ambiguity it's a good idea to give the placeholders a distinctly
visible initial and final character. This is particularly true in large
scripts, where the author might overlook the fact that a placeholder without
clear delimiters occurs as a substring of another value.  `@` is common choice
for the delimiter.

In a small script `[list]` is more convenient, but in a larger script where the
additional escaping of other characters makes the script more difficult to read
`[string map]` is cleaner.

[ycl%|%ycl string template] provides a more concise way to use `[string map]`.



** Composing a Script for Later Evaluation  **

When composing a script for later evaluation, consider that the current
evaluation environment might not be the environment the script eventually is
evaluated in. The following example prints 'goodbye" instead of "hello"
because the scheduled script is evaluated at the top [level]:
evaluated in.
Two examples:
====== set msg goodbye namespace eval ns1 { set msg hello proc doitlater {} { set msg hello after 1000 { puts $msg set ::done 1 } } doitlater vwait ::done } ======
Here, the scheduled script returns 'hello'. Reason is not that "set msg" is defined in "proc doitlater". This is defined locally in the proc and the code executed by 'after' is executed in the global namespace.
But, there is the "set msg hello" inside "namespace eval ns1". Msg is not defined as a variable of that namespace. That means msg is the variable defined in the calling context, which is the global one.
======
set msg goodbye
namespace eval ns1 {
proc doitlater {} {
variable msg hello
after 1000 {
puts $msg
set ::done 1
}
}
doitlater
vwait ::done
}
======
Now, we get 'goodbye'. There exists two variables msg, one in the global context and one in the namespace ns1.
The script provided to 'after' is executed in the global namespace. That means the global msg is used which is set to 'goodbye'.
Instead, either use `[apply]` to capture the local value of `$msg` or use
`[namespace code]` to capture the current namespace:
Instead, either use '[apply]' to capture the local value of $msg, use namespace code to capture the current namespace or use the fully-qualified variable name:
====== set msg goodbye namespace eval ns1 {
set msg hello
proc doitlater {} {
set msg hello
variable msg hello
after 1000 [list apply {msg { puts $msg }} $msg] after 2000 [namespace code { puts $msg
set ::done 1
}]
after 3000 {
puts $::ns1::msg
set ::done 1
}
} doitlater vwait ::done } ======
"namespace eval ::ns1" is used to assure we know the fully-qualified name of the namespace and with this the fully-qualified name of the variable. If this is not wanted you have to stay with 'apply' or 'namespace code'.
Alternatively, `[apply]` could also be used to capture the current namespace: ====== set msg goodbye namespace eval ns1 { variable msg hello proc doitlater {} { set msg hello after 1000 [list apply [list {} { variable msg puts $msg set ::done 1 } [namespace current]]] } doitlater vwait ::done } ====== ** Passing `$args` to Another Command ** In the following script, `$args` are any additional options to `[button]`, e.g., `-fg blue` ====== #! /bin/env tclsh package require Tk proc mybutton {parent name label args} { if {$parent eq {.}} { set myname $parent$name } else { set myname $parent.$name } button $myname -text $label -command [list puts stdout $label] {*}$args pack append $parent $myname {left fill} } mybutton . hello whadda -bg aquamarine ====== `{*}$args` is the right way to do it, but prior to Tcl 8.5, it was necessary to use `[eval]`: ====== eval {button $myname -text $label -command [list puts stdout $label]} $args ====== Alternatively, explicitly combine the lists first rather than relying on `[eval]` to concatenate its arguments: ====== set cmd [concat {button $myname -text $label -command [list puts stdout $label]} $args] eval $cmd ====== ** Quoting at the [C] level ** When composing an individual command to be evaluated from inside [C] code, use Tcl_Merge, which composes a list from ''argv''. The result can then be passed to Tcl_Eval. Tcl_VarEval does not take pains to compose a list from its arguments, but simply concatenates them together. If its arguments happen to not be lists, the results could be unexpected. [APN] This advice is really applicable only when you have the command in the form of ''argv'' to begin with. Normally, at the C level arguments already exist as an array or list of Tcl_Obj structures and you should use Tcl_EvalObj* variants to evaluate commands, not Tcl_Eval or Tcl_VarEval. ** Misc ** [AMG]: [http://www.airs.com/blog/archives/495%|%Tcl] by Ian Lance Taylor, 2011-03-31, discusses Tcl and alleges that its [EIAS] philosophy is its downfall. He argues that EIAS requires very precise quoting in order to get anything nontrivial to work right: ''"Even then I recently wrote some Tcl code with seven consecutive backslashes, admittedly in a complex use case. That's too much for easy reasoning, and in practice requires trial and error to get right."'' Sounds like a case of Quoting Hell, alright. I wish I had the chance to see the code in question and suggest an alternative, since in my experience there's always been a safe, clean way to avoid Quoting Hell. [AMG]: Here's [pooryorick]'s response to the article linked above: : This post mis-characterizes most of the aspects of Tcl that it attempts to describe. In particular, the comment about seven consecutive backslashes is a strong hint that Ian didn’t grasp the elegance of Tcl quoting. This happens when people come to Tcl steeped in other language traditions, and attempt to apply those traditions to to Tcl, which is a different sort of creature. With all due respect to Ian, each of the criticisms in this article indicate that he just didn’t stick with Tcl long enough, or perhaps look into it deeply enough to resolve his misunderstandings of the language. More info at [http://www.ynform.org/w/Pub/ResponseToTclIanLanceTaylor20110331]. ** To Do ** * fold [Quoting Heaven!] into this page !!!!!! |[Beginning Tcl%|%previous : Learn Tcl]|[Contents%|%up: Contents]|[Learn to Program%|%next : Learn to Program]| !!!!!! <<categories>> Tcl Syntax Help