Version 38 of exec quotes problem

Updated 2014-08-27 19:56:16 by pooryorick

exec quotes its arguments automatically which can lead to problems when unescaped quotes need to be inserted in a commandline.

This sounds like a misinterpretation of what exec does. There is no quoting going on, because exec is directly constructing the argv array of the program being called!

Except on Windows, where there's no way to pass a raw argv array to a program. In that case Tcl tries to quote things, but may fail as not all windows program have the same quoting rules.


If arguments contain spaces, exec will quote the argument before passing it on (some testing shows that it is actually a bit more involved than this). So {c:\Documents and Settings} will be changed to "c:\Documents and Settings". However, sometimes you want to force this quoting of an argument. One example I have encountered is when passing parameters to a batch file that contain comma's:

 test.bat a,b

will call test.bat with two parameters a and b. This can be very useful, but leads to problems when you want to send a,b as one parameter. The solution is to quote the a,b

 test.bat "a,b"

If we want to call this batchfile from Tcl, the first idea would be to use

 exec test.bat {"a,b"} 

but here exec will quote this to:

 test.bat \"a,b\"

Instead the << argument to exec can be used in combination with cmd.exe. << will pipe the remaining args to the stdin of the program. So

 exec cmd << "test.bat \"a,b\"\n" ; # note the trailing \n

will lead to the desired effect. -- MJ

RS Very cool - I had exactly that problem yesterday! To prevent forgetting the \n, one might wrap it like this:

 proc do cmd {exec cmd << $cmd\n}

MHo

  • The problem with << is, that in secured environments (which are today almost everywhere) the command interpreter cmd.exe is disabled and therefore cannot be EXECed itself. Which means that only BATches are allowed. Though, the availability to do a correct exec to a .BAT is required....
  • I noticed that there are still difficulties with EXEC, see the following:
 % glob *.pdf
 Admin-Columbus-V1.01.pdf Benutzer-Columbus-V0.99.pdf Telefonliste_HV.pdf
 % puts "[auto_execok start] Telefonliste_HV.pdf"
 D:/WINNT/system32/CMD.EXE /c start Telefonliste_HV.pdf
 % exec -- "[auto_execok start] Telefonliste_HV.pdf"
 couldn't execute "D:\WINNT\system32\CMD.EXE \c start Telefonliste_HV.pdf": invalid argument
 %
  • The reason here is that /c is accidently translated to \c. eval solves this.

MJ I have noticed before that there seems to be some sort of problem synchronization on the Tcl chat. Yesterday (2006-08-26) was apparently exec day ;-)

MJ The main issue with exec quoting seems to be that the automatic quoting exec applies (even though it is correct and convenient most times) sometimes gets in the way. It would therefore be nice if exec allowed a flag that disables all automatic quoting to allow the programmer to specify the exact quoting in case exec gets it wrong. This would allow a solution like:

 exec -noquoting {test.bat "a,b"}

This will also make use of auto_execok without eval possible (although in 8.5 {*} solves this in different way):

 exec -noquoting "[auto_execok start] http://www.tcl.tk" ; # 8.4.13
 exec {*}[auto_execok start] "http://www.tcl.tk" ; # 8.5

This new flag will only lead to problems with older scripts if a command named -noquoting exists which seems unlikely.

RS Another point that exec -noquoting should handle is to take < and > literal, not as redirections, e.g.

 exec echo "<StartTag>"

MJ Ideally, -noquoting would refrain from interpreting any special characters such as " < << 2> etc. So the tag case should be handled correctly eg:

 % exec echo "<starttag>"
 couldn't read file "starttag>": no such file or directory
 % exec -noquoting echo "<starttag>"
 <starttag>
 %

MJ You can download a patch on 8.5a3 from [L1 ] that adds a -noquote option to exec. The command must still be expanded so:

 exec [auto_execok start]

will still not work. All other quoting of " | & < etc. will be disabled. Note however that | & and others will still have a meaning for the shell. To apply the patch go to the root of the Tcl source dir and type:

  patch -p1 < exec.patch

After running all the test cases for exec only one test case fails (understandably), so the patch shouldn't break any existing functionality:

 ==== exec-14.3 unknown switch FAILED
 ==== Contents of test case:

    list [catch {exec -gorp} msg] $msg

 ---- Result was:
 1 {bad switch "-gorp": must be -keepnewline, -noquote, or --}
 ---- Result should have been (exact matching):
 1 {bad switch "-gorp": must be -keepnewline or --}
 ==== exec-14.3 FAILED

Examples

 % exec cmd /c echo \"test\"
 \"test\"
 % exec -noquote cmd /c echo \"test\"
 "test"

 % # example of disabling interpretation of <
 % exec  cmd /c echo <> ; # error from Tcl
 couldn't read file ">": no such file or directory
 % exec  cmd /c echo \"<>\" ; # quoting doesn't help
 \"<>\"
 % exec  -noquote cmd /c echo <> ; # error from shell
 > was unexpected at this time.
 % exec  -noquote cmd /c echo \"<>\" ; # quoting helps
 "<>"

MJ - After using the exec -noquote version for a while I noticed that TWAPI already offers much of the requested functionality in twapi::createprocess

 package require twapi
 interp alias {} twexec {} ::twapi::create_process {} -showwindow hidden -cmdline
 twexec "echo <>"

I don't see a clear way to get the output of the command into Tcl yet so it might not be usable in cases where you need the output of the command.

VZ I need to do in my tcl script something like this:

 exec awk /pattern/ {print $1} file

Sure I met no success. Does someone know how to do it? - RS: Yes. Single-quotes are specific to shells, where Tcl uses {braces}. But exec quoting does another bad magic here:

 % exec gawk {/0/{print $1}} file
 gawk: cmd. line:1: /0/\{print $1}
 gawk: cmd. line:1:    ^ backslash not last character on line

The only solution I so far have for this (which is rather unelegant) is to write the awk script into a file, and call

 exec awk -f tmp.awk file

jcw - The following line seems to work (on Unix):

    exec awk {/pattern/ {print $1}} file

IOW, use Tcl quoting with braces where you'd have used '...' quoting in the shell.

You could also try using "exec sh -c {command}" to work around quoting issues. VZ Yes,it really works! Thank you.


LV 2007 June 7 - to move forward it would be worthwhile to submit a TIP if you haven't already, detailing the -noquote option, and pointing to your patch. That way, the TCT can debate what you've proposed, and perhaps help tweak things so that it works even better.


HaO Example opening files in windows explorer with a file selected. Possible invocations on a windows command prompt:

C:\Windows\explorer.exe /select,C:\Program Files\tcl8.5\doc\tcl85.hlp

or

C:\Windows\explorer.exe /select,"C:\Program Files\tcl8.5\doc\tcl85.hlp"

but not

C:\Windows\explorer.exe "/select,C:\Program Files\tcl8.5\doc\tcl85.hlp"

which is issued by the exec quoting rules when using:

set filename [file nativename {C:/Program Files/tcl8.5/doc/tcl85.hlp}]
eval exec [auto_execok explorer] [list /select,$filename]

Found solutions:

eval exec [auto_execok explorer] [string map {\\ \\\\} /select,$filename]
eval exec [auto_execok cmd] [list << "explorer.exe /select,$filename\n"]
eval exec [auto_execok cmd] [list << "explorer.exe /select,\"$filename\"\n"]

The first passes the path as multiple arguments and thus avoids exec adding quotes. The second and thiered use the upper mentioned cmd.exe magic to get the required data one and two. I was not able to get controled quotes using solution 1.


MHo 2014-08-27: I still have problems using exec from time to time.... Here's another one:

% exec {*}[auto_execok echo] Dies ist ein "Test" | find /v "Test"
FIND: Parameterformat falsch

I ask myself how I could give the right commandline to FIND.EXE here...

PYK 2014-08-27: First, to eliminate unnecessary Tcl quotes:

% exec {*}[auto_execok echo] Dies ist ein Test | find /v "Test"
FIND: Parameterformat falsch

Next literal double quotes are part of the syntax of find, find, so they should be escaped in Tcl:

% exec {*}[auto_execok echo] Dies ist ein Test | find /v {"Test"}
Access denied - \

The previous command should have worked, so what's that Access denied - \ error message about? Must be a Tcl bug.