Playing with TIP 759, I ended up with weird constructs like :
[foreach row $M {(
...
[foreach e $row {(
...
)}];
)}];While this is working well, anybody can admit there is an over-abondance of delimiters.
So, First, I imagined to create a Foreach mathfunc command. The code is less weird.
proc tcl::mathfunc::Foreach {args} {
set body [lpop args]
uplevel [list foreach {*}$args ($body)]
}
...
Foreach("row", $M, {
...
Foreach("e", $L, {
...
});
});Because of the necessity to quote arguments,
as well as the need to separate them by comas,
I was not really satisfied with the writing, wich for looks too verbose :
I like the command syntax minimalism on that aspect
Moreover, the resulting code was really inneficient, allmost one order of magnitude slower.
Then I came up to a new idea.
Expr parser is more rigid than Tcl parser. In Expr, we can have keywords. Things are not intended to change like they are in command parser. That's how I came up with this "Raw command" concept.
What if, in expr, we define some commands as keywords ?
We can detect those keywords during the parsing and report them as a new kind of lexem : I named this new lexem code COMMAND.
Of course, the parsing of this COMMAND lexem can't follow the rules of the usual SCRIPT lexem, since there is no bracket around it. The function Tcl_CompileTokens would not be able to handle it as usual. So we have to define a new Token type, and to make Tcl_CompileTokens handling it correctly. I named this TOKEN : TCL_TOKEN_CMD_IN_EXPR.
To know where to stop the parsing of this command, we must make the parse function Tcl_ParseCommand aware of what it is parsing. To do so, I defined a new nested mode : TCL_CMD_NESTED_IN_EXPR which has a value of 2.
I defined a new TYPE_OP char type and affect this value to every Ascii Operator in the tclCharTypeTable of tclParse.c, then I defined that Tcl_ParseCommand has to stop the parse at any char that is of type TYPE_OP.
Finally, the boundaries of this new command lexems type are defined to be from the begining of the keywords, to the char just before the next ascii operator (or end of script).
This allows to write the previous example like this :
foreach row $M {(
...
foreach e $row {(
...
)};
)};It is as fast as using script substitution, there is no need to use comas, no need to quote arguments (except if they contain ascii operators), it is more clear.
Moreover, this has interesting consequences.
Consider this proc, that is working as expected in tip-759b-tcl90 branch :
proc dot@ari {U V} {(
+ lindex $U 0 * lindex $V 0
+ lindex $U 1 * lindex $V 1
+ lindex $U 2 * lindex $V 2
)}Let's compare with the same kind of proc taken from tcllib math package :
proc dot@cmd {vector1 vector2} {
set inproduct 0.0
foreach v1 $vector1 v2 $vector2 {
set inproduct [expr {$inproduct + $v1 * $v2}]
}
return $inproduct
}Let's make measurments :
% string length [info body dot@ari]
# 99
% string length [info body dot@cmd]
# 143
% timerate {dot@ari {1 2 3} {4 5 6}}
# 0.470540 µs/# 2125215 # 2125217 #/sec 999.999 net-ms
% timerate {dot@cmd {1 2 3} {4 5 6}}
# 0.823209 µs/# 1214757 # 1214758 #/sec 999.999 net-msThe code is shorter, but also faster.
Why faster ? A very expensive operation is the assignement operation. Sometimes, we are using assignement of variables, not because we need it, but because we are trying to make the code more clear. The loop has a cost, too.
On my point of view, the code of the dot@ari proc is cristal clear. It doesn't need to use extra variables to improve its clarity. It doesn't need loop either. That's why, it the speedest possible.
In tcllib math package, the matrix product proc is using two auxilliary procs :
# Cmd syntax version in tcl lib:
proc transpose@cmd {matrix} {
set c -1
lmap col [lindex $matrix 0] {
incr c
lmap row $matrix {
lindex $row $c
}
}
}
# timerate {transpose@cmd {{1 2 3} {4 5 6} {7 8 9}}}
# 1.897695 µs/# 526954 # 526955 #/sec 999.998 net-ms
# TIP 759 raw commands version :
proc transpose@ari {M} {(
(lindex $M 0 0, lindex $M 1 0, lindex $M 2 0),
(lindex $M 0 1, lindex $M 1 1, lindex $M 2 1),
(lindex $M 0 2, lindex $M 1 2, lindex $M 2 2)
)}
# timerate {transpose@ari {{1 2 3} {4 5 6} {7 8 9}}}
# 0.900455 µs/# 1110548 # 1110549 #/sec 999.999 net-msThe test shows that the raw command version is twice faster (no vars, no loops). The possiblity of removing the first level of bracket allows to get a good level of readibility.
What about the multiplication itself ?
# cmd version from tcllib
proc matmul@cmd { mat1 mat2 } {
set newmat {}
set tmat [transpose@cmd $mat2]
foreach row1 $mat1 {
set newrow {}
foreach row2 $tmat {
lappend newrow [dot@cmd $row1 $row2]
}
lappend newmat $newrow
}
return $newmat
}
# timerate {matmul@cmd {{1 2 3} {4 5 6} {7 8 9}} {{1 2 3} {4 5 6} {7 8 9}}}
# 11.5052 µs/# 86916 # 86917.0 #/sec 999.988 net-ms
# Arithmetic proc version :
proc matmul@ari {M N} {(
T = [transpose@ari $N];
lmap i {0 1 2} {
lmap j {0 1 2} {
dot@ari [lindex $M $i] [lindex $T $j]
}
}
)}
# timerate {matmul@ari {{1 2 3} {4 5 6} {7 8 9}} {{1 2 3} {4 5 6} {7 8 9}}}
# 7.368321 µs/# 135716 # 135716 #/sec 999.999 net-msOnce again, the proc build with arithmetic script is faster.
Avoiding the use of transpose and dot procs, we can even get a faster proc
proc matmul@ari {M N} {
lmap i {0 1 2} {
lmap j {0 1 2} {(
lindex $M $i 0 * lindex $N 0 $j + lindex $M $i 1 * lindex $N 1 $j + lindex $M $i 2 * lindex $N 2 $j
)}
}
}
# timerate {matmul@ari {{1 2 3} {4 5 6} {7 8 9}} {{1 2 3} {4 5 6} {7 8 9}}}
# 4.822302 µs/# 207369 # 207369 #/sec 999.996 net-msBut the loops still have a cost. Avoiding any loop, we can get the fastest proc possible :
# pure arithmetic proc version
proc matmul@ari {M N} {(
((lindex $M 0 0 * lindex $N 0 0 + lindex $M 0 1 * lindex $N 1 0 + lindex $M 0 2 * lindex $N 2 0),
(lindex $M 0 0 * lindex $N 0 1 + lindex $M 0 1 * lindex $N 1 1 + lindex $M 0 2 * lindex $N 2 1),
(lindex $M 0 0 * lindex $N 0 2 + lindex $M 0 1 * lindex $N 1 2 + lindex $M 0 2 * lindex $N 2 2)),
((lindex $M 1 0 * lindex $N 0 0 + lindex $M 1 1 * lindex $N 1 0 + lindex $M 1 2 * lindex $N 2 0),
(lindex $M 1 0 * lindex $N 0 1 + lindex $M 1 1 * lindex $N 1 1 + lindex $M 1 2 * lindex $N 2 1),
(lindex $M 1 0 * lindex $N 0 2 + lindex $M 1 1 * lindex $N 1 2 + lindex $M 1 2 * lindex $N 2 2)),
((lindex $M 2 0 * lindex $N 0 0 + lindex $M 2 1 * lindex $N 1 0 + lindex $M 2 2 * lindex $N 2 0),
(lindex $M 2 0 * lindex $N 0 1 + lindex $M 2 1 * lindex $N 1 1 + lindex $M 2 2 * lindex $N 2 1),
(lindex $M 2 0 * lindex $N 0 2 + lindex $M 2 1 * lindex $N 1 2 + lindex $M 2 2 * lindex $N 2 2))
)}
# timerate {matmul@ari {{1 2 3} {4 5 6} {7 8 9}} {{1 2 3} {4 5 6} {7 8 9}}}
# 3.545945 µs/# 282012 # 282012 #/sec 999.999 net-msTo be compare with its equivalent command proc :
proc matmul@cmd {M N} {
list \
[list \
[expr {[lindex $M 0 0] * [lindex $N 0 0] + [lindex $M 0 1] * [lindex $N 1 0] + [lindex $M 0 2] * [lindex $N 2 0]}]\
[expr {[lindex $M 0 0] * [lindex $N 0 1] + [lindex $M 0 1] * [lindex $N 1 1] + [lindex $M 0 2] * [lindex $N 2 1]}]\
[expr {[lindex $M 0 0] * [lindex $N 0 2] + [lindex $M 0 1] * [lindex $N 1 2] + [lindex $M 0 2] * [lindex $N 2 2]}]]\
[list \
[expr {[lindex $M 1 0] * [lindex $N 0 0] + [lindex $M 1 1] * [lindex $N 1 0] + [lindex $M 1 2] * [lindex $N 2 0]}]\
[expr {[lindex $M 1 0] * [lindex $N 0 1] + [lindex $M 1 1] * [lindex $N 1 1] + [lindex $M 1 2] * [lindex $N 2 1]}]\
[expr {[lindex $M 1 0] * [lindex $N 0 2] + [lindex $M 1 1] * [lindex $N 1 2] + [lindex $M 1 2] * [lindex $N 2 2]}]]\
[list \
[expr {[lindex $M 2 0] * [lindex $N 0 0] + [lindex $M 2 1] * [lindex $N 1 0] + [lindex $M 2 2] * [lindex $N 2 0]}]\
[expr {[lindex $M 2 0] * [lindex $N 0 1] + [lindex $M 2 1] * [lindex $N 1 1] + [lindex $M 2 2] * [lindex $N 2 1]}]\
[expr {[lindex $M 2 0] * [lindex $N 0 2] + [lindex $M 2 1] * [lindex $N 1 2] + [lindex $M 2 2] * [lindex $N 2 2]}]]
}
% (string length [info body matmul@cmd] - string length [info body matmul@ari])
148
# timerate {matmul@cmd {{1 2 3} {4 5 6} {7 8 9}} {{1 2 3} {4 5 6} {7 8 9}}}
3.507285 µs/# 285120 # 285120 #/sec 999.997 net-msWhich is, once again, longer to write, with similar speed.
Raw command in arithmetic syntax proposes another way to call for some commands in expr mini-language. The command name must be recognized by the expr parser as a new lexem type. The command parser must be instruct to stop at the next ascii operator. This allows to remove the first level of bracket, improving the readability of the code.
Adding to this property, the native list support of TIP 759, in the matmul@ari example above, the number of delimiters at the end of line is reduced, the maximum beeing )), while it has to be ]}]\ in a pure command context.
The command version is so inconfortable, that most of us will just end up, in such a case, with creating variables, and using loops, even if they are not a real need for the logic, just for readibility. But this has a cost in speed.
For instance, the implementation of matrix multiplication in Tcl lib math package is about 3 times slower than the fastest implementation.
To improve readiblility, without lost of speed, tcl could be made to expand constant size loop (like lmap i {0 1 2} {...}) as a simple lists of instruction (that the programmer want Tcl engine write for him).
"Raw commands" in expr have this advantage that the code is visualy less polluted by brackets, so can remain readable, even in reasonnably long command.
What do people think about this ?