Raw commands in expr (TIP759)

Playing with TIP 759, I ended up with weird constructs like :

[foreach row $M {(
    ...
    [foreach e $row {(
        ...
   )}];
)}];

While this is working well, anybody can admit there is an over-abondance of delimiters.

So, First, I imagined to create a Foreach mathfunc command. The code is less weird.

...
Foreach("row",  $M, {
    ...
    Foreach("e", $L, {
        ...
    });
});

But this was really inneficient. Moreover, because of the neccessity to quote arguments, as well as to separate them by comas, I was even not totally satisfied with the writing.

Then I came up to a new idea.
Expr parser is more rigid than Tcl parser. In Expr, we can have keywords. That's how I came up with this "Raw command" concept.
What if, in expr, we define some commands as keywords ?

We can detect those keywords during the parsing and report them as a new kind of lexem : I named this lexem code "COMMAND".

Of course, the parsing of this COMMAND lexem can't follow the rules of the SCRIPT lexem, since there is no bracket around it. So we have to define a new Token type. I named this TOKEN : TCL_TOKEN_CMD_IN_EXPR.

To know where to stop the parsing, we must make Tcl_ParseCommand aware of the kind of code it is parsing. To do so, i defined a new nested mode : TCL_CMD_NESTED_IN_EXPR which has a value of 2.

I defined a new TYPE_OP char type and affect this value to every Ascii Operator in the tclCharTypeTable of tclParse.c, then I defined that Tcl_ParseCommand has to stop to parse when char is of type TYPE_OP.

Finally, the boundaries of this new command lexems type are defined to be from the begin of the keywords, to the next ascii operator (or end of script)

This allows to write the previous example like this :

foreach row $M {(
    ...
    foreach e $row {(
        ...
   )};
)};

It is as fast as using script substitution, there is no need to use comas, no need to quote arguments (except if they contain ascii operators), it is more clear.

Moreover, this has interesting consequences. Consider this proc, that is working as expected in tip-759b-tcl90 branch :

proc dot@ari {U V} {(
   + lindex $U 0 * lindex $V 0
   + lindex $U 1 * lindex $V 1
   + lindex $U 2 * lindex $V 2
)}

Let's compare with the same kind of proc taken from tcllib math package :

proc dot@cmd {vector1 vector2} {
    set inproduct 0.0
    foreach v1 $vector1 v2 $vector2 {
        set inproduct [expr {$inproduct + $v1 * $v2}]
    }
    return $inproduct
}

Let's make measurments :

% string length [info body dot@ari]
# 99
% string length [info body dot@cmd]
# 143
% timerate {dot@ari {1 2 3} {4 5 6}}
# 0.470540 µs/# 2125215 # 2125217 #/sec 999.999 net-ms
% timerate {dot@cmd {1 2 3} {4 5 6}}
# 0.823209 µs/# 1214757 # 1214758 #/sec 999.999 net-ms

The code is shorter, but also faster.

Why faster ? A very expensive operation is the assignement operation. Sometimes, we are using assignement of variables, not because we need it, but because we are trying to make the code more clear.

On my point of view, the code of the dot@ari proc is cristal clear. It doesn't need to use extra variable to improve its clarity. That's why, it the speedest possible.

What do people think about this ?