hat0 writes the brick engine.
hat0 has also written a simple ssh launcher.
Here is a link to a page about a new packaging system called Ultimate Package Blast-o-rama.
And here is my favorite quote from the tcl chat: <lm> The question anyone of us should ask, every day, is "how is what we are doing helping?"
AMG: Happy belated birthday!
hat0: thank you very much!!
CMcC wonders if a belated birthday is where you're actually older than you think you are.
It seems to me that there are three types of wiki page, generally speaking, informational pages (such as those for a given command or extension), discussion pages (such as BraceStarBrace), and miscellaneous (Salt and Sugar, Predicates: on being and having).
I wonder how difficult it would be to modify the HTML-generating parts of the wiki code to render these sorts of pages differently, if they were marked off differently somehow, for example by category. So if a page had "Category Discussion" attached to it, it'd render the header and page contents differently, thus making it immediately clear to the reader, "this probably isn't what you're looking for, if you're looking for a howto or a tip on using a command." Further extending this idea, if a page were "Category thoughtexperiment", then pages like Playing Prolog would be immediately distinguishable as, "There's probably little of interest to the pragmatic Tcl programmer here".
(Yes, I'm drawing broad strokes here about what's useful and what's not. There's a lot of useful stuff on the wiki, but too much of this stuff is buried under ancient crap. It feels like an archaeological dig sifting through these pages at times.)
What I'd propose for the categories that would receive different markup:
Anything else could be rendered as-is.
This is my to-do list of wiki pages that badly need work: pages that either contain useless/outdated/wrong information, or need reformatting. When I find a wiki page but don't have time to clean it up on the spot, I will add it here.
It's March of 2010. I've been asking around today, what sort of vision people have for Tcl, five years out.
Alex Ferrieux says, "freeze as perfect, like /bin/sh"
Colin says, "if I have any vision at all, it's about making Tcl smaller by making its innards more introspective and able to be implemented in Tcl. If you can generate bytecode from Tcl, then you can write all (or some, or many) of the compilation of Tcl commands to Bytecode *in* Tcl." Benefit is an advantage in speed of writing/improving the standard library. Drawback is a period of lack of stability in the core.
Colin amends: In 5 years I would like Tcl to be ubiquitous... the internal combustion engine of the Net. The thing which powers the thing which dominates the computing environment.
kbk says, "It might be nice if there were some killer app (other than eggdrop) that captured the imagination enough to overcome Tcl's stigma. (Or some other marketing that would legitimate it.)"
Cameron says, "*I* want IPv6, UDP, multi-core savvy, more introspection, and graphics-stuff-that-I-don't-know-how-to-say-briefly. Enough consensus that I don't have to hear again about how good A Real Repository would be." And then Cameron adds, "Were Steve around, should I ask for an improved test suite for Tk in the Five-Year Fantasy? It *has* improved; how far remains to go?"
Gerry would like for jacl to keep up with Tcl development, and Aejaks or descendant or godchild is one of the regularly used tools for development of interactive web sites. Also, he would prefer not to have to recode things to fit them into the version of Tcl currently in jacl (for some strange reason that seems to bother me more than converting from Tk to the Aejaks widgets and geometry managers). And he would be much happier if echo2 did not have problems with some kinds of server clustering (which is still unresolved).
SEH: Are you soliciting further comment?
hat0 Heck yeah! Please, speak your mind, I'd love to hear your thoughts/ideas.
These are my thoughts about things that could improve tcl, or various different directions tcl could go, so that it is not just another language in 2015, but a language of growing importance/relevance.
Note to anyone who sees this page: this isn't supposed to be another cloverfield or "tcl 9"-styled effort. I'm not interested in telling anyone what to do, or trying to force my ideas onto anyone. This is my own personal brainstorm, and all are welcome to add to it.
Lars H: In my opinion, the syntax ideas are red herrings. If something is awkward to code, then it's typically better to come up with a new little language to help with the task than to complicate the base syntax (which should probably be frozen as perfect).
hat0 It sounds to me like you are in agreement with Alex Ferrieux, in saying, "present-day Tcl is perfect/almost perfect," and that if present-day Tcl doesn't do it right, don't use Tcl (i.e. invent your own little language, or write it in C). While this is certainly a fair attitude to have, and I can sympathize to some extent with it, I guess I don't agree. I don't think that the syntax is perfect, just familiar.
Lars H: Just to clarify, I believe "invent you own little language" is an important part of the Tcl way — don't require one language to suit all problems, instead make it easy to mix and match. Core features to make it easier to at the script level create little languages is an interesting area for future development.
AMG: Parallel foreach? I'd leave foreach and friends alone, instead introducing a pair of commands to (1) execute a script in a separate thread and (2) execute a script and collect information from or about the threads spawned inside that script. Further levels of nesting, inside the parent thread or in the newly spawned threads, can make this confusing. So this is a truly half-baked idea. accumulate and collect is a possible source of inspiration.
hat0: #2 seems like a good option if it could be made to parallelize the usual constructs very simply, maybe even as simply as prepending the command name to your ordinary command, like so: foreach x list { bleee } -> runmany foreach x list { bleee } ... As for further levels of nesting getting confusing, sure, but just don't do that. :)
Lars H: The idea of a parallel foreach is interesting, but probably not practical. Loop-level parallellisation is rather fine grained, so one would probably want to get the impression of having all threads seeing the same interpreter, as opposed to the current situation where each thread has a separate interpreter. Rewriting Tcl's internals to handle concurrency is going to be difficult (e.g., it would as I understand it be necessary to move away from hash tables). Coroutines could be a first step in this direction however (i.e., separate iterations of the parallel loop could experience the same level of separation as coroutines do).
Looking at it another way, it is rather likely that anything that would benefit from loop-level parallellisation would first benefit from having its body rewritten in C, and then it's rather the C code that one needs to parallelise.
hat0: I cringe whenever I see "write it in C" as the recommended course of action. It feels like a copout to me. The developers of other languages/environments seem much less willing to go to C (e.g. Hiphop for PHP, Pyrex for Python, LuaJIT) for performance, seeking out scripting-level performance increases first. As for it being difficult, sure, I don't doubt it, but I'm not really worried about implementation details right now.
SEH: do coroutines provide part of what you're looking for?
AMG: Coroutines share an interpreter but do not run simultaneously. Threads run simultaneously but do not share an interpreter. hat0 is looking for simultaneous execution, so I don't know how coroutines would help. Coroutines offer interleaved execution; that's probably the best way to describe it.
hat0: That is correct. Also, I'm interested in a very simple way to access and manage the simultaneous execution (such as a single command prepended to the looping construct). That is to say, if the concurrency is achieved through a mechanism that requires some auxiliary code for 90% of use cases, but allows the remaining 10% to happen at all, I'd rather scrap the 10% and auxiliary code and make the 90% accessible.
MS: does Tcl have to care if your parallel foreach body has side effects? Like adding or removing commands, modifying some global variable, writing to a channel. I wish we had a way to insure that some script does nothing of the sort, would come in handy elsewhere too.
hat0: No, I don't think it has to care. Adding or removing commands seldom happens in the lifetime of a program, and almost certainly would never happen in a parallelized script body. Same for writing to a channel, since you'd probably want your writes to be ordered. Maybe some additional sort of syntax could indicate "this script is side-effect free", like maybe using double braces instead of single braces for a script body: foreach item $list {{ # no side-effects here }}
Maybe some kind of modified syntax like that could also be used to tell the bytecode interpreter, "disregard all traces in here," and so on.
CMcC thinks, if you're waiting for the core to do this, you're missing something ... this kind of facility is doable, immediately, in pure tcl, right now. Why wait? It's an afternoon's work to have this happen, and the only possible objection is performance, to which I say (a) I'd bet performance is dominated by interthread stuff, (b) beware premature optimisation, (c) this is what a TAL would be good for.
SEH: While reading the docs for the new reflected channel feature, I noticed a mention that a scripted channel can be opened with its two ends in separate interpreters, including interpreters in separate threads. It occurred to me that that this facility could be used for parallel processing. A master interpreter could start worker threads to do parallel tasks, and establish a link via reflected channel. The master registers a fileevent callback on the channel and waits in the event loop for results, thus combining thread and event processing. One might be able to build the foreach parallelization that you're talking about on that sort of structure.
hat0: Hi SEH, I believe I saw your notes on this on one of the GSOC pages as well. I think this is a great idea and a great first approach, especially since it can be done right now and would work on the trusty 8.5 series. Maybe even turned into a package suitable for Tcllib inclusion.
AMG: I support expr sugar because expr is a critical command that is often misused (see: Brace your expr-essions). The sugar would be the recommended way to do math, and the expr command would be retained for situations where the math expression is intentionally the result of substitution. This closely parallels the [set var] versus $var situation: one-argument set is useful when the variable name is computed, and (for both one- and two-argument set) it's a common bug to prefix the variable name with $ when it doesn't need one.
hat0: Nice parallel/explanation!
AMG: {n} for lindex? What forms can n take? Is it anything that's a valid index argument to lindex? Is arbitrary substitution allowed? Is it a math expression? (In my opinion, Cloverfield naming of list elements is a cleaner and more flexible alternative.)
hat0: I'll take another look at Cloverfield. My first thought is that n is limited to variable interpolation, no script execution, and n can specify a number, range, or combination of them, e.g. {1} or {2-end} or {1, 3-end}.
AMG: Many of my views on Cloverfield aren't published on this wiki, so I'll write a little bit on references. It's great to be able to substitute in the such-and-such element of a list, a dict, or a nested mix, but it's also important to be able to identify that element to a command like [set]. (This way [set] subsumes [dict set], [lset], etc.) FB and I differ on the notation for identifying an element without actually substituting in its value. I think avoiding expr-like performance and security problems requires a special notation for naming variables or elements thereof. When the interpreter sees this notation (leading &), it enters variable parse mode, but instead of immediately substituting in the value (leading $), it constructs a Tcl_Obj to contain the parse tree information. Later, the command may pass that reference object to a function like Tcl_ObjSetVar2(), which processes the parse tree embedded in the reference object to find the actual named element. For example:
set &num (en (zero one two) es (cero uno dos) fr (zéro un deux))
set &num(de) (null eins zwei)
set &num(en){end+1} three
set &var num
set &key fr
set &idx 2
set &"$var"($key){$idx} # returns "deux"
puts $"$var"($key){$idx} # prints "deux"I put this information here as a possible source of inspiration. Take from it what you will.
hat0: Interesting, very interesting. I like the idea of references, and of claiming the & punctuation to note them. The last two examples seem a little punctuation-soup-ish to me, but I could get used to it, especially considering the flexibility presented.
AMG: C++ uses & in a variable or parameter declaration to signify that it be an indirection that is transparently constructed and dereferenced. C uses & and * in an expression to explicitly construct and dereference an indirection. My idea is a mixture of the two: construction is explicit by putting & in the expression, and dereferencing is transparent. I also have an idea for pointers which have to be explicitly dereferenced, so they have a string representation that is independent of the pointed-to value. My recollection is getting fuzzy here, but maybe there was also another difference: references perhaps referred directly to the value (e.g. a variable name is a reference), but pointers referred to the variable name (they're capable of dangling, and they don't affect reference counts). I think my pointer notation was to construct with a leading @ (instead of $) and to dereference with a trailing @, which could be intermixed with other access notations.
set &langs (@num(en) @num(es) @num(fr) @num(de))
puts $langs{3} # prints "@num(de)"
puts $langs(3)@ # prints "null eins zwei"
puts $langs(3)@{1} # prints "eins"As for punctuation soup, I was just showing off a variety of methods of indirection, including having the variable name being the result of substitution. This flexibility makes it possible to always use $ substitution instead of single-argument set.
hat0: Sure, I didn't mean to suggest that it was gratuitous, just that I personally would probably err on the side of simplicity and reduced expressiveness.
CMcC would rather see {X} where X is some arbitrary identifier, representing a command in some scope (maybe tcl::readmacros) run at parse-time. Completely arbitrary read macros!
AMG: I have a question about removing support for arrays. Arrays support traces on individual elements, whereas dicts do not. Do you have any thoughts on this? Do we need a means for attaching traces to dict elements? That can get really hairy really fast. I'll summarize the difference between arrays and dicts: an array is a quasi-variable that is a collection of variables, whereas a dict is a value which is itself a collection of values. (I say "quasi-variable" because arrays don't have all properties of variables; for instance, they don't have values.)
hat0 Agreed, traces on arrays/array elements are useful. I'd suggest extending traces to work on dicts/dict elements.
AMG: This is a fundamentally different kind of trace, since it's attached to a value rather than a variable. Variables are named and distinct, values are anonymous and shared. But obviously you don't want the trace to be on all shared copies of a value; maybe attaching a trace causes it to become unshareable, as if you had modified it. However, this seems messy and wasteful of memory. And in order to create the trace and to usefully process a traced event, you'd need a way of naming values, despite them being anonymous in principal. The only way I can think of is to special case the data types you'd like to support (dict and "top-level" variable values, but consider lists as well). Then the name (or a reference thereto) would have to be placed in the Tcl_Obj struct. (By the way, Cloverfield, or at least my vision of it (I've been out of contact with FB, so we've surely diverged), includes the ability to name list and dict elements.)
hat0: How does Jim do it, do you know off-hand? I seem to recall that Jim does use dicts for arrays. There's probably a solution that, at the outset, seems inelegant but in practice works out all right.
hat0: Ah ha.
CMcC doesn't think deprecating array is so good, or so important. Leave it be.
EE vehemently objects. Don't take my arrays away!
hat0: you mean a syntactical equivalent to the functionality of the list command (with expected substitutions and all)?
AMG: Yeah, I had in mind using matched parentheses as a quoting mechanism which preserves word boundaries. The trouble is that it wouldn't produce a pure list, because both a string and list internal representation would be generated. Maybe it would be okay to drop the string representation. I don't know; I never asked FB why he wanted to preserve the string representation.
hat0: I like it. I'm not too keen on worrying about internals right now.. from a programmer's perspective, some syntactical sugar on [list] is nice.
AMG: This could facilitate resurrection of TIP 251 [L1 ], minus the hack described in the "Incompatibilities" section which should stay dead.
SEH: You may find Tclx's keyed lists interesting.
hat0: Yes, so true, the keyed lists are a good approach, given the syntax as it is now. I'm not afraid to add another syntax rule (placing an additional restriction on variable names/dict key names), though, in order to gain a direct access to the dict value for a given key. Simplifying access to values within dicts--that is, treating them like values like named variables--is a good thing in my opinion.
CMcC: +1 to this! I think making $ understand dict as well as array would be good. I note some conversation on the chat about it. $d(x)(y) for [dict get $d x y] when d is a scalar, otherwise treat it as array already does. I also think this is a subset of the idea of making $ always call a tcl::set command for its function.
hat0: Not really sure about this one yet.. apply is nice. How could anonymous functions be worked more thoroughly into the language? Maybe a new operator that would facilitate this? So that, for example, the map example on the apply man page could be written as a one-liner: [*]{expr {$x**2 + 3*$x - 2}} x -4 -3 -2 -1 0 1 2 3 4
AMG: You're asking for two different things. One is tighter integration with anonymous functions. The other is list comprehensions. These ought to be kept separate.
List comprehensions: There's no need for special syntax. The apply man page shows how easy it is to make a command that does list comprehensions. Possibly you'd want something with the full generality of foreach; the wiki already has code, which can probably be improved by using apply.
Anonymous functions: My idea is for procs to be simply lambdas that are bound to variable names. The first word of a command would be the name of a variable bound to the proc (or compiled command, as the case may be). So the following two lines would be equivalent, for all values of "command" and "args":
command {*}$args
apply $command {*}$argsThis does not mean that the following is valid:
{{a b} {expr {$a + $b}} 1 2Since the apply equivalent would be as follows. Note the dollar sign after the word "apply".
apply ${{a b} {expr {$a + $b}} 1 2In other words, {{a b} {expr {$a + $b}} would be taken as a variable name. Even with this "shorthand" for apply, the apply command itself would remain useful for cases where the lambda isn't bound to a variable name, local or otherwise. Kind of like my suggestion for expr shorthand.
This means merging variables and procedures/commands into the same namespace, which is a major departure with many ups and downs. One "up" is local scoping of procedures, so they don't have to be given a global name, and they can go out of scope; imbuing procs with the properties of local variables gives you most of what you need when you ask for anonymous functions. Obviously the primary "down" is loss of compatibility with existing scripts.
Anyway, to see the impact effect of this change, just mentally delete the word "apply" from the example we've been discussing:
proc map {lambda list} {
set result {}
foreach item $list {
lappend result [lambda $item]
}
return $result
}This change doesn't stand on its own. To be really useful, it should be possible to give this revised map procedure the name of another procedure to execute. I think this would require references, described elsewhere on this page.
proc frob {x} {
expr {$x ** 2 + 3 * $x - 2}
}
map &frob -4 -3 -2 -1 0 1 2 3 4A reference is needed, rather than just a name, because a name does not give sufficient context for the map procedure to find the object being named. We currently handle this situation by requiring that names be global or in the caller's scope, depending on the specific application. But there isn't a general solution at the moment.
On second thought, maybe references aren't required in this case, if all procs and commands have a meaningful string representation which can be used to execute them. Change the last line of the above example to: map $frob -4 -3 -2 -1 0 1 2 3 4 . The value of $frob would be frob's lambda, including namespace information. Now, what's the string representation of a (non-proc) command? I guess it would contain the name or other identifier internally used by Tcl to look up the function pointer. Some indirection would be required to preserve the security of safe interps.
hat0: Thanks for the thoughts and ideas. I do like the distinction between list comprehensions and anonymous functions that you elaborated. In fact, enough that I'm putting list comprehensions into its own section, directly below. I will think about the rest of what you said a bit before adding anything.
aspect: I've been playing with this idea a bit in Playing Scheme. It turns out that with relatively few restrictions on existing code (basically: no spaces in names), the procedure and variable namespaces can be unified and we can even have closures with a little bit of core hacking (conceptually it seems like a small change to me - I might be wrong). While the approach in that page succeeds in replacing proc with sugar for set, that has a huge impact on existing code which expects a flat command namespace. The flat command namespace has become a core part of Tcl and many developers rely on it - things like tdom and Tk would be quite different beasts without it. I think there's room for this to be introduced next to proc with a different name and the rule for resolving command names to be augmented in a way that doesn't break existing code, but it will require significant care and input from core developers who can comment on bytecode compilation, the garbage-collection impacts of closures and the amount of needed extra complexity in the info command.
I'll also note that my attempts to write in a more functional programming style in Tcl are so far leading to fairly ugly code. I don't know if this is simply because my FP is rusty, existing code that I'm interfacing with isn't designed with FP in mind, or something deeper. That said, I miss freely using combinators and creating higher-order functions while Tcl'ing and would like to see these available enough that I'm prepared to put a lot of work into making it possible. If there's enough interest, perhaps we should break out another wiki page for deeper discussion?
hat0: Not sure what this would look like. But I do think some built-in syntax is better than a `map' command, if nothing else to spare the clumsiness of, say, a line like: foo {*}[map $bar [list a b c d]]], with a 'map' defined elsewhere. List transformations.. hm..
AMG: A "clumsy" but powerful list comprehension can be had by combining foreach with accumulate and collect. Each input element is able to generate any number of output elements, by calling collect a variable number of times inside the foreach body. The possibilities are endless. But! The "syntax" is overkill for simple, common cases. But again! The resulting code tends to be clear to read, if verbose. Abuse of functional programming tricks can result in an overly clever trainwreck.
I don't believe a new syntax rule is required for streamlining list comprehensions. I think list comprehensions can be implemented as as command, same as if, foreach, and all the rest. In fact, I don't think the list comprehension part of your example is clumsy at all. I think your example's apparent clumsiness comes from peripheral issues. One, it uses {*}. Two, it builds a list using the list command, which definitely is clumsy. Three, it embeds the list as a literal, even though the list inputs to practical list comprehensions nearly always come from variables. Replace with "foo [map $bar {a b c d}]" and it looks pretty good. Even better is "foo [map $bar $list]". The Python analogue would be "foo([bar(x) for x in data])".
The Python version appears more complex because Python list comprehensions are far more general than your map procedure. I don't know how flexible you want your list comprehensions to be, so let me summarize how Python list comprehensions work. Python list comprehensions start with an expression (not a script, although a function can be invoked) and are followed by for loops and if conditionals, which nest to produce a sort of cross product between all the input lists, filtered by the ifs. The expression is evaluated at the nexus of the nested fors, and its value is collected into the result list.
This is equivalent to nested foreach commands, where only the most basic form of foreach is used: one variable, one list. At the center of the nesting is a call to expr, and the results are collected. The behavior of multiple list foreach can be had in Python by using zip() on all the lists, then unpacking into individual variables: "[bar(x, y) for (x, y) in zip(xdata, ydata)]". For multiple variable foreach behavior, use zip() on strided slices on the lists, then unpack: "[bar(x, y) for (x, y) in zip(data[0::2], data[1::2])]".
So, how much functionality do you want in your list comprehensions? Do you want the core generator part to be a proc/command name plus arguments, a lambda script, an expression, or a substitution? Do you want nested list iteration? Parallel list iteration? Multiple variable list iteration? Filtering with if? Other features?
hat0: Yes, to all, and also a pony. After all, that's what this wishlist is all about. How can the language itself be modified/extended/etc to be easier/more powerful/better/etc? To recap the goal, I'm not looking for "what you can do in Tcl today with some helper procs" but "if you weren't constrained by the 12 rules and the 25 years of history, where could Tcl be in 5 years?" So talk of how Python does it is very useful..if other languages can inform a future direction for Tcl, so much the better. (I still don't have any suggestions for how this syntax would look or work..)
I'm certainly no fan of excess cleverness, and I suspect my tolerance for it is lower than most. But, all the same, if the simple cases can be tackled by a less-general solution, and we may still have the possibility remain for a more general solution handling all other cases, then yes, by all means, enable the simple cases.
AMG: Here's a list comprehension that has everything but the pony.
proc lcomp {expression args} {
set expressions [list $expression]
while {[lindex $args 0] ni {for if}} {
lappend expressions [lindex $args 0]
set args [lrange $args 1 end]
}
if {[llength $args] == 0} {
error "wrong # args: must have at least one opcode"
}
set structure {}
while {[llength $args]} {
switch -- [lindex $args 0] {
for {
set nest [list foreach]
while {[llength $nest] == 1 || [lindex $args 0] eq "and"} {
if {[llength $args] < 3} {
error "wrong # operands: must be \"for\" vars vals\
?\"and\" vars vals? ?...?"
}
lappend nest [lindex $args 1] [lindex $args 2]
set args [lrange $args 3 end]
}
} if {
if {[llength $args] < 2} {
error "wrong # operands: must be \"if\" condition"
}
set nest [list if [lindex $args 1]]
set args [lrange $args 2 end]
} pony {
error "TODO: pony not yet implemented"
} default {
error "bad opcode \"[lindex $args 0]\": must be \"for\" or \"if\""
}}
lappend structure $nest
}
set script [list lappend Result]
foreach expression $expressions {
append script " \[expr [list $expression]\]"
}
foreach nest [lreverse $structure] {
set script [concat $nest [list $script]]
}
set Result {}
eval $script
return $Result
}
lcomp {($a + $b + $c) * $d} for a {1 2} and {b c} {3 4 5 6} for d {1 -1 2 3} if {$d > 0}
# returns: 8 16 24 13 26 39
#
# explanation of each returned element:
# 8=(1+3+4)*1 16=(1+3+4)*2 24=(1+3+4)*3
# 13=(2+5+6)*1 26=(2+5+6)*2 39=(2+5+6)*3
#
# value of $script:
# foreach a {1 2} {b c} {3 4 5 6} {
# foreach d {1 -1 2 3} {
# if {$d > 0} {
# lappend Result [expr {($a + $b + $c) * $d}]
# }
# }
# }
lcomp {"$w $x $y $z"} for w {A B} and {x y} {C D E F} for z {G H I J} if {$z in "G I J"}
# returns: {A C D G} {A C D I} {A C D J} {B E F G} {B E F I} {B E F J}
lcomp {$x ** 2 + 3 * $x - 2} for x {-4 -3 -2 -1 0 1 2 3 4}
# returns: 2 -2 -4 -4 -2 2 8 16 26
lcomp {"$a+$b"} {$a + $b} for a {1 2} for b {3 4}
# returns: 1+3 4 1+4 5 2+3 5 2+4 6This code supports the full generality of foreach, so it's more powerful than Python list comprehensions, although as I've shown before foreach is equivalent in power to Python for plus zip() and slices. Also more than one expression can be given. Using two expressions is useful for dict comprehensions. Using more than two is silly but might be useful someday.
Here's a list of syntax differences between lcomp and Python list comprehensions. lcomp doesn't have the "in" noise word (which would be trivial to add), it has "and" for parallel iteration, it supports multiple variable iteration, it requires you to brace your expr-essions, and multiple expressions are supported (giving dict comprehensions and more).
The complexity of the args parser is due to the varying number of operands per opcode. Things get much simpler if the for opcode took a single operand which is itself a list alternating between variable names and values. But my testing showed that this extra level of "listiness" made lcomp too hard to use. So, the only way to parse args is to read one word at a time, starting with the first word (guaranteed to be an opcode). The script is much easier to generate if I start at the other end, so I lreverse the output of the parser. I had another version which avoided lreverse but instead did [lset {*}[lrepeat $depth end] end+1 $script] which is seriously weird.
All this feels like a lot of work for no real gain, when you compare the arguments that lcomp takes with the script it generates. They are so similar that I wonder why I don't just write the script directly. The benefit of lcomp is that it automatically puts in the accumulate and collect functionality, but that's all it does. I guess this is typical of functional programming, which reformulates scripts into expressions.
I opted for the core of the comprehension to be an expression, but it could just as easily have been a substititution, a script, a proc/command name with arguments, or a lambda to be apply'ed. It's quite possible to go from any one of these to any other, so the choice is mostly arbitrary. For example, my second and last examples do substitutions simply by surrounding the expression in double quotes.
The reason I wrote this code is to demonstrate my point that simple and powerful list comprehensions can be had Right Now without any need to modify Tcl's syntax. Actually, the real reason I wrote this code is because I'm bored and 2000 miles away from home on Easter and have nothing better to do. :^)
hat0: A very nice piece of code, although it does lack the pony. Might be nice to have different selectable modes of operation such as -expr, -cmdprefix, -subst, etc. Not that I'm assigning you more work! Just sayin', this maybe deserves its own page.
AMG: I have copied it to the lcomp page, tidying it up along the way. Alas, the pony is gone.
Enable tcl scripts to generate bytecode directly, then use that facility to permit any command to be compiled by such scripts, perhaps even to the extent of allowing a tcl script to override the current C compilation routine for built-in commands. Enables some new commands to be added without performance concerns. If you don't see that this would be all kinds of wonderful, I can't help it.
AMG: One thing I've dreamed about is the ability to identify the origin of any given Tcl_Obj. Is it a literal found in a particular line of a source file? Is it the product of a concatenation that happened on a certain line? Was it generated by a command (e.g. [read] or [expr]) somewhere in the sources? Etc. Might be interesting. (Might be expensive!)
hat0: Maybe Tcl_Obj-tracing is enabled/disabled via a pragma, so that the expense would be programmer-determined.
AMG I imagine it would be expensive even when disabled, if the pragma had to be checked all the time. If it was a compile-time option, it would be expensive to test and maintain, and it would be almost useless in practice because it would never be enabled when you need it.
hat0: Agreed that it wouldn't make sense as a compile-time option, but I'm not convinced it'd be too expensive as a run-time option. Then again, I don't know what all would be involved.
AMG: Fixing the unbalanced brace issue would require greatly increasing the complexity of the rules for determining whether or not any given brace counts toward the opening/closing brace count. See Cloverfield - Parser for code which skips braces inside double quotes and comments.
AMG: I'd love to have Tcl be native in all popular browsers! But I fear we missed that boat a long, long time ago. However, I will settle for having Tcl be native in the browser running on my computer. Here's a neat idea: if there's a Tcl+DOM plugin, then we can write Tcl/Tk applications that require or bundle the plugin and embed browsers in the GUI, and the script running outside the browser can interact with the script running inside the browser, and as a side effect users of the applications will have the plugin and will be able to access Web sites that make use of it.
hat0: I don't think the ship has sailed. Or, I guess I'd say, there could be another one coming along. There's a ton of mindshare for JavaScript right now, but in the early 2000s there was a perl->php transition for doing server-side web programming. I'm not holding my breath for any sort of victory here, just acknowledging that anything's possible five years out.
CMcC: this is a major win. There's a tcldom package for FF. You'll never get this into IE. FF now permits <script lang='tcl></script> form, too. We absolutely need this. There's a GSoC project for it.
AMG: Don't we already have JIT compilation? (Script is converted to bytecode as it is executed.) You imply that we don't have JIT, so I assume your definition of JIT is to convert from script or bytecode to native machine code. Would this simply consist of concatenating machine code blocks corresponding to the bytecodes? This would cut down on overhead time spent branching through the bytecode execution dispatch loop (which I have never examined), but you need to compare the magnitude of the overhead time with the useful execution time. If 1% of the total time is overhead, you won't gain much of a speedup, and the cost will be a dramatic increase in the size of a compiled script. Also, how would you handle constructs for conditional and repeated execution? C compilers tend to compile such things as body code bracketed by conditional branches, whereas concatenated machine code blocks would (probably) result in function calls or long-distance gotos embedded in a tight loop. To be honest, I'm not sure if this difference would significantly impact performance, so long as the instruction cache can hold more than one contiguous block of code.
SEH: Python has psyco and pypy, just-in-time compilers that use a restricted subset of the language to generate code that the authors claim runs almost as fast as compiled C code. If it can be done with Python, something similar should be possible with Tcl. In the next five years I'd like to see Tcl leverage its elegance in interfacing with C code, with tools that make it possible to delay the necessity of writing C as long as possible, and segregate the necessary compiled code to just where it's critically needed. I'd like to see JO's vision of two-tiered programming (dynamic script plus compiled code) evolve into a three-tier paradigm (dynamic script, compiled script, compiled code). It's still too hard to write stable and secure C code, and we all pay the price. In five years I'd like for Tcl to be understood as meaning "Terrified of C, Largely."
hat0: I like your vision, SEH. I'll give a bold, italicized +1 to this.
SEH: Speed.
AMG: Here's the result of a simpleminded comparison of [info commands] in Jim 0.51 [L2 ] and Tcl 8.6b1 [L3 ].
Commands in both: append, break, catch, concat, continue, dict, error, eval, exit, expr, for, foreach, format, global, if, incr, info, join, lappend, lindex, linsert, list, llength, load, lrange, lreverse, lset, lsort, package, proc, puts, rename, return, scan, set, source, split, string, subst, switch, tailcall, time, unset, uplevel, upvar, while
Commands in Jim only: *, +, -, /, collect, debug, env, finalize, getref, lambda, lambdaFinalizer, lmap, rand, range, ref, setref
Commands in Tcl only: after, apply, array, auto_execok, auto_import, auto_load, auto_load_index, auto_qualify, binary, case, cd, chan, clock, close, coroutine, encoding, eof, exec, fblocked, fconfigure, fcopy, file, fileevent, flush, gets, glob, history, interp, lassign, lrepeat, lreplace, lsearch, namespace, open, pid, pwd, read, rechan, regexp, regsub, seek, socket, tclLog, tell, throw, trace, try, unknown, unload, update, variable, vlerq, vwait, yield, zlib
hat0: Good point, good way of distinguishing between the two.
AMG: We already have a Tk GUI builder. It's called Tcl. :^) Obviously, there are people coming from other environments which strongly encourage or practically require GUI builders which are themselves GUI applications. Having such a tool in Tcl/Tk would be a fine invitation. I would suggest that it prominently show the generated Tcl alongside the Tk GUI being assembled, since I want to showcase how easy it is to directly make a Tk GUI. Also it would be neat if each view updates automatically as the other is edited. But... don't we already have a few programs that do this? Are you suggesting bundling one with the standard Tk distribution? Unless it turns out to be a really simple program (on a par with the demos), I think it would be more appropriate to put it into ActiveTcl.
hat0: I like that idea a lot - to have a GUI builder that shows the Tcl as the interface is built, and vice versa. I strongly dislike all suggestions that "Tk is so easy to build, you just write it!!". I think this attitude makes Tcl/Tk seem primitive and crude, and I think it also leads to very lazy interface design. It's also nice, if you're toying with interface ideas for a program, to be able to drag and drop things to see what best suits your fancy. I'm aware of some of the GUI builder projects, too, but my intent behind that idea is to have a single blessed GUI builder that's included with the core distribution, so that anyone who gets Tcl/Tk automatically gets the dev tools.
SEH: I agree a good visual GUI builder similar to Delphi would be the best possible gateway drug for getting new users to start using Tcl/Tk. Even if experienced Tcl'ers think scripting Tk is the superior way to build GUI's, the influx of new users would be worth it IMO.
hat0: Also, thanks very much for the good feedback/ideas! Keep 'em coming.
Just to clarify, the goal of my "wish I had a pony" list on this page is not to say, "this is what the language should look like," but rather, to say, "Tcl should be more popular/widespread in 2015" and ask, what could be done to get there? (If anyone reading this page does not share that goal, then this is probably not the discussion for them.)
I don't think telling programmers new to Tcl, "it's difficult/unlike anything you're used to, and that's just how it is," is a very good way to reach that goal. I do think that adapting the language a little to non-Tcl developer tastes (like some of the syntax changes I mentioned) would help reach that goal.
Ro 2012-04-04 DUDE WHERE IS BRICK, THE SITE IS DOWN 'N' IM NOT FEELIN IT. U OK? DONT PULL A PILGRIM OR A WHY, WE NEED U MAN
AMG: Try [L4 ]. The most recent change (as of today) was 11 March 2012, made by hat0 himself.