hat0 writes [the brick engine]. hat0 has also written a simple [ssh launcher] and a [pure-tcl BMP reader/writer]. Here is a link to a page about a new packaging system called [Ultimate Package Blast-o-rama]. And here is a link to a little package which can be used to preload DLLs on a Windows-based machine: [dllfix] ---- [AMG]: Happy belated birthday! [hat0]: thank you very much!! [CMcC] wonders if a belated birthday is where you're actually older than you think you are. ---- It's March of 2010. I've been asking around today, what sort of vision people have for Tcl, five years out. Alex Ferrieux says, "freeze as perfect, like /bin/sh" Colin says, "if I have any vision at all, it's about making Tcl smaller by making its innards more introspective and able to be implemented in Tcl. If you can generate bytecode from Tcl, then you can write all (or some, or many) of the compilation of Tcl commands to Bytecode *in* Tcl." Benefit is an advantage in speed of writing/improving the standard library. Drawback is a period of lack of stability in the core. [kbk] says, "It might be nice if there were some killer app (other than eggdrop) that captured the imagination enough to overcome Tcl's stigma. (Or some other marketing that would legitimate it.)" Cameron says, "*I* want IPv6, UDP, multi-core savvy, more introspection, and graphics-stuff-that-I-don't-know-how-to-say-briefly. Enough consensus that I don't have to hear again about how good A Real Repository would be." And then Cameron adds, "Were Steve around, should I ask for an improved test suite for Tk in the Five-Year Fantasy? It *has* improved; how far remains to go?" Gerry would like for jacl to keep up with Tcl development, and Aejaks or descendant or godchild is one of the regularly used tools for development of interactive web sites. Also, he would prefer not to have to recode things to fit them into the version of Tcl currently in jacl (for some strange reason that seems to bother me more than converting from Tk to the Aejaks widgets and geometry managers). And he would be much happier if echo2 did not have problems with some kinds of server clustering (which is still unresolved). [SEH]: Are you soliciting further comment? [hat0] Heck yeah! Please, speak your mind, I'd love to hear your thoughts/ideas. ---- **A five-year plan for Tcl** These are my thoughts about things that could improve tcl, or various different directions tcl could go, so that it is not just another language in 2015, but a language of growing importance/relevance. ''Note to anyone who sees this page: this isn't supposed to be another cloverfield or "tcl 9"-styled effort. I'm not interested in telling anyone what to do, or trying to force my ideas onto anyone. This is my own personal brainstorm, and all are welcome to add to it.'' ***Syntax and syntactic sugar*** [Lars H]: In my opinion, the syntax ideas are red herrings. If something is awkward to code, then it's typically better to come up with a new [little language] to help with the task than to complicate the base syntax (which should probably be frozen as perfect). [hat0] It sounds to me like you are in agreement with Alex Ferrieux, in saying, "present-day Tcl is perfect/almost perfect," and that if present-day Tcl doesn't do it right, don't use Tcl (i.e. invent your own little language, or write it in C). While this is certainly a fair attitude to have, and I can sympathize to some extent with it, I guess I don't agree. I don't think that the syntax is perfect, just familiar. [Lars H]: Just to clarify, I believe "invent you own little language" is an important part of [the Tcl way] — don't require one language to suit all problems, instead make it easy to mix and match. Core features to make it easier to at the script level create little languages is an interesting area for future development. ****Some syntax- or command-level concurrency (e.g. a foreach that parallelizes the body?)**** [AMG]: Parallel [foreach]? I'd leave foreach and friends alone, instead introducing a pair of commands to (1) execute a script in a separate thread and (2) execute a script and collect information from or about the threads spawned inside that script. Further levels of nesting, inside the parent thread or in the newly spawned threads, can make this confusing. So this is a truly half-baked idea. [accumulate and collect] is a possible source of inspiration. [hat0]: #2 seems like a good option if it could be made to parallelize the usual constructs very simply, maybe even as simply as prepending the command name to your ordinary command, like so: foreach x list { bleee } -> runmany foreach x list { bleee } ... As for further levels of nesting getting confusing, sure, but just don't do that. :) [Lars H]: The idea of a parallel [foreach] is interesting, but probably not practical. Loop-level parallellisation is rather fine grained, so one would probably want to get the impression of having all threads seeing the same interpreter, as opposed to the current situation where each thread has a separate interpreter. Rewriting Tcl's internals to handle concurrency is going to be ''difficult'' (e.g., it would as I understand it be necessary to move away from hash tables). [Coroutine]s could be a first step in this direction however (i.e., separate iterations of the parallel loop could experience the same level of separation as coroutines do). Looking at it another way, it is rather likely that anything that would benefit from loop-level parallellisation would first benefit from having its body rewritten in C, and then it's rather the C code that one needs to parallelise. [hat0]: I cringe whenever I see "write it in C" as the recommended course of action. It feels like a copout to me. The developers of other languages/environments seem much less willing to go to C (e.g. Hiphop for PHP, Pyrex for Python, LuaJIT) for performance, seeking out scripting-level performance increases first. As for it being difficult, sure, I don't doubt it, but I'm not really worried about implementation details right now. [SEH]: do [coroutine]s provide part of what you're looking for? [AMG]: Coroutines share an interpreter but do not run simultaneously. Threads run simultaneously but do not share an interpreter. hat0 is looking for simultaneous execution, so I don't know how coroutines would help. Coroutines offer interleaved execution; that's probably the best way to describe it. [hat0]: That is correct. Also, I'm interested in a very simple way to access and manage the simultaneous execution (such as a single command prepended to the looping construct). That is to say, if the concurrency is achieved through a mechanism that requires some auxiliary code for 90% of use cases, but allows the remaining 10% to happen at all, I'd rather scrap the 10% and auxiliary code and make the 90% accessible. [MS]: does Tcl have to care if your parallel [foreach] body has side effects? Like adding or removing commands, modifying some global variable, writing to a channel. I wish we had a way to insure that some script does nothing of the sort, would come in handy elsewhere too. [hat0]: No, I don't think it has to care. Adding or removing commands seldom happens in the lifetime of a program, and almost certainly would never happen in a parallelized script body. Same for writing to a channel, since you'd probably want your writes to be ordered. Maybe some additional sort of syntax could indicate "this script is side-effect free", like maybe using double braces instead of single braces for a script body: foreach item $list {{ # no side-effects here }} Maybe some kind of modified syntax like that could also be used to tell the bytecode interpreter, "disregard all traces in here," and so on. [CMcC] thinks, if you're waiting for the core to do this, you're missing something ... this kind of facility is doable, immediately, in pure tcl, right now. Why wait? It's an afternoon's work to have this happen, and the only possible objection is performance, to which I say (a) I'd bet performance is dominated by interthread stuff, (b) beware premature optimisation, (c) this is what a TAL would be good for. ****Some syntactic sugar on "expr"?**** [AMG]: I support [expr] sugar because expr is a critical command that is often misused (see: [Brace your expr-essions]). The sugar would be the recommended way to do math, and the expr command would be retained for situations where the math expression is intentionally the result of substitution. This closely parallels the [[[set] var]] versus $var situation: one-argument set is useful when the variable name is computed, and (for both one- and two-argument set) it's a common bug to prefix the variable name with $ when it doesn't need one. [hat0]: Nice parallel/explanation! ****Extend the {*} operator in various discussed ways ({n} as sugar for "lindex", etc)**** [AMG]: {n} for [lindex]? What forms can n take? Is it anything that's a valid index argument to lindex? Is arbitrary substitution allowed? Is it a math expression? ''(In my opinion, Cloverfield naming of list elements is a cleaner and more flexible alternative.)'' [hat0]: I'll take another look at Cloverfield. My first thought is that n is limited to variable interpolation, no script execution, and n can specify a number, range, or combination of them, e.g. {1} or {2-end} or {1, 3-end}. [AMG]: Many of my views on Cloverfield aren't published on this wiki, so I'll write a little bit on references. It's great to be able to substitute in the such-and-such element of a list, a dict, or a nested mix, but it's also important to be able to identify that element to a command like [[[set]]]. (This way [[set]] subsumes [[[dict set]]], [[[lset]]], etc.) [FB] and I differ on the notation for identifying an element without actually substituting in its value. I think avoiding expr-like performance and security problems requires a special notation for naming variables or elements thereof. When the interpreter sees this notation (leading &), it enters variable parse mode, but instead of immediately substituting in the value (leading $), it constructs a Tcl_Obj to contain the parse tree information. Later, the command may pass that reference object to a function like Tcl_ObjSetVar2(), which processes the parse tree embedded in the reference object to find the actual named element. For example: ====== set &num (en (zero one two) es (cero uno dos) fr (zéro un deux)) set &num(de) (null eins zwei) set &num(en){end+1} three set &var num set &key fr set &idx 2 set &"$var"($key){$idx} # returns "deux" puts $"$var"($key){$idx} # prints "deux" ====== I put this information here as a possible source of inspiration. Take from it what you will. [hat0]: Interesting, very interesting. I like the idea of references, and of claiming the & punctuation to note them. The last two examples seem a little punctuation-soup-ish to me, but I could get used to it, especially considering the flexibility presented. [AMG]: [C++] uses & in a variable or parameter declaration to signify that it be an indirection that is transparently constructed and dereferenced. [C] uses & and * in an expression to explicitly construct and dereference an indirection. My idea is a mixture of the two: construction is explicit by putting & in the expression, and dereferencing is transparent. I also have an idea for pointers which have to be explicitly dereferenced, so they have a string representation that is independent of the pointed-to value. My recollection is getting fuzzy here, but maybe there was also another difference: references perhaps referred directly to the value (e.g. a variable name is a reference), but pointers referred to the variable name (they're capable of dangling, and they don't affect reference counts). I think my pointer notation was to construct with a leading @ (instead of $) and to dereference with a trailing @, which could be intermixed with other access notations. ====== set &langs (@num(en) @num(es) @num(fr) @num(de)) puts $langs{3} # prints "@num(de)" puts $langs(3)@ # prints "null eins zwei" puts $langs(3)@{1} # prints "eins" ====== As for punctuation soup, I was just showing off a variety of methods of indirection, including having the variable name being the result of substitution. This flexibility makes it possible to always use $ substitution instead of single-argument set. [hat0]: Sure, I didn't mean to suggest that it was gratuitous, just that I personally would probably err on the side of simplicity and reduced expressiveness. [CMcC] would rather see {X} where X is some arbitrary identifier, representing a command in some scope (maybe tcl::readmacros) run at parse-time. Completely arbitrary read macros! ****Deprecate now and eventually remove arrays (confusing, unnecessary, dicts are better, etc)**** [AMG]: I have a question about removing support for arrays. Arrays support traces on individual elements, whereas dicts do not. Do you have any thoughts on this? Do we need a means for attaching traces to dict elements? That can get really hairy really fast. I'll summarize the difference between arrays and dicts: an array is a quasi-variable that is a collection of variables, whereas a dict is a value which is itself a collection of values. (I say "quasi-variable" because arrays don't have all properties of variables; for instance, they don't have values.) [hat0] Agreed, traces on arrays/array elements are useful. I'd suggest extending traces to work on dicts/dict elements. [AMG]: This is a fundamentally different kind of trace, since it's attached to a value rather than a variable. Variables are named and distinct, values are anonymous and shared. But obviously you don't want the trace to be on all shared copies of a value; maybe attaching a trace causes it to become unshareable, as if you had modified it. However, this seems messy and wasteful of memory. And in order to create the trace and to usefully process a traced event, you'd need a way of naming values, despite them being anonymous in principal. The only way I can think of is to special case the data types you'd like to support ([dict] and "top-level" variable values, but consider [list]s as well). Then the name (or a reference thereto) would have to be placed in the Tcl_Obj struct. ''(By the way, Cloverfield, or at least my vision of it (I've been out of contact with [FB], so we've surely diverged), includes the ability to name list and dict elements.)'' [hat0]: How does Jim do it, do you know off-hand? I seem to recall that Jim does use dicts for arrays. There's probably a solution that, at the outset, seems inelegant but in practice works out all right. [AMG]: [Jim] doesn't have traces. [hat0]: Ah ha. [CMcC] doesn't think deprecating array is so good, or so important. Leave it be. ****[AMG]: Alternative syntax to [[[list] a b c]] for constructing lists**** [hat0]: you mean a syntactical equivalent to the functionality of the list command (with expected substitutions and all)? [AMG]: Yeah, I had in mind using matched parentheses as a quoting mechanism which preserves word boundaries. The trouble is that it wouldn't produce a pure list, because both a string and list internal representation would be generated. Maybe it would be okay to drop the string representation. I don't know; I never asked [FB] why he wanted to preserve the string representation. [hat0]: I like it. I'm not too keen on worrying about internals right now.. from a programmer's perspective, some syntactical sugar on [[list]] is nice. [AMG]: This could facilitate resurrection of TIP 251 [http://tip.tcl.tk/251], minus the hack described in the "Incompatibilities" section which should stay dead. ****Some syntactic sugar to enable dicts to be treated like structs (e.g. '''set value $dict.key''')**** [SEH]: You may find Tclx's [keyed list]s interesting. [hat0]: Yes, so true, the keyed lists are a good approach, given the syntax as it is now. I'm not afraid to add another syntax rule (placing an additional restriction on variable names/dict key names), though, in order to gain a direct access to the dict value for a given key. Simplifying access to values within dicts--that is, treating them like values like named variables--is a good thing in my opinion. [CMcC]: +1 to this! I think making $ understand [dict] as well as [array] would be good. I note some conversation on the chat about it. $d(x)(y) for [[dict get $d x y]] when d is a scalar, otherwise treat it as array already does. I also think this is a subset of the idea of making $ always call a tcl::set command for its function. ****Syntax for anonymous functions**** [hat0]: ''Not really sure about this one yet..'' [apply] is nice. How could anonymous functions be worked more thoroughly into the language? Maybe a new operator that would facilitate this? So that, for example, the '''map''' example on the [apply] man page could be written as a one-liner: [[*]]{expr {$x**2 + 3*$x - 2}} x -4 -3 -2 -1 0 1 2 3 4 [AMG]: You're asking for two different things. One is tighter integration with anonymous functions. The other is [list comprehension]s. These ought to be kept separate. ''List comprehensions:'' There's no need for special syntax. The apply man page shows how easy it is to make a command that does list comprehensions. Possibly you'd want something with the full generality of [foreach]; the wiki already has code, which can probably be improved by using apply. ''Anonymous functions:'' My idea is for [proc]s to be simply [lambda]s that are bound to variable names. The first word of a command would be the name of a variable bound to the proc (or compiled command, as the case may be). So the following two lines would be equivalent, for all values of "command" and "args": ====== command {*}$args apply $command {*}$args ====== This does ''not'' mean that the following is valid: ====== {{a b} {expr {$a + $b}} 1 2 ====== Since the apply equivalent would be as follows. Note the dollar sign after the word "apply". ====== apply ${{a b} {expr {$a + $b}} 1 2 ====== In other words, {{a b} {expr {$a + $b}} would be taken as a ''variable name''. Even with this "shorthand" for apply, the apply command itself would remain useful for cases where the lambda isn't bound to a variable name, local or otherwise. Kind of like my suggestion for expr shorthand. This means merging variables and procedures/commands into the same namespace, which is a major departure with many ups and downs. One "up" is local scoping of procedures, so they don't have to be given a global name, and they can go out of scope; imbuing procs with the properties of local variables gives you most of what you need when you ask for anonymous functions. Obviously the primary "down" is loss of compatibility with existing scripts. Anyway, to see the impact effect of this change, just mentally delete the word "apply" from the example we've been discussing: ====== proc map {lambda list} { set result {} foreach item $list { lappend result [lambda $item] } return $result } ====== This change doesn't stand on its own. To be really useful, it should be possible to give this revised map procedure the name of another procedure to execute. I think this would require references, described elsewhere on this page. ====== proc frob {x} { expr {$x ** 2 + 3 * $x - 2} } map &frob -4 -3 -2 -1 0 1 2 3 4 ====== A reference is needed, rather than just a name, because a name does not give sufficient context for the map procedure to find the object being named. We currently handle this situation by requiring that names be global or in the caller's scope, depending on the specific application. But there isn't a general solution at the moment. ****Syntax for iterators**** [hat0]: ''Really not sure about this one..'' ***TAL*** Enable tcl scripts to generate bytecode directly, then use that facility to permit any command to be compiled by such scripts, perhaps even to the extent of allowing a tcl script to override the current C compilation routine for built-in commands. Enables some new commands to be added without performance concerns. If you don't see that this would be all kinds of wonderful, I can't help it. ***How can tcl/Tk be improved for developers? how can this be a more developer-friendly system?*** ****Improve error reporting in bad "expr" expressions**** ****Improve error reporting in general**** [AMG]: One thing I've dreamed about is the ability to identify the origin of any given Tcl_Obj. Is it a literal found in a particular line of a source file? Is it the product of a concatenation that happened on a certain line? Was it generated by a command (e.g. [[[read]]] or [[[expr]]]) somewhere in the sources? Etc. Might be interesting. (Might be expensive!) [hat0]: Maybe Tcl_Obj-tracing is enabled/disabled via a pragma, so that the expense would be programmer-determined. [AMG] I imagine it would be expensive even when disabled, if the pragma had to be checked all the time. If it was a compile-time option, it would be expensive to test and maintain, and it would be almost useless in practice because it would never be enabled when you need it. [hat0]: Agreed that it wouldn't make sense as a compile-time option, but I'm not convinced it'd be too expensive as a run-time option. Then again, I don't know what all would be involved. ****Fix the unbalanced brace issue**** [AMG]: Fixing the unbalanced brace issue would require greatly increasing the complexity of the rules for determining whether or not any given brace counts toward the opening/closing brace count. See [Cloverfield - Parser] for code which skips braces inside double quotes and comments. ****Work alongside or replace javascript as an in-browser DOM-manipulation language?**** [AMG]: I'd love to have Tcl be native in all popular browsers! But I fear we missed that boat a long, long time ago. However, I will settle for having Tcl be native in the browser running on my computer. Here's a neat idea: if there's a Tcl+DOM plugin, then we can write Tcl/Tk applications that require or bundle the plugin and embed browsers in the GUI, and the script running outside the browser can interact with the script running inside the browser, and as a side effect users of the applications will have the plugin and will be able to access Web sites that make use of it. [hat0]: I don't think the ship has sailed. Or, I guess I'd say, there could be another one coming along. There's a ton of mindshare for JavaScript right now, but in the early 2000s there was a perl->php transition for doing server-side web programming. I'm not holding my breath for any sort of victory here, just acknowledging that anything's possible five years out. [CMcC]: this is a major win. There's a tcldom package for FF. You'll never get this into IE. FF now permits