Michael Schlenker has very succinctly stated the need for a rich standard library for Tcl in several usenet postings
He says:
Tcl has two competing groups in the discussion: The embedders that are concerned about startup time, library size, memory footprint and things like that on the one side, the "batteriers-included-huge-stdlib-in-the-core" fraction on the other side. Those targets are impossible to reach at the same time. So instead of making it a problem, try to create a virtue. a) Let the embedders factor out as much functionality as possible from the core into a modular stdlib b) Let the batteries group add as much useful functional as required into a modular stdlib
... and ...
- Integrated, cross platform build system (TEA or better)
- Useful documentation, probably doctools based
- Stubs enabled if at all possible
(sometimes hard or near impossible to achieve)
- Export their own stubs table if their functionality could be useful
- BSD license
- Provides fundamental extra functionality (Thread, msgcat, XML, ASN1,
TclX/Registry/ffidl/Twapi, KBK's localized clock, Datastructures, One or
more OO-Systems, RPC-Support, TLS/Crypto,... )To which I (davidw) would add:
- The package should have a maintainer. - I would prefer to have one and only one OO system. Experimentation in this area has gone on for long enough. We need to provide something practical that works, for the "get it done" crowd.
So, what shall we do to make this a reality?
Something along the lines of:
One of the things we will need to decide is what code to include directly in CVS, and which to import into the build system from other sources.
Having seen tcllib in action, I do like the idea of having the source collected in one place, with the possibility of accessing different modules, as in tcllib, to foster "cross-pollination" and interest in the common library as a whole (maintainers fix bugs in packages that aren't necessarily their own).
Anyway, just added this to keep the ball rolling.
davidw: please people, use comp.lang.tcl or something else to discuss this if you don't like the idea. Some of us do like it and are going to do something about it. We'd like to use this page to that end, without filling it up with your comments on other libs, Perl, or the like (which may be valid, but aren't really relevant to the task at hand). Thanks.
---
daapp: yet another library ...
Every advanced Tcl programmer has their own stdlib :(
Maybe it would be better to concentrate efforts on modules building system and network archive with automatic building and installation?
I meet with Tcl/Tk in 1999, before I meet with UNIX and Perl. Since this time I trace the evolution of Tcl and these observations disapoint me sometimes very much. I will try to explain why.
I differentiate a Language and a Toolkit. I think that Tcl is a very good Language: it has simple and consistent syntax, it is not overloaded with different features, but the best Toolkit is not Tcl, it's Perl. Why? Because it's simple (yes, simple as a Toolkit!) and very effective. What do I mean? I mean the next:
Development with Perl is very comfortable, despite of heavy and difficult Perl syntax.
I like to develop in Perl, but I would like to develop more using Tcl. But when I remember how much work I need to do to install all the modules I need :(, I often switch to Perl, too often. Only when I need to make GUI do I use Tcl.
I would like to call Tcl community to direct efforts not at yet another standard library, but at automation of the development process:
Look around, we have an interpreter at version 8 but we have no normal development environment (I don't mean IDE or RAD)! Success of the Language depends on the Toolkit very much, look at Perl for example. No Python, neither Ruby or other popular scripting language approach to Perl.
I know about Cantcl and ETEN. They both look like unused and dropped. Why? Maybe because no documents in the standard Tcl distribution describe how to use these archives.
I call to members of TCT: please direct your attention to creation and maintenance of a network repository and module building system; at the current stage of Tcl evolution it's more important for future of Tcl. More important than the Language itself now.
SC: CANTCL hasn't seen active development in a while mainly because I've not had time to work on it but also because no-one else seems to be interested. My enthusiasm might be enhanced if even one other person stepped forward to contribute to the project. I will work on it again when time permits at least for my own purposes, but that probably won't be until mid-year.
About libraries: I think there is no need in yet another library; the better idea is to split the current tcllib into smaller parts - network part, text processing part, ...
Small notes: smake can be a base for something like Makefile.PL in Perl. CriTcl can be a base to replace autoconf and automake which are too complex.
LV writes: so the intent of this page (other than the excellent comment by daapp) is that a parallel distribution to tcllib, for compiled extensions, is needed? Is that what is meant when the term "modular standard library" is used? And the intent there, I suppose, is to create a source library (or a binary library) which contains basically a set of extensions that those working on the project see as essential to most Tcl programming? And for the most part, the initial work is to be centered not on writing new extensions, but to work with the authors of existing code, where possible, to organize code into a single downloadable file that can be compiled. Is that a correct summary? And for those cases where extensions are already being maintained in CVS, some discussion about whether to move the code into this new project, or make use of the code where it is, needs to take place.
I guess I come back to daapp's comment, and ask this question:
Is the problem that people want to find entire CVS repositories and cannot find them? Is it that the code does not already reside in CVS or some other control mechanism? Or is the problem, as daapp discusses, that people want to be able to find a distribution point (ala CPAN) for releases of the code (preferably with some info as to dependencies)?
schlenk 19/04/2004 Your comments are noteworthy, but i think a bit too nearsighted.
Tcl has deficiencies in its handling of module/package distribution, updating and building. This manifests in symptoms like every extension under the sun uses a different build system, a different installer, and the lack of a central archive.
Take a look at the evolution of the current tcllib. A common installer is used, a common documentation format is used and a modular structure of the library is kept.Technically not far away would be features like installing only the network stuff, or only the text modules. It's basically there, SAK just needs the metadata and some lists of modules. Tcllib is nearly there, but is too small and to restricting, as it focuses just on the (important and easier) part of pure script modules and their critcl equivalents.
Tcl needs an network repository in the long term, but first it needs a common build system, documentation standards and an enhanced infrastructure library.
You say, Perl scripts are often some use statements and few lines of scripts and implicitly say tcl scripts are not. I doubt this. Many Tcl scripts are longer as code from libraries is duplicated, inlined, because it isn't standard and cannot be expected to just be there. In the Perl case you start the usual CPAN invocation and get the modules you want. In the Tcl case you start google or use the wiki and start hunting for the packages, fight with their build system, their installation and so on.
Think of a common library not as, one large .so/.dll that contains all the useful stuff in one binary. Would be nice if the build system enabled such mega-libraries for static linking, but that isn't the target. It is more like a conventional library for books, you have a well organized collection of extensions, a general catalogue, a lending scheme, notifications of major updates to the collection and some volumes like the thesauri and dictionaries that are always present for reference and help. Add a caring librarian to the picture, and you have a better picture.
As I was cited above, some extensions are fundamental building blocks enabling large fields of usage, providing services to other extensions via a stubs interface of their own. Those are precious gems that should be treated well, so they can shine and last for a long time.
In my point of view a common library has to be a reliable place providing those easy to build core packages. But on the other hand those packages shouldn't be treated as absolutly necessary, just as very common dependencies. It's the need for an intermediate place, a collection of extensions that would be highly welcome in the core for many, but harshly rejected in the core by embedders wanting a tiny and modular core.
I could imagine a structure for single modules like this (maybe it matches Jeff Hobbs TM proposal, i don't have a copy of that document):
* package code in c and tcl * package example code in tcl and/or c * package documentation * package support data (encodings, timezone tables, msgcats, ...) * package build configuration * package dependency map
The great goal would be to harmonize things like build systems, documentation formats and so on, but why not start with something Tcl is best. Glue different systems together to get a whole. Autoconf may be broken, but the TEA system basically is there and works for many extension. In the long term a Tcl based build system will probably evolve from such a collection, but it isn't the first step. Take TEA, require it as front end format, add some Tcl glue code creating the current package makefiles from a TEA system and get going. Don't waste a man year creating a brand new build system and experiencing no one using it afterwards because of design errors, user ignorance or whatever.
If a system is fragile but works, someone will use it. If a system is perfect but doesn't work yet, no one will use it. Better reuse code than create all current extension build systems from scratch.
LV: schlenk writes: "Take TEA, require it as front end format, ..."
There is no way to require a programmer to do anything. TEA has been available for a number of years. Many, perhaps most, use it. But even then, I know that I struggle to get extensions to build successfully.
schlenk This wasn't meant as harsh as it may sound. There is a way to require things, look at tcllib. It works in a way by consensus. The core of my suggestion is: Take a common front end, write some glue code to keep using the custom build systems out there for an arbitrary long transition period for those extension not already directly using the common front end, make the common front end better, so it works and is well documented. The two common frontends i see at the moment are TEA and Critcl.
Your right, building extensions is hard. But having a well maintained, well documented common build system helps extension authors create more robust build systems. I too have problems building extensions successfully in many cases, many of those come from inappropriate use or age of TEA configuration, hard coded assumptions about the build environment and other hacks.
LV The reason that tcllib works as well as it does is that there are people who act as gatekeepers to tcllib, who provide either assistance or actually change the code that gets turned over.
Note that I am not arguing against a well maintained documented common build system. I do, however, suggest that for such a thing to occur, people who want it badly enough are going to have to start writing / coding / etc. to see that it happens.
davidw Larry, I think it's understood that there will be work to do, which is what this page was about, in part.
LV Okay, then what needs to be written first? Would documenting TEA be the first step? Or is something needed before hands?
schlenk As others on the chat have mentioned, coordination with Jeff Hobbs would be good (especially as he is more or less the TEA maintainer).
I see some tasks on the road:
LV This looks like a great list of work to do. Anyone reading these pages who has experience working with TEA2, please feel free to contribute.
Peter Newman 21 April 2004: Break Tcl/Tk Up I don't know if this is relevant to the above, or whether or not it can be achieved at the same time, but IMHO Tcl/Tk has it's legs stuck in concrete due to the fact that it's all one big fat system. It needs to be broken up.
More specifically, every Tcl command (string, expr, set, etc) - and every Tk widget - should be a separate stand-alone package - that can be package required/whatever, or not, as the programmer sees fit. That way, people who want to bug-fix, enhance, or totally replace, some Tcl/Tk functionality can easily do so.
For example, I want background images for the text widget. Now if it were a small number of source code files that I could modify and re-compile (according to the simple-to-use, extremely-well-documented system you guys seem to be talking about above), this would be no problem.
But at the present moment it looks like 2000 years work to me. And that's just figuring out how the hell to even do it. You've even gotta spend that much time writing a TIP. And then wait 400 years while the Tcl Masters decide whether to adopt it or not. (You've even got to write a TIP if you want to submit bug-fixes or examples to the totally pathetic, crap Tcl/Tk documentation.) It's a joke - and the reason why Tcl/Tk is stuck in the mud, and going nowwhere.
Anyway, it'd make my day (and save Tcl/Tk), if we could break Tcl/Tk up (one separate, stand-alone, individually editable/re-compilable package or the like, per command/widget), along with what you're doing above.
Peter, that's 1) not going to happen and 2) this is not the appropriate place do discuss it either. Would you be so kind as to move the above text to another page? Not that the idea in itself is completely wrong, it's just not related to this standard library plan, and is something that's probably not likely in any case, so those of us working on this idea would appreciate a little less clutter. Thankyou, davidw.
Peter Newman 21 April 2004: Thanks David, and I don't mind obliging. But first can you explain why it's not part of the standard library plan. It seems to me to be the central feature.
My understanding is that the essense of Tcl is (from the Tcl manpage):-
1 Commands. A Tcl script is a string containing one or more commands. Semi-colons and newlines are commandseparators unless quoted as described below. Close brackets are command terminators during command substitution (see below) unless quoted. 2 Evaluation. A command is evaluated in two steps. First, the Tcl interpreter breaks the command into words and performs substitutions as described below. These substitutions are performed in the same way for all commands. The first word is used to locate a command procedure to carry out the command, then all of the words of the command are passed to the command procedure. The command procedure is free to interpret each of its words in any way it likes, such as an integer, variable name, list, or Tcl script. Different commands interpret their words differently.
In other words it's a system to allow programmers to program in C - and create tools or commands that can then be executed/interpreted either interactively or from a script.
So Tcl itself has no commands. set, unset and proc etc are just one set of possible commands that could be provided to provide script level programming functionality.
In this sense then, every Tcl command is an (optional) part of the standard Tcl library. So a standard Tcl library should include the currently existing Tcl commands (set, unset, proc, etc) - plus the extensions commonly provided as packages (plus of course the mechanisms for adding/subtracting commands and extension packages to/from this standard library).
So what's the problem with that? Surely it's what Tcl is (supposed to be) all about. And something that Tcl has wandered away from (for reasons I don't understand - and to it's detriment). And surely something that, if you're creating a standard library, is the #1 thing to fix.
schlenk Peter is right, in a rather radical sense. If someone did the work to split up Tcl it could fit in. But unless someone does it it will not happen. A near core standard library would be the correct place to put the parts.
LV going to the op to load hundreds of separate libraries, one for each command, would slow down smaller tcl scripts, wouldn't it? P.S. It doesn't take hundreds of years - or even months - to get a TIP passed. It does take, from what I can tell, someone serious enough about the effort to work out sufficient details in advance that the discussion period focuses on alternative methods of doing it, if possible, and then the time of one or two people to actual implement the change . The more of this work that someone is willing to do, the better the chance of the TIP passing. And for a TIP regarding a complete reorganization of the tcl core into dozens of libraries - one per command - seems to me to best be served by someone doing the work in advance to show people that the performance costs, and the maintenance improvements, are justiified.
SDW We don't need a library, we need a methodology, and by in large several good ones exist. If you follow the style guide, you produce code that works well and is somewhat predictable to debug. Package management is simple and elegant. You write a package, you call pkg_mkIndex, you copy package into your library. I have in-house libraries of dozens of packages with millions of lines of code.
What holds my libraries together is that I have developed a way for my various modules to communicate with one another. The problem is that every TCL developer seems to have a different idea of what a "standard" message passing system should look like. When I try to integrate someone else's code, I have to write a wrapper so my stuff knows how to talk with it. Some like to feed arguments as field value pairs. Others use TK like option flags. Still others insist on hard-codeing the arguments for their functions with no clue to the outside world which argument says what.
There are the folks who layer all of their commands in a namespace. There are those who create a single command containing a giant switch statement. Some packages just load. Other load, but need to run some sort of initialization code.
So rather than focus our energy on the code itself, we really need to work on community standards on how it should behave. I have some ideas. You all have some ideas. There are no right answers, we simply need to select one.
Peter Newman 24 April 2004: Specification Time It seems to me that the idea of a Standard Library is a good one - and one that a lot of Tcl'ers feel a need for. The only problem is the many different things involved in it, as detailed above. And while the downside of lots of people interested, is that every new person seems to add new aspects/issues to the problem - the upside is that Many hands make light work. In addition, many of the issues that have been raised probably have existing solutions. It's just a case of putting it all together.
It seems to me that what we need at this stage is a Tcl Standard Library Specification - that describes this whole Standard Library - and that hopefully resolves all the issues raised above (and any new issues that will undoubtedly come up).
The problem is how to create that spec - especially since IMHO it should be co-operatively produced - and there should be some sort of democratic/concensus approach as to the final solution it describes - and that we agree on and adopt.
Anybody got any ideas as to how we can all work together to create that spec. And work together to firstly define - and secondly create - this Tcl Standard Library?
Peter Newman 24 April 2004: See Tcl Standard Library Specification.