Version 21 of Tcl Common Library

Updated 2004-04-20 20:42:34

Michael Schlenker has very succinctly stated the need for a rich standard library for Tcl in several usenet postings

http://groups.google.com/groups?q=g:thl2123827996d&dq=&hl=en&lr=&ie=UTF-8&oe=UTF-8&safe=off&selm=c5mb9q%242pd6o%241%40ID-102549.news.uni-berlin.de

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&safe=off&selm=c5n813%243913h%241%40ID-102549.news.uni-berlin.de

He says:

 Tcl has two competing groups in the discussion: The embedders that are 
 concerned about startup time, library size, memory footprint and things 
 like that on the one side, the 
 "batteriers-included-huge-stdlib-in-the-core" fraction on the other side.

 Those targets are impossible to reach at the same time.

 So instead of making it a problem, try to create a virtue.

 a) Let the embedders factor out as much functionality as possible from 
 the core into a modular stdlib

 b) Let the batteries group add as much useful functional as required 
 into a modular stdlib

... and ...

 - Integrated, cross platform build system (TEA or better)
 - Useful documentation, probably doctools based
 - Stubs enabled if at all possible
    (sometimes hard or near impossible to achieve)
 - Export their own stubs table if their functionality could be useful
 - BSD license
 - Provides fundamental extra functionality (Thread, msgcat, XML, ASN1, 
 TclX/Registry/ffidl/Twapi, KBK's localized clock, Datastructures, One or 
 more OO-Systems, RPC-Support, TLS/Crypto,... )

To which I (davidw) would add:

 - The package should have a maintainer.
 - I would prefer to have one and only one OO system.  Experimentation in this area has gone on for long enough.  We need to provide something practical that works, for the "get it done" crowd.

So, what shall we do to make this a reality?

Something along the lines of:

  • Draw up some guidelines using this page.
  • Create a SF project
  • Decide on initial packages
  • Add code to CVS
  • Beat compile system into shape

One of the things we will need to decide is what code to include directly in CVS, and which to import into the build system from other sources.

Having seen tcllib in action, I do like the idea of having the source collected in one place, with the possibility of accessing different modules, as in tcllib, to foster "cross-pollination" and interest in the common library as a whole (maintainers fix bugs in packages that aren't necessarily their own).

Anyway, just added this to keep the ball rolling.


davidw: please people, use comp.lang.tcl or something else to discuss this if you don't like the idea. Some of us do like it and are going to do something about it. We'd like to use this page to that end, without filling it up with your comments on other libs, Perl, or the like (which may be valid, but aren't really relevant to the task at hand). Thanks.

---

daapp: yet another library ...

Every advanced Tcl programmer has their own stdlib :(

Maybe it would be better to concentrate efforts on modules building system and network archive with automatic building and installation?

I meet with Tcl/Tk in 1999, before I meet with UNIX and Perl. Since this time I trace the evolution of Tcl and these observations disapoint me sometimes very much. I will try to explain why.

I differentiate a Language and a Toolkit. I think that Tcl is a very good Language: it has simple and consistent syntax, it is not overloaded with different features, but the best Toolkit is not Tcl, it's Perl. Why? Because it's simple (yes, simple as a Toolkit!) and very effective. What do I mean? I mean the next:

  • standard automated mechanism to create new modules and it's distribution (h2xs for example)
  • one network archive with all available modules
  • standard mechanism to work with this archive (upload, search, download and build with dependencies)
  • often small Perl application consist of some use pragma and few lines of code

Development with Perl is very comfortable, despite of heavy and difficult Perl syntax.

I like to develop in Perl, but I would like to develop more using Tcl. But when I remember how much work I need to do to install all the modules I need :(, I often switch to Perl, too often. Only when I need to make GUI do I use Tcl.

I would like to call Tcl community to direct efforts not at yet another standard library, but at automation of the development process:

  • build module system
  • central network archive
  • easy installation module procedure

Look around, we have an interpreter at version 8 but we have no normal development environment (I don't mean IDE or RAD)! Success of the Language depends on the Toolkit very much, look at Perl for example. No Python, neither Ruby or other popular scripting language approach to Perl.

I know about Cantcl and ETEN. They both look like unused and dropped. Why? Maybe because no documents in the standard Tcl distribution describe how to use these archives.

I call to members of TCT: please direct your attention to creation and maintenance of a network repository and module building system; at the current stage of Tcl evolution it's more important for future of Tcl. More important than the Language itself now.


SC: CANTCL hasn't seen active development in a while mainly because I've not had time to work on it but also because no-one else seems to be interested. My enthusiasm might be enhanced if even one other person stepped forward to contribute to the project. I will work on it again when time permits at least for my own purposes, but that probably won't be until mid-year.


About libraries: I think there is no need in yet another library; the better idea is to split the current tcllib into smaller parts - network part, text processing part, ...

Small notes: smake can be a base for something like Makefile.PL in Perl. CriTcl can be a base to replace autoconf and automake which are too complex.


LV writes: so the intent of this page (other than the excellent comment by daapp) is that a parallel distribution to tcllib, for compiled extensions, is needed? Is that what is meant when the term "modular standard library" is used? And the intent there, I suppose, is to create a source library (or a binary library) which contains basically a set of extensions that those working on the project see as essential to most Tcl programming? And for the most part, the initial work is to be centered not on writing new extensions, but to work with the authors of existing code, where possible, to organize code into a single downloadable file that can be compiled. Is that a correct summary? And for those cases where extensions are already being maintained in CVS, some discussion about whether to move the code into this new project, or make use of the code where it is, needs to take place.

I guess I come back to daapp's comment, and ask this question:

Is the problem that people want to find entire CVS repositories and cannot find them? Is it that the code does not already reside in CVS or some other control mechanism? Or is the problem, as daapp discusses, that people want to be able to find a distribution point (ala CPAN) for releases of the code (preferably with some info as to dependencies)?


schlenk 19/04/2004 Your comments are noteworthy, but i think a bit too nearsighted.

Tcl has deficiencies in its handling of module/package distribution, updating and building. This manifests in symptoms like every extension under the sun uses a different build system, a different installer, and the lack of a central archive.

Take a look at the evolution of the current tcllib. A common installer is used, a common documentation format is used and a modular structure of the library is kept.Technically not far away would be features like installing only the network stuff, or only the text modules. It's basically there, SAK just needs the metadata and some lists of modules. Tcllib is nearly there, but is too small and to restricting, as it focuses just on the (important and easier) part of pure script modules and their critcl equivalents.

Tcl needs an network repository in the long term, but first it needs a common build system, documentation standards and an enhanced infrastructure library.

You say, Perl scripts are often some use statements and few lines of scripts and implicitly say tcl scripts are not. I doubt this. Many Tcl scripts are longer as code from libraries is duplicated, inlined, because it isn't standard and cannot be expected to just be there. In the Perl case you start the usual CPAN invocation and get the modules you want. In the Tcl case you start google or use the wiki and start hunting for the packages, fight with their build system, their installation and so on.

Think of a common library not as, one large .so/.dll that contains all the useful stuff in one binary. Would be nice if the build system enabled such mega-libraries for static linking, but that isn't the target. It is more like a conventional library for books, you have a well organized collection of extensions, a general catalogue, a lending scheme, notifications of major updates to the collection and some volumes like the thesauri and dictionaries that are always present for reference and help. Add a caring librarian to the picture, and you have a better picture.

As I was cited above, some extensions are fundamental building blocks enabling large fields of usage, providing services to other extensions via a stubs interface of their own. Those are precious gems that should be treated well, so they can shine and last for a long time.

In my point of view a common library has to be a reliable place providing those easy to build core packages. But on the other hand those packages shouldn't be treated as absolutly necessary, just as very common dependencies. It's the need for an intermediate place, a collection of extensions that would be highly welcome in the core for many, but harshly rejected in the core by embedders wanting a tiny and modular core.

I could imagine a structure for single modules like this (maybe it matches Jeff Hobbs TM proposal, i don't have a copy of that document):

 * package code in c and tcl
 * package example code in tcl and/or c
 * package documentation
 * package support data (encodings, timezone tables, msgcats, ...)
 * package build configuration
 * package dependency map

The great goal would be to harmonize things like build systems, documentation formats and so on, but why not start with something Tcl is best. Glue different systems together to get a whole. Autoconf may be broken, but the TEA system basically is there and works for many extension. In the long term a Tcl based build system will probably evolve from such a collection, but it isn't the first step. Take TEA, require it as front end format, add some Tcl glue code creating the current package makefiles from a TEA system and get going. Don't waste a man year creating a brand new build system and experiencing no one using it afterwards because of design errors, user ignorance or whatever.

If a system is fragile but works, someone will use it. If a system is perfect but doesn't work yet, no one will use it. Better reuse code than create all current extension build systems from scratch.


LV: schlenk writes: "Take TEA, require it as front end format, ..."

There is no way to require a programmer to do anything. TEA has been available for a number of years. Many, perhaps most, use it. But even then, I know that I struggle to get extensions to build successfully.


schlenk This wasn't meant as harsh as it may sound. There is a way to require things, look at tcllib. It works in a way by consensus. The core of my suggestion is: Take a common front end, write some glue code to keep using the custom build systems out there for an arbitrary long transition period for those extension not already directly using the common front end, make the common front end better, so it works and is well documented. The two common frontends i see at the moment are TEA and Critcl.

Your right, building extensions is hard. But having a well maintained, well documented common build system helps extension authors create more robust build systems. I too have problems building extensions successfully in many cases, many of those come from inappropriate use or age of TEA configuration, hard coded assumptions about the build environment and other hacks.


LV The reason that tcllib works as well as it does is that there are people who act as gatekeepers to tcllib, who provide either assistance or actually change the code that gets turned over.

Note that I am not arguing against a well maintained documented common build system. I do, however, suggest that for such a thing to occur, people who want it badly enough are going to have to start writing / coding / etc. to see that it happens.


davidw Larry, I think it's understood that there will be work to do, which is what this page was about, in part.


LV Okay, then what needs to be written first? Would documenting TEA be the first step? Or is something needed before hands?


schlenk As others on the chat have mentioned, coordination with Jeff Hobbs would be good (especially as he is more or less the TEA maintainer).

I see some tasks on the road:

  • Documenting TEA
  • Documenting and establishing "Best Practices for using TEA"
  • Selecting and documenting a set of criteria for extensions (that could be the first) to be included in a TCL.
  • Formulating some contributors guideline, something like: What has to be done, so your extension can be included.
  • reusing stuff from tcllib as far as possible (doctools, installer/sak, critcl bindings to build c helpers)
  • checking how this fits with the soon to be proposed Tcl Module system.
  • checking how this stays modular and plays well with binary archives (starchive, Cantcl, eten, whatever)
  • setting up a project on sourceforge
  • getting accustomed to their compile farm and doing automatic builds there
  • deciding how the library wants to include foreign extensions, import releases via cvs import as vendor branch, take from cvs/url directly and patch like rpm does
  • ...

LV This looks like a great list of work to do. Anyone reading these pages who has experience working with TEA, please feel free to contribute.


Peter Newman 21 April 2004: Break Tcl/Tk Up I don't know if this is relevant to the above, or whether or not it can be achieved at the same time, but IMHO Tcl/Tk has it's legs stuck in concrete due to the fact that it's all one big fat system. It needs to be broken up.

More specifically, every Tcl command (string, expr, set, etc) - and every Tk widget - should be a separate stand-alone package - that can be package required/whatever, or not, as the programmer sees fit. That way, people who want to bug-fix, enhance, or totally replace, some Tcl/Tk functionality can easily do so.

For example, I want background images for the text widget. Now if it were a small number of source code files that I could modify and re-compile (according to the simple-to-use, extremely-well-documented system you guys seem to be talking about above), this would be no problem.

But at the present moment it looks like 2000 years work to me. And that's just figuring out how the hell to even do it. You've even gotta spend that much time wroting a TIP. And then wait 400 years while the Tcl Masters decide whether to adopt it or not. It's a joke - and the reason why Tcl/Tk is stuck in the mud, and going nowwhere.

Anyway, it'd make my day (and save Tcl/Tk), if this could be achieved along with what you're doing above.


Category Dev. Tools