Version 73 of Tcllib

Updated 2004-07-23 19:32:17 by AK

Purpose: to discuss the tcllib, uses, got-chas, etc. Most of this document is for "insiders". Newcomers might find "How to start with tcllib" useful.


Location: http://sourceforge.net/projects/tcllib/ is the official home. Additionally ftp://www.tcl.tk/pub/tcl/nightly-cvs/ is a source for snapshots of the library. A mailing list is maintained at sourceforge for reporting bugs. Manual pages for each module are available through http://tcllib.sourceforge.net/doc/index.html .

This package (and everything it contains) is part of the ActiveTcl Batteries Included distribution.


The sf project http://tcllib.sf.net/ is also home for efforts such as:


tcllib is a distribution of several packages for Tcl, all written entirely in Tcl. Here is a summary of (some of) the packages in the tcllib (version 1.6 has been released in early 2004) distributions and (some of) the commands each package provides:

  1. base64 - ::base64::decode, ::base64::encode, ::uuencode::encode, ::uuencode::decode, ::uuencode::uudecode, ::uuencode::uuencode, yencode::encode, yencode::decode, yencode::ydecode, yencode::yencode
  2. calendar - calendar::CivilYearToAbsolute, calendar::EFYWDToJulianDay, calendar::EYDToJulianDay, calendar::EYMDToJulianDay, calendar::EYMWDToJulianDay, calendar::IsLeapYear, calendar::JulianDayToEFYWD, calendar::JulianDayToEYD, calendar::JulianDayToEYMD, calendar::JulianDayToEYMWD, calendar::WeekdayOnOrBefore (see also tcllib calendar module)
  3. cmdline - ::cmdline::getArgv0, ::cmdline::getKnownOpt, ::cmdline::getKnownOptions, ::cmdline::getfiles, ::cmdline::getopt, ::cmdline::getoptions, ::cmdline::usage, ::cmdline::typedGetopt, ::cmdline::typedGetoptions, ::cmdline::typedUsage
  4. comm - ::comm::comm ::comm::comm_send
  5. control - ::control::assert, ::control::control, ::control::do, ::control::no-op
  6. counter - ::counter::count, ::counter::exists, ::counter::get, ::counter::histHtmlDisplay, ::counter::histHtmlDisplayBarChart, ::counter::histHtmlDisplayRow, ::counter::init, ::counter::names, ::counter::reset, ::counter::start, ::counter::stop,
  7. crc - ::crc::crc32, ::crc::sum, ::crc::cksum
  8. csv - ::csv::join, ::csv::joinlist, ::csv::read2matrix, ::csv::read2queue, ::csv::report, ::csv::split, ::csv::split2matrix, ::csv::split2queue, ::csv::writematrix, ::csv::writequeue,
  9. des - ::DES::des (not yet installed)
  10. dns - dns::address, dns::cleanup, dns::cname, dns::configure, dns::name, dns::reset, dns::resolve, dns::status, dns::wait,
  11. doctools - ::doctools::toc , doctools::idx , doctools::csv , doctools::changelog
  12. exif - exif::analyze exif::fieldnames
  13. fileutil - ::fileutil::cat, ::fileutil::find, ::fileutil::findByPattern, ::fileutil::foreachLine, ::fileutil::grep, ::fileutil::touch,
  14. ftp - ::ftp::Append, ::ftp::Cd, ::ftp::Close, ::ftp::CloseDataConn ::ftp::Command, ::ftp::CopyNext, ::ftp::Delete, ::ftp::DisplayMsg, ::ftp::ElapsedTime, ::ftp::FileSize, ::ftp::Get, ::ftp::HandleData, ::ftp::HandleList,ftp::HandleVar, ::ftp::HandleOutput, ::ftp::InitDataConn, ::ftp::List, ::ftp::ListPostProcess, ::ftp::MkDir, ::ftp::ModTime, ::ftp::ModTimePostProcess, ::ftp::Newer, ::ftp::NList, ::ftp::Open, ::ftp::OpenActiveConn, ::ftp::OpenControlConn, ::ftp::OpenPassiveConn, ::ftp::Put, ::ftp::PutsCtrlSock, ::ftp::Pwd, ::ftp::Quote, ::ftp::Reget, ::ftp::Rename, ::ftp::RmDir, ::ftp::StateHandler, ::ftp::Timeout, ::ftp::Type, ::ftp::WaitComplete, ::ftp::WaitOrTimeout, ftp::geturl
  15. ftpd - ::ftpd::config, ::ftpd::hasCallback, ::ftpd::logStderr, ::ftpd::fileAuth, ::ftpd::anonAuth, ::ftpd::unixAuth, ::ftpd::server, ::ftpd::accept, ::ftpd::read, ::ftpd::docRoot, ::ftpd::fs
  16. grammar_fa - Currently only available through CVS head. Provides operations on finite automatons.
  17. html - html::author, html::author, html::bodyTag, html::cell, html::checkbox, html::checkSet, html::checkValue, html::closeTag, html::default, html::description, html::description, html::end, html::eval, html::extractParam, html::font, html::for, html::foreach, html::formValue, html::getFormInfo, html::getTitle, html::h, html::h1, html::h2, html::h3, html::h4, html::h5, html::h6, html::hdrRow, html::head, html::head, html::headTag, html::if, html::init, html::init, html::keywords, html::keywords, html::mailto, html::meta, html::meta, html::minorList, html::minorMenu, html::openTag, html::paramRow, html::passwordInput, html::passwordInputRow, html::radioSet, html::radioValue, html::refresh, html::refresh, html::row, html::select, html::selectPlain, html::set, html::submit, html::tableFromArray, html::tableFromList, html::tagParam, html::textarea, html::textInput, html::textInputRow, html::title, html::title, html::urlParent, html::varEmpty, html::while,
  18. htmldoc - This is not a true module but the place where tcllib 1.3 installed the tcllib documentation in html format.
  19. ::htmlparse::parse, ::htmlparse::debugCallback, ::htmlparse::mapEscapes, ::htmlparse::2tree, ::htmlparse::removeVisualFluff, ::htmlparse::removeFormDefs,
  20. inifile - ::ini::open, ::ini::close, ::ini::commit, ::ini::sections, ::ini::keys, ::ini::value
  21. ip -
  22. irc - irc::config, irc::connection,
  23. javascript - javascript::BeginJS, javascript::EndJS, javascript::MakeMultiSel, javascript::MakeClickProc, javascript::makeSelectorWidget, javascript::makeSubmitButton, javascript::makeProtectedSubmitButton, javascript::makeMasterButton, javascript::makeParentCheckbox, javascript::makeChildCheckbox
  24. jpeg - to appear in the next release
  25. log - ::log::levels, ::log::logMsg, ::log::lv2longform, ::log::lv2color,::log::lv2priority,
  26. logger - ::logger::walk, ::logger::services, ::logger::enable, ::logger::disable (part of the log module)
  27. math - ::math::calculus, ::math::combinatorics, ::math::cov, ::math::fibonacci, ::math::integrate, ::math::interpolate, ::math::max, ::math::mean, ::math::min, ::math::optimize, ::math::product, ::math::random, ::math::sigma, ::math::statistics, ::math::stats, ::math::sum
  28. md4 - ::md4::md4, ::md4::hmac, ::md4::MD4Init, ::md4::MD4Update, ::md4::MD4Final
  29. md5 - ::md5::md5, ::md5::hmac, ::md5::test, ::md5::time, ::md5::<<<,
  30. md5crypt - ::md5crypt::md5crypt, ::md5crypt::aprcrypt
  31. mime - ::mime::initialize, ::mime::parsepart, ::mime::finalize, ::smtp::sendmessage
  32. multiplexer - [fill in the external interfaces]
  33. ::ncgi::reset, ::ncgi::urlStub, ::ncgi::urlStub
  34. ::nntp::nntp, ::nntp::NntpProc, ::nntp::NntpProc, ::nntp:;okprint, ::nntp::message,
  35. ::ntp::time
  36. png - to appear in the next release
  37. ::pop3::open, ::pop3::close, ::pop3::status,
  38. pop3d - pop3d::new
  39. ::profiler::tZero, ::profiler::tMark, ::profiler::stats, ::profiler::Handler, ::profiler::profProc, ::profiler::init
  40. ::rc4::rc4 - stream encryption
  41. ::report::report , ::report::defstyle, ::report::rmstyle,
  42. ::sha1::sha1, ::sha1::hmac
  43. ::smtpd::start, ::smtpd::stop, ::smtpd::configure, ::smtpd::cget
  44. snit - Snit's Not Incr Tcl - ::snit::type, ::snit::widget, ::snit::widgetadaptor
  45. soundex::knuth
  46. stooop - stooop::class, stooop::virtual, stooop::new, stooop::delete, stooop::classof
  47. [struct1] - Version 1 of struct (see below), provided for backward compatibility.
  48. ::struct::list, ::struct::graph, ::struct::matrix, ::struct::queue, ::struct::stack, ::struct::Tree, ::struct::record, ::struct::skiplist, ::struct::prioqueue, new: ::struct::sets
  49. ::textutil::adjust::adjust, ::textutil::expander, ::textutil::split::splitx, ::textutil::tabify::tabify, ::textutil::tabify::untabify, ::textutil::tabify::tabify2, ::textutil::tabify::untabify2, ::textutil::strRepeat, ::textutil::trim::trimleft, ::textutil::trim::trimright, ::textutil::trim::trim,
  50. ::uri::split, ::uri::join, ::uri::register, ::uri::geturl, ::uri::resolve, ::uri::canonicalize
  51. ::uuid::uuid generate, ::uuid::uuid compare
  52. stats no longer distributed
  53. devtools distributed, but not installed

A tklib module in the sourceforge tcllib project ( http://tcllib.sourceforge.net/ ) has begun, providing the structure for submitting script only tk add-on commands as well.


When Tcllib was originally released, the following was written:

In response to popular demand, the Tcl core group is introducing tcllib, a Tcl standard library. This meta-package will contain many modules, each of which is itself a standalone Tcl package. The intention is to provide commonly used functions and libraries, bundled together under a single license (BSD), and with no binary dependencies. This will encourage use and growth of the library. Once tcllib is downloaded and installed, users will be able to "package require tcllib" or "package require any_module_in_tcllib".

Releases of tcllib will be made mostly independently of releases of the Tcl/Tk core. This will allow a faster release cycle, which is important in the early stages of development. It is possible that future releases of the core will include a snapshot of the tcllib.

Initially, we have seeded the library with several modules, some written in-house at Scriptics and by the core group:

  * struct:    Tcl implementations of common data structures
  * profiler:  Function level Tcl profiler
  * [cmdline]:   getopt style command-line parser
  * base64:    base64 encoder/decoder
  * fileutil:  Tcl implementations of file utilities (grep, find, ...)
  * [pop3]:      pop3 client
  * math:      math functions (min, max, ...)

We want this to be a community-driven library, so we encourage submissions to this library. If you have a function or set of functions that you find you use frequently, send it in! Ultimately, we want community members to be responsible for modules in tcllib, having CVS write access directly to specific modules in the tree. Please do not feel that you must have a "complete" module in order to contribute -- individual functions are welcome as well.

In order to encourage growth and use of tcllib, there are some ground rules that modules added to tcllib must follow:

  • the module must be Tcl only: no C extensions. This minimizes the barrier to use -- not everybody has the means or desire to compile extensions. We may someday start another meta-package of C extensions.
  • The module must use a namespace for its commands and variables
  • The module must be a proper Tcl package
  • The name of the package must be the same as the name of the namespace
  • The module must be released under the BSD license. If you cannot accept the terms of that license, please feel free to release your package independently of the tcllib. We feel that this license (the same one that Tcl itself is under) is the most conducive to use.
  • The module should have both documentation (in XML or man form) and a test suite. The module _must_ have either documentation or a test suite, and preferably both.
  • The module should adhere to Tcl coding standards. See: http://www.purl.org/tcl/home/doc/
  • The module should be of general use to a large number of programmers. We don't want tcllib to become the dumping ground for every tcl programmer's personal, highly specialized functions.

We feel that these rules will help to encourage contribution to the library and use of the library. Remember that almost everybody can use the BSD license, and not everybody is willing or able to compile extensions. Our ultimate goal is to encourage the use of Tcl by providing a broad, robust library of functionality.

The current status of tcllib is open alpha. We would like users to check out the current module, and see what they could add. After some period of discussion (a couple of weeks), we'd like to settle on something for an initial 1.0 release.

With regards to stdtcl, tcllib supercedes that module. Some of the packages from stdtcl were moved into tcllib, some were not. Those that were not were left out because they did not follow one or more of the guidelines above. cgitcl, for example, is not in a namespace. ftpd lacks documentation and a test suite. tcldebugger is only really useful to people with tclpro debugger. Proper replacements for these would be welcome.

Eric Melski Scriptics Corporation


A follow up article by Jeff Hobbes said:

Several have asked some basic questions of tcllib, which I will address here. We will also be creating a FAQ and web space for it (just after we figure out the best way to do it on http://tcl.activestate.com/ ). If you respond to this, please change the subject line to something appropriate and trim down this (rather long) message for your points.

  • Who maintains the names?

The Tcl core group, with help from the community, will manage the toplevel package structure. This will prevent unlimited proliferation of names and confusing structure. We will also rely on a source like NICS:

    http://pitch.nist.gov/cgi-bin/nics/cgibin/domain_list.cgi

to make sure that uniqueness is maintained amongst all packages (those in tcllib and external ones).

Names should follow some reasonable/logical structure. Expanded math functions under ::math (like setmin, setmax), extended string functions under ::str (like (un)tabify, wrap_lines), etc. We'd like to make the 1.0 release have a good set of logical packages (namespaces).

  • How can software be submitted?

Modules (packages) within tcllib will be assigned maintainers (1+). Each maintainer will have CVS write access to that module. We would expect maintainers to follow all the basic style and quality standards for the tcllib, or risk losing CVS write privileges (not something we'd do without warning or good reason). If you want to become a module maintainer, you would just make the request to the Tcl Core Team.

  • What software can be submitted?

tcllib can be populated with any Tcl-only packages that adhere to the tcllib standards (built as package, all in one namespace). We don't want to include C packages, following basic KISS principles. Included packages may contain reqs for C-based extensions, but they must have a Tcl-only fallback. Also, any submitted software should be under BSD, so people know that they can pick this up and use it just as they do the Tcl/Tk core now.

  • What is the current availability?

For the moment, we are in the alpha state, where we have only established CVS access. The framework is not yet complete (like missing easy installers and such). For that, community input would be helpful. Also, as I already said, we want to ensure that the 1.0 release is fairly "package" complete, meaning most of the toplevel packages are there, even if sparsely populated. When ready, the package will be at the Scriptics FTP site. BTW, for those who want to give us better installers, a prereq assumption is that Tcl is installed, so you could use it in the installer.

  • What is the release schedule?

We'd like to make a 1.0.0 release sometime in April (arbitrary date chosen). After the 1.0 release, we would basically make checkpoint releases every couple of months or as a certain amount of new stuff comes in. The version management is planned as follows:

   #   Major version bump: reserved for when new toplevel packages are added
   #   Minor version bump: reserved for when new subpackages are added or significant functionality (like changed API) is made to existing packages
   #   Patch version bump: bug fixes to existing packages

We could go to a S.M.m.p system, where the new significant digit "S" refers to Structure, to allow ourselves versions in which we restructure tcllib.

  • Will it become part of the core?

Likely starting with 8.4, the core will include a snapshot release of tcllib, where tcllib maintains its own version number and is installed as a sibling to tcl<M.m>. tcllib would continue to be released separately.


A subsequent article said

 From: Dan Kuchler <[email protected]>
 Newsgroups: comp.lang.tcl
 Subject: Re: Math extensions to TCLlib
 Date: Sun, 05 Nov 2000 08:37:57 -0800
 Organization: Ajuba Solutions
 Message-ID: <[email protected]>
 References: <[email protected]>
 To: Morgan <[email protected]>

 Morgan wrote:
 >
 > [interesting proposed changes snipped]
 >
 > Also, are there any specific procedures for contributing to the
 > TCLlib?  Anyone I need to contact or specific software I need to use?
 > Would this go better in a sub-package of the math portion of the TCLlib?
 >

It's great to hear that you are interested in helping contribute to tcllib. I thought your proposed set of functionality sounded interesting.

Some folks have shown interest in having access to the functionality available through the NumPy package (a python package) in tcl.

This might give you ideas for things that people might want.

The NumPy package is at:

http://sourceforge.net/projects/numpy/

Currently tcllib is in the process of being moved to sourceforge where it will live in the tcllib project:

http://tcllib.sourceforge.net/

I think the process for volunteering to work on a project there is to contact an admin for a project. Currently Eric Melski has been organizing tcllib development, so it might be worthwhile to contact him ( [email protected] )

Currently the web page that talks about tcllib (a little out of date) is at:

http://www.purl.org/tcl/home/software/tcllib/

As far as guidelines for contributing, I think the current requirement is that things in tcllib must be able to be run in a tcl-only mode (i.e. no binary package dependencies), and the contributed code should have either tests or documents, but preferably will have both.

There are good contribution guidelines (that also contain a link to the Tcl Sytle Guidelines) at:

http://www.purl.org/tcl/home/software/tcltk/contributing.tml

(while that page is focused on contributing to tcl/tk, most of it is relevent to tcllib as well).

I hope that helps get you started!

--Dan


Info about reporting bugs relating to tcllib was posted:

I'm the primary maintainer of Tcllib, with lots of help from various people in the community who are responsible for different bits. The library now lives at:

http://tcllib.sourceforge.net/

You can submit a bug there. Be sure to include:

- your name - your email address (so you can be contacted when the bug is fixed) - a description of the behavior you expected to see - a description of the behavior you actually saw - the version of Tcl you are using - the version of tcllib you are using - the os you are using, and the version of the os - anything else you think might be relevant

A sample script demonstrating the problem is always very helpful.

   Eric Melski                            The Other Tcl Guy
   ericm at interwoven.com                Interwoven, Inc.


The README [L1 ] for tcllib reads, in part, :

RCS: @(#) $Id: 1246,v 1.74 2004-07-24 06:00:15 jcw Exp $

Welcome to the tcllib, the Tcl Standard Library. This package is intended to be a collection of Tcl packages that provide utility functions useful to a large collection of Tcl programmers.

The structure of the tcllib source hierarchy is:

 tcllib
 +- modules
     +- <module1>
     +- <module2>
     +- ...

The install heirarchy is:

 .../lib/tcllib
        +- <module1>
        +- <module2>
        +- ...

There are some base requirements that a module must meet before it will be added to tcllib:

  • the module must be a proper Tcl package
  • the module must use a namespace for its commands and variables
  • the name of the package must be the same as the name of the namespace
  • the module must reside in a subdirectory of the modules directory in the source heirarchy, and that subdirectory must have the same name as the package and namespace
  • the module must be released under the BSD License, the terms of which can be found in the toplevel tcllib source directory in the file license.terms
  • the module should have both documentation (in XML, man, or HTML form) and a test suite (in the form of a group of *.test files in the module directory).
  • the module must have either documentation or a test suite. It can not have neither.
  • the module should adhere to Tcl coding standards

When adding a module to tcllib, be sure to add it to the Makefile.in so it will be installed. Add a line like:

 MYNEWMODULE=mynewmodule

to the list of modules at the top of the Makefile.in, and then add $(MYNEWMODULE) to the definition of the MODULES variable. This will allow users to choose which modules to install by commenting or uncommenting lines in the Makefile.

Each module source directory should have no subdirectories (other than the CVS directory), and should contain the following files:

  • source code *.tcl
  • package index pkgIndex.tcl
  • tests *.test
  • documentation *.n, *.xml

If you do not follow this directory structure, the tcllib Makefile will fail to locate the files from the new module.


Cameron Laird has, at least in principle, sanction to comment and update tcllib. So far I've worked mostly with mime, base64, smtp, ftp, pop3, ... If you're having tcllib problems, do get in touch with me [L2 ]; it might be something I've already studied, without having yet checked in an update.


Q. What are the rules, requirements, restrictions for adding code to the tcllib?

Q. Who decides what does, or does not, get included into tcllib?

Q. Does anything get added into the library that someone submits?

Q. Can Tk specific commands be added to the tcllib?

Q. Can Expect/Tix/Snack/etc. specific commands, which depend on some outside extension but which themselves are only script based, be added?


From Andreas Kupries, this msg was provided:

1) What are the rules, requirements, restrictions for adding code to the tcllib?

See (5). Also note that a patch of an existing module or a new module should contain not only the code, but extend the testsuite and documentation (manpages) too. Both of the latter will make acceptance easier as they allow to check the correctness of the code now and in the future.

If there is trouble to write a manpage in *roff see

   Mike Hall <[email protected]>
   Steve Simmons <[email protected]>
   [email protected] (Joe English)

who volunteered to help Larry Virden with writing such. Maybe they are willing to help others too. Note that I (AK) am also able to write basic *roff now.

2) Who decides what does, or does not, get included into tcllib?

Essentially the people who are registered as developers for the Tcllib SF project.

3) Does anything get added into the library that someone submits?

Yes. Recent changes were the acceptance of a patch [x] fixing a bug in pop3 and several new modules ("md5" [*], "csv", "matrix" (as part of "struct"), "report", "log" and "htmlparse").

[x] Well, the patch was accepted a long time but Scott had troubles with committing it. When he tried yesterday it worked and so this bug finally squashed too.

[*] Based upon Don Libes implementation of MD5, "md5pure", with changes by me (Trf soft dependency).

4) Can Tk specific commands be added to the tcllib?

I personally would prefer to collect such in a "tklib" project.

5) Can Expect/Tix/Snack/etc. specific commands, which depend on some outside extension but which themselves are only script based, be added?

The main rule/restriction which was setup at the beginning was that 'tcllib' is a script-only library which should work without any C-level extension. So the answer to the above is 'no'.

This however does not exclude the usage of extensions to make tcllib faster, if it does work without the extensions too, albeit slower.

Examples of this are the "mime" and "md5" modules. Both have a soft-dependency on the C-level extension "Trf". If "Trf" is present, "mime" and "md5" will use it in their implementations to make certain operations faster (base64 and md5 respectively). If it is not present, both modules will fallback to pure-tcl implementations.


In general I (AK) would advise the following:

  • Go through your set of utilities and see if they match one of the existing modules of tcllib. If yes, submit a patch containing the new code and extended testsuite, documentation.
  • If there is no match create a new module in the style of the existing modules (see for example md5). Submit either a patch or an archive containing the new files.

tclguy has written, "From http://tcllib.sf.net/ go to the patch manager and post your code. You might want to examine the tcllib distro first, to see if it already fits well into an existing module, or whether one should be created for it."


See cgraph for a C version of the tcllib graph command.


Recently someone on the Tcllib mailing list (sponsered at SourceForge) asked What would one have to do to be added as a developer to Tcllib?. The answer by AK was One of the project administrators (like me) needs the name of your sourceforge user account to be able to add you to the list of developers for the project.

Other things mentioned in the recent thread were:

  • extending test cases when one fixes a bug
  • creating test suites if none exist
  • dropping a note to the tcllib-developers mailing list if any major features or changes are planned for a module, letting people know and making certain the direction being taken makes sense,
  • writing doctools documentation if none exists
  • follow the Tcl style Guide when writing or modifying code
  • don't add new features without adding new doc
  • bump revision numbers when changing files
  • on questions of design, or naming, ask the developers mailing list
  • be certain to specify the required level of Tcl using package require

25 May 2004 Has the tcllib team considered adding some form of Michael Cleverly's package, nstcl? http://michael.cleverly.com/nstcl/ While most functionality is duplicated in the current distribution of tcllib, the nsset datatype is a wonderful thing to have; an indexed array! index - key - value ]. His api has some commands to utilize it as well. As an aolserver tcl coder, I work with it extensively and for me it renders the array command obsolete.

   set id [ns_set create]
   %t2
   ns_set put $id fname Michael
   %0
   ns_set put $id lname Cleverly
   %1
   ns_set get $id fname
   %Michael

As documented above, it generally isn't a matter of some team determining whether some code is worthy to add. It is typically a matter of some interested party contacting the tcllib maintainers to work out the details of their adding a new module to the library. Why not contact Michael and ask if he would like to add his code to tcllib? Over the years, it has turned out to work better if the author is involved with this process, so that no misunderstandings, etc. occur.


[ Category Package | Category Mailing List ]

http://cvs.sourceforge.net/viewcvs.py/tcllib/tcllib/README?rev=1.8&view=auto