Where Tcl Needs Work/Workers

The basic question is valid for any lovers of the language. I just wanted to know where I could provide some useful contributions. Maybe others free the same way and so the page was written.

So let's break this question up:

  • Optimization for speed and/or memory use?
  • Language features that might have a large audience?
  • Improve the API/bindings to other languages to speed extensions and embedding?
  • Particular libraries to support X,Y,Z?
  • Clean up the code and/or build system?
  • Update/improve testing?

Personal observations and ideas

  • I don't like the code base, it breaks many of my personal rules on how to organize C code.
  • The C/API is hard to work with. I prefer a C++ API, but even with in C we could do a lot better.
  • The fact that the commands have built-in help was great, but I feel we could do so much better.
  • I always planned on speeding up the code by adding more memory options but never got the time. Using cache's (if done right) could speed the code a lot.
  • Also want to build a way to support personal libraries of code. Something like giving every user their own namespace and package on install, and easy way to manage it so an easy way to add/remove/change the code in it (maybe with versioning?).
  • More static analysis would be hard but good.

So what does the community think? I could just work on what interest me personally, but I thought I might let myself be guided at least a little but what the community think.

As far as what I can contribute, I will give you a very short resume and remember I'm not looking for a job.

  • Have a master's in Theoretical Atomic Physics UCF 1989
  • Have worked in a more languages, OS(s), and projects than I can to say.
  • Been heavy involved in C/C++/Tcl since 1990. Tcl has been my secret weapon in both testing and exploring ideas.
  • Done embedded work without any OS at all.
  • Worked on disassemblers and reverse engineering in X86,PowerPC,ARM,and a bunch of others.

Restatement of what I heard below in a slightly more organized way Earl Johnson

Tcl Feature Proposals Summary

This document summarizes all the feature suggestions from the Tcl community discussion. Each feature has a symbolic tag (for discussion) and a restatement of the idea.


Organization and Overhead Issues

Symbolic Name Description
ORGANIZE_WORK_GOALS Figure out how and where to continue this discussion. (new 4/16/25)
DO_WE_HAVE_SUPPORT Figure out what kind of support we can get for this effort. (new 4/16/25)
PRIORITY_LIST Can at least some groups of devs produce priority list of efforts. (new 4/16/25)
DEV_RESEARCH_TASK Maybe some people can just research some of these task. (new 4/16/25)

Core Language Improvements

Symbolic Name Description
TYPED_VAL Allow users to define semantic types for values (like typedef struct in C) to enforce model correctness and improve debugging.
NAMED_ARGS Support named arguments in proc definitions to improve readability and reduce reliance on position-based parameters.
ARGS_NOT_LAST Permit args to appear in the middle of an argument list, not only at the end.
STATIC_ALLOC Introduce ways to declare memory as statically allocated where predictable, for improved speed.
ANNOT_SYNTAX Expand syntax such as {*}{} with forms like {=}{} for expressions and argument annotation.
DEV_DOCS Need to document how some of the complicated parts of the code work to ease in new developers. (new 4/16/25)

Runtime & Performance Enhancements

Symbolic Name Description
CHANNEL_REDO Redesign the I/O channel subsystem to avoid repeated copying and enhance performance.
ENCODE_FAST Optimize common encoding/decoding routines to avoid multiple data scans.
STRING_FAST Improve performance of string operations (e.g., range, split) using techniques similar to list optimizations.
LIST_INPLACE Add more list commands that operate in-place (like lappend) to avoid unnecessary memory usage.
STRING_SEARCH_FAST Use faster algorithms (Boyer-Moore, KMP) for string first and similar searches.
TIMER_WHEEL Rework event/timer scheduling (e.g., with a timer wheel) to reduce latency and overhead.
INTERP_FAST Speed up creation of safe interpreters to make them more usable in performance-sensitive scenarios.
CALLOUT_SPEED Reduce overhead when calling C-implemented commands from bytecode.
BYTECODE_RT_FAST Improve bytecode runtime performance—under exploration by Donal.
STATIC_CACHE Add support for predictable caching mechanisms to improve speed for repeated tasks.

Developer Tools & Diagnostics

Symbolic Name Description
STATIC_ANALYSIS Improve or update static analysis tooling, including the Tcl Dev Kit (e.g., tclchecker).
DEBUGGER Provide a modern interactive debugger (e.g., revive or improve ProDebug).
FORMATTER Build or refine a Tcl formatter/pretty-printer that aligns with style guidelines.
TREE_SITTER Implement a TreeSitter grammar for Tcl for better syntax highlighting and tooling.
LSP_SERVER Create a Tcl Language Server Protocol (LSP) implementation for IDE integration.

API / Shell / Scripting Enhancements

Symbolic Name Description
CMD_LINE_TCLSH Support executing Tcl code directly from the command line (e.g., tclsh -e).
SHELL_HISTORY Add readline-style history and popup completion to tclsh.
ASYNC_SHELL Add event and background task awareness to tclsh.
HELP_SYSTEM Integrate a help system similar to TIP 194 with built-in command documentation.
BETTER_EXEC Replace or revise exec to handle Unicode, errors, and cross-platform issues more gracefully.
GLOB_ALL Add -all flag to glob to return hidden and non-hidden files in one go.
NAMESPACE_TOOLS Introduce namespace-aware tools (e.g., normalize, join namespace-qualified names).
UNICODE_NORMAL Provide Unicode normalization support using ICU or similar.
DNS_ASYNC Enable non-blocking DNS resolution in sockets.
UNIX_SOCKETS Provide full support for Unix domain sockets (also on Windows).
UDP_SUPPORT Add UDP sockets to Tcl for modern protocols like QUIC/HTTP3.
LONG_PATH Enable long path (64K) support especially on modern Windows systems.
WIN_PTY Replace Windows console with a PTY-style driver for better scripting.
INTEGRATE_PARSER Integrate TclParser into the core for consistent parsing and tooling support.

Packages & Extensions

Symbolic Name Description
TDBC_MAINTAIN Resume active development and support for TDBC (Tcl DB Connectivity).
THREAD_MAINTAIN Provide ongoing maintenance and improvements for the thread package.
AI_LLMS Build Tcl bindings to modern AI and LLM libraries (currently Python-dominated).
WEBUI_BINDING Create Tcl bindings for the WebUI GUI framework (browser-based UI toolkit).
ES_BINDING Build a Tcl wrapper for ElasticSearch’s web APIs.
SCOTTY_TNM Revive or replace the aging SNMP libraries like Scotty and Tnm.
TKPATH Adopt and update tkpath, a modern canvas implementation with richer graphics.
LEXBOR_BIND Create bindings to the LexBor HTML rendering engine.

Documentation & Web Presence

Symbolic Name Description
SITE_REBUILD Rebuild and modernize www.tcl-lang.org including navigation and styling.
BROKEN_LINKS Audit and fix broken internal and external links in the Tcl Wiki.
OUTDATED_CODE Isolate or flag content referencing outdated Tcl versions (pre-8.4).
PACKAGE_INDEX Create and maintain a central searchable index of Tcl packages and extensions.
OFFLINE_TEACUP Host ActiveState package archive offline and enable teacup to use it.

(Earl, I took the liberty of moving your summarization to the top as it is much more lucid than the general discussion below).

APN Glad you created this page. I've noted my personal wishlist below but first couple of general comments. From the perspective of maximizing Tcl adoption, I feel the focus should be on extensions and bindings rather than Tcl itself (To be noted, I left out Tk there as I don't feel I have enough competence in Tk to voice an opinion). The exception is Tcl performance which I feel could benefit from some focus, particularly to make the 64-bit data size support in Tcl 9 actually usable.

Anyway, here where I would like to see work being done, in no particular order. Folks interested in picking some of this up might make a note here so as to avoid duplication of work. Of course, I hope others will add their thoughts and suggestions.

Tcl core

  • Tclsh - Command line editing / recall (built-in readline etc.) with pop-up completion support a la Ruby's irb
  • Tclsh - Options to execute Tcl commands from the command line
  • Tclsh - Background processing support (i.e. event driven shell)
  • Tclsh - Help system (TIP 194 or similar?)
  • Support for normalization using ICU or similar libraries. Effort: high
  • Asynchronous DNS lookup for socket -async. Effort: medium
  • Unix domain sockets (now also available on Windows) Effort: medium
  • UDP sockets so as to enable QUIC and HTTP/3 support in Tcl. Effort: high (mainly to get agreement on datagram API!)
  • A better exec, either TIP 424 or similar, to eliminate the many warts and misfeatures of exec. Effort: medium
  • namespace commands to manipulate namespace-qualified names (join, normalize, etc.) Effort: low
  • Support for named arguments in procs. Effort: medium but getting agreement is hard
  • Support for args as non last parameter. Effort: medium
  • New -all flag to glob. Currently glob has to be called twice to retrieve all files in a directory. Once with and once without -hidden. The -all flag would make that unnecessary. Also perhaps, a flag to not return . and ... Effort: low
  • Long path name support (64K path length). May apply to all platforms, but particularly Windows. Effort: unknown
  • Replace Windows console channel driver with pty. Effort: medium
  • Integrate TclParser into the core. Effort: medium

Performance-related

  • Channel I/O: the channel subsystem could use a revisit to improve performance. In particular, data is copied and iterated over multiple times. Sharing of buffers, combining encoding operations and copies etc. might be possibilities. The channel framework is both sophisticated and complex so not an easy task. Effort: massive
  • Encoding routines: common cases can be separated and optimized. Some operations scan data multiple times under certain conditions. I have started experimenting on this in the apn-encoding-refactor branch. Effort: high
  • String search (string first) could use a faster algorithm like Boyer-Moore or Knuth-Morris-Pratt. Effort: high
  • Event and timer loops: empirically speaking, passing events and timers through the event loop feels slow. For timers, perhaps an timer wheel implementation would be one option. For event loops, some investigation would be needed. Effort: unknown
  • Benchmarking: no project to improve performance is workable without benchmarks. The tcl-metrics project on github is an offshoot of the list performance benchmarks from Tcl core focused on individual commands. That is WIP. Even more important are application level benchmarks that reflect real world workloads. A Tcl version of PyPerformance would be great. Effort: high but incremental.
  • List operations: the new abstract list API in Tcl 9 should enable very efficient implementation (speed and more important, memory usage) of list operations like concat, zip, merge etc. Effort: medium
  • More efficient string implementations for range, split, insert etc. akin to what was implemented for lists in Tcl 9. Strings are more complex beasts however because of multiple internal representations. Effort: high
  • Faster interp creation: safe interps are great for isolation, for example, Web clients. Creation is however too slow for them to be used for that purpose. Effort: unknown
  • Faster callout from bytecode compiler: Invocations of C-implemented commands are a lot slower than commands that are byte-compiled. Speeding these up would be a big win. Effort: unknown.
  • Faster byte code runtime. Donal is exploring this area (see Effort: more than massive.

Tk core

  • Accessibility support. Kevin W. is working on this currently, may be able to use help on different platforms. Effort: high.
  • RTL language support. Effort: massive?
  • Make the wish console a real channel that will work with file events etc. Effort: medium
  • Anti-aliasing in canvas

Packages

List of packages would of course be endless. Some that come to mind...

  • TDBC - although part of the core packages, I am not sure anyone is actively maintaining and enhancing them.
  • Thread - also part of the core packages. Again, I suspect needs a maintainer.
  • AI / LLM interfaces - essentially all tooling has Python libraries. Their equivalents in Tcl would be great to have.
  • WebUI - https://github.com/webui-dev/webui . GUI toolkit that uses the browser (would be a Tk alternative)
  • Elastic Search - wrapper on top of their Web API
  • Scotty / Tnm - SNMP package that seems to have been orphaned.
  • tkpath - orphaned? a better canvas which now has multiple forks
  • Bindings to the LexBor browser engine

Development tools

  • Language server protocol implementation (perhaps transliterate the python tclint tools to Tcl). Effort: high
  • Tcl formatter / pretty printer. Effort: medium
  • Interactive debugger (perhaps resuscitate ProDebug which mostly works but could use some upgrades)
  • Treesitter parser for Tcl

Web presence

  • Rebuild www.tcl-lang.org. There are already plans for this using the wikit code as I understand. I suspect it could use more hands.
  • Package directory - list of Tcl packages and extensions (only a directory, not a repository). Torsten is working on this: Towards a central catalogue of Tcl packages

TWu - 2025-04-15 09:04:28

(1) To "More static analysis would be hard but good.": How about updates and binary builds to the open-sourced Tcl Dev Kit tools? It include tclchecker and tcldebugger, which has code coverage and profiling. But much is in C/C++ so I can't help.
To "A better exec, either TIP 424 or similar, to eliminate the many warts and misfeatures of exec.": Under Windows (and especially other languages than English, like German) it is often a hard work to get exec work correctly.
To "New -all flag to glob. ...": I highly be with You! Additionally there should be the possibility to ask for entry by entry as the actual glob blocks long time on large folders.
To "Long path name support ...": I use TCL/TK since 2004 under different Windows versions. Usually the long names are given back for each component of a path. If You need the 8.3-short-name there is file attributes, option -shortname. What do You miss else? But on file command under Windows there seems to be one or more bottleneck(s) asking some preferences/values. It seems file command open the file, which triggers the anti-virus unnecessary and thus slow down enormous. APN Modern Windows systems support path lengths as long as 64K and are not limited to MAX_PATH. That is what I was referring to.
To "Benchmarking": Please see my point (1) above to the subject profiling.
To "List operations": Since my first contact to TCL/TK I feel bad on list handling. Nearly all commands give their result back as "stringified" list. I don't know enough on the internals (Tcl_Obj), but on large lists and/or with large list entries, we move many bytes unnecessary. Maybe we could have list commands acting like arrays (doing more on references)? APN This is not true. Tcl does not use strings for its internal list representation. It only generates the string if you print it or pass the value to a command that operates on strings. TWu Yes, internal not. I want more like lappend, lset (both work in-place) and not like linsert, lrange, lreplace, and lsort (give back a new list). Maybe the byte-code-compiler do some optimization. But doing one step after the other there is always a "set l ..." between, doubling the needed memory and needs copying. I know of tcl::unsupported::representation and shimmering internal representation as well as tricks like ";#" at commands end to mitigate some special behavior (suppress output as string). Sorry, I was to unprecise.
To "More efficient string implementations": Not so hard like before, but the same for "string" sub-commands.
To "Accessibility support.": I have a hugh community of blind people. (See my web-site name / mail-address.) I can support on testing under Windows (and are willing to test).
To "Development tools", "Tcl formatter / pretty printer" and "Interactive debugger": Mostly I don't need a formatter and not print out source since a very long time. But if the Tcl Style Guide (respecting Tcl style discussion) is not enough, we can see... Please see my point (1) above to the subject tcldebugger.
To "Web presence - Rebuild www.tcl-lang.org." there is many to do - not only a refresh -
(a) find and resolve references from old / wiki.tcl.tk (internal) links and images to new place;
(b) this may include the move of the images or a recreation;
(c) the same for external links, there should be a tool (the real one not available/link itself broken!), see Broken Link Report and Category Broken Links;
(d) there are some pages with hugh content we can no longer diff (compare), these should be collected, and afterwards these shortened or splitted;
(e) additionally we could look for old/obsolete code for versions older than 8.4, a hint maybe the last changed date. These pages could be get into an own part of the wiki, so users can search for or without them as results. With this, obsolete or now wrong TCL/TK code is no longer spread around or explicitly marked as "outdated".
I try to help since some month and use the way-back-machine from web.archive.org to resolve after not found on two main search engines. But it maybe automated in many parts too?
(f) To "Package directory - list of Tcl packages and extensions": I can support on the latest and complete ActiveState repository. I host a 1 GiBy zip together with a script, allowing the Teacup to work locally with the content of the zip (as online no longer works, because of left https support!). It may a big and good base, and I can generate lists for the web presence automatically, I think.
Hope, this helps and give some further power - I'm happy to be in the Tcler's community! Greetings, TWu. APN @TWu, appreciate your taking the time to comment.

FM Tcl-core

  • For the programmer
    Tcl scripts are hard to maintain (at least, for me). This is because of Tcl's very dynamic nature, which allow allmost everything (variables, procs, commands) to be changed at any time. Very often I have to investigate : what has changed this value ? Then run, run and run again the script to known what had happening... In my opinion, this dynamic aspect can be both a solution and a problem. As I came from « C », I allways miss « typedef Struct ». When I create a model, I'm used to think by « Typolgy ». I'm not speaking about the data Types of computer Science, the ones Tcl_Obj adress internaly, but I'm thinking about the type of things I'm putting in relation in my model. I'd like to be able to attribute a "Type" to my value, to guard me to misuse it, and also to enforce the correctness of my modelisations. In fact, if I can use any value in any context, what does constraint me to be carefull whith the meaning I give to it ?
    • Improving of the language rules
      I like to extend the prefix construct, à la {*}{...}. A proposal {=}{...} exist for expr. It could give a solution for named argument (see [L1 ]). I call this annotation.
    • For performance
      Every programmer knows that dynamic allocation of memory make a program slower than a program that use static allocation of it. Of course, I need a program to be able to make dynamic allocation, since everything can't be predict before the start. But "everything can't be predict" dosen't mean "nothing can never be predict". So I like to be able to say : "Hey, this is predictable, so statically allocate this into the memory, please". This way, Tcl could distinguish between the static part of the logic, that can be compiled natively, to be fast, and the dynamic part, that must be allocated dynamically, to be flexible.