Version 39 of Stubs

Summary

Stubs: A dynamic linking mechanism for Tcl, Tk and extensions.

Propaganda

Georgios Petasis enthuses, "For me stubs is simply a brilliant feature of tcl."

Description

A stubs table is a statically-linked table of symbols. This table can be included into statically-linked libraries, and then, at runtime, initialized against a library that provides the actual implementations for the symbols. This provides an alternative to OS-provided dynamic linking of shared objects.

What does it do?

Adds support for backlinking to all platforms, thus making it possible for static executables to dynamically load extensions on any platform.
Eliminates all the operating system-specific problems associated with dynamically linking an extension to Tcl/Tk and any other library which exports a stub interface. This may sound like a wild claim but it is true simply because the operating system does not have to do anything.
Extensions and libraries built with different compilers will work together even if it is not possible to link them normally. The only requirement for them to be able to work together is that function calls are compatible.
As extensions do not have a hard coded reference to the library it is possible to use them with any library which is compatible. e.g. an extension which was originally built for Tcl 8.0 should work with Tcl 8.1, and an extension built for Tcl 8.1 could work with Tcl 8.0.

How does it do it?

Basically the stub mechanism provides a cross-platform dynamic linking mechanism using tables of function pointers.

A library which wishes to provide a stub interface populates a function table with the addresses of its interface functions.
A pointer to that table is then made available to any extension which needs to access that library's interface.
Once an extension has obtained the pointer it simply uses it to get the address of the function it wants to call and calls it.
At no time in this process does the operating system become involved.

To see the differences take a look at the following which describes what happens when Tcl loads two different extensions.

Loading an extension which is linked directly to Tcl.
1. Tcl asks the operating system to load the extension.
2. The operating system loads the extension and then tries to resolve any undefined symbols. This process is very operating system dependent but involves one or more of the following steps.
  1. Resolve any symbols which are defined in the current process context. (Backlinking).
  2. Find and load any libraries that the extension is dependent on. This involves searching paths defined through a variety of operating system dependent methods.
    1. Resolve any symbols which are defined in the new libraries.
  3. Tcl then calls the extension's initialisation entry point.
Loading an extension which uses Tcl's stub interface.
1. Tcl asks the operating system to load the extension.
2. The operating system does so, resolving any non Tcl symbols.
3. Tcl calls the extension's initialisation entry point.
4. The extension obtains the pointer(s) to Tcl's stub table(s) and uses that to call Tcl.

Levels of Stubs Support

There are two/three levels of stubs support.

Level 0 - The extension doesn't support stubs at all. On many platforms (but not all) this means it is bound to an exact version of Tcl, but it always means that you'll probably have to recompile your extension to upgrade for a new version of Tcl. - Lars H: Upgrade what? The extension or Tcl? (The latter is probably more annoying.) UKo: made the description more clear (additions are emphasized)

Level 1 - The extension is stubs-enabled, using the stubs mechanism to dynamically bind to the Tcl (and Tk and other) library so that it can be loaded by any version of interpreter that supports the required version of API. Many many extensions provide this level of support (e.g. TclUDP) and this is the minimum level of support required for an extension to be supported inside a starkit.

Level 2 - The extension is a stubs-source and exports its own stubs table. This allows other code to dynamically bind to it. This is only useful where an extension provides its own meaningful C API. While this is theoretically separate from whether it is stubs-enabled, it is normally the case that all stubs-sources are also enabled. Tk and memchan are examples of stubs-sources.

Who's responsible for the Stubs implementation?

Perhaps Jean-Claude Wippler suggested it to Paul Duffin in 1999, and Paul and Jan Nijtmans, with whom Jean-Claude had also been discussing ideas, implemented it in 2000. Others involved in the first generation were ??? (backlinking details) ...

DKF: I certainly remember suggesting something like the mechanism we ended up with back before Paul produced his initial version, though his initial version was a lot more elegant...

Questions

How can I tell if a binary extension was built with Stubs?

Under Linux and similer Unixes, check with ldd. No Tcl or Tk library references should appear. For Windows, do the same with "dumpbin /dependents".

Where is it most important to use Stubs?

Well, one place is that development against a Tclkit/Starkit/Starpack environment, extensions that are not built using Stubs result in more difficulty. (Or is it actually that they cannot be used at all?)

What are my options if the extension I want to use isn't Stubs-compatible?

Is all this stubs stuff necessary? Why does the build system have to be so difficult. Believe us: this is far less difficult than the alternative. Yes, libtool and autoconf are maddening, but there are things outside the Tcl world that demonstrate we don't have it so bad. DKF briefly defends stubs in a follow-up [L1 ] to a lengthy thread that itself touches several aspects of generation.

Helmut Giese notes that, even when compiler providers agree on object formats, they might still construct libraries in incompatible ways. In particular, under Windows, those who choose to work with gcc-based compilation might need to rebuild tclstub84.lib before Stubs-enabled extensions load correctly.

Georgios Petasis and Michael Schlenker observe that Stubs-less languages need "batteries included'', because reliance on version-specific extensions would otherwise be prohibitively onerous: "the C interface of these languages seems so primitive that you have to recompile everything each time a new version is out."

Phils take on stubs Stubs - Another explanation

KBK: "Even in version lockstep, stubs can be handy if only to keep all of your libraries following the same linkage conventions."

hat0 Please note that, in building a cross-platform stub-enabled extension, that Windows may require this modification to your C code's declaration:

- int Extension_Init(Tcl_Interp *interp)
+ int DLLEXPORT Extension_Init(Tcl_Interp *interp)

Without this, linking may fail in curious ways, with errors such like:

Cannot export ??_C@_03KBFG@Tcl?$AA@: symbol not found
Cannot export ??_C@_03PFGM@?$CJ?3?5?$AA@: symbol not found
Cannot export ??_C@_0BC@JJNH@?0?5actual?5version?5?$AA@: symbol not found

Category Porting

Category Package

Category Development