No Magic or Policy in Starkits

an absence of magic, and no policy

This was all lifted from this message [L1 ]:

There is no presupposition about how your files are arranged other than that the top level main.tcl is to be interpreted at launch.

Unlike some other wrapping methods there isn't an enforced or assumed layout.

The above is worth repeating, IMO. All a starkit does, is contain a dir hierarchy (even if just a flat one) which gets "mounted". That mount point happens to be the same spot as the starkit itself - in other words, file "/path/to/my/starkit.kit" ends up becoming dir "/path/to/my/starkit.kit/" while in use, with things like "/path/to/my/starkit.kit/lib/" being a preferred place to put packages, because that's what "starkit::startup" will set up for you, but it's all 100% wide open and totally scripted/inspectable.

This is quite true - but I'll add the comment that the "starkit::startup" procedure that is usually invoked from the main.tcl does add the /path/to/my/starkit.kit/lib directory to the auto_path.

So, although there isn't any enforced structure it sure is convenient to just drop packages into .../lib and have them available in your application - stevel

Ro: yep steve you're right on the mark. It does say that in the paragraph that I lifted from Jean-Claude Wippler and Steve Blinkhorn, but I'm glad that you expounded on that fact. A nice standard structure is easier to use, because you know that it works.

CMCc There's a downside, or at least niggle, with starkits: making loadable binary extensions (like kitten, but less ambitious) is a black art. First problem is that starkit's tcl interpreter is a statically linked binary, so you can't expect a libtcl .so to be in your path. Second problem is that any library that is available won't have the same name at compile time as it has at runtime, which can make compilation relatively hard.

I would appreciate any assistance with these problems. LV has this page for discussion and consideration.

08feb03 jcw - Your perception of "black art" is something I would very much like to help change. If it is, then starkits have failed to deliver, IMO - it should not be tricky, or even confusing. There are really only two stumbling blocks I know of:

  • shared libs *have* to be compiled with stubs (-DUSE_TCL_STUBS and -ltclstub8.4)
  • paths need to be relative far more often - i.e. using idioms such as
        load [file join [file dirname [info script]] blah].so

The former is gradually becoming the norm. It has been in Tcl since 8.1, and solves a couple of issues along the way, so I'm sure it is just a matter of time to see more and more extensions compiled that way. This addresses your point of "" not being around: correct, because with stubs, they no longer play a role.

The latter is actually quite valuable - you are being forced to consider that your code may not run in the same place as you are developing it in. It now becomes an issue slightly sooner in the lifetime of a software project. The name itself changes if you are referring to VFS's default behavior of copying a shared lib out to temp before running it. But if that is an issue, why not work around it and copy things yourself to a spot of your own choosing? (such as the current dir: "file copy blah [pwd]") - the point I'm making is that starkits + VFS can do funny things, but you can always take over if you prefer.

So yes, there are some unavoidables in playing nice with/for starkits, but the benefit of "wrap-and-go" is hard to overestimate. I routinely wrap things in the midst of active development, to the other side of the globe, to allow someone else to check something out. No testing, nothing. We often afterwards laugh, because we work on different platforms, and never even noticed.

Maybe the lesson from your comments is that things need to be better documented. I'll gladly do so - am starting to do this more now, on the website.

cmcc You're right, jcw, once I scanned the 2Mb Starkit mailing list archive, it became clear that most of what we need to do is ensure that extensions are stub-capable, and compiled with stubs. I was trying to package tclhttpd, and found that the crypt.o stuff wasn't loadable - it turned out that the tcl version being passed toTcl_InitStubs was wrong, and STUBS weren't enabled by default, so the loading failed noisily.

More docco is good. The need for stubs has to be emphasized, the need for relative paths seems to be clearly stated and obvious.

Suggestion1 - Kitten is a good start, if not yet properly implemented.

If I may: I don't mind if packaging for starkits is more difficult than just 'make install', because the benefits are there, and are clear, but it's a symptom that something wrong when people can't/don't make custom `kitten'-like packages, and when kitten itself has an out of date itk/iwidgets relative to tclkit itself. Something's not easy or obvious enough, or the evident convenience of starkits would have led to more uptake of the 3-pack distro concept.

I would be interested in seeing more docco on precisely what I have to do to make a kitten-like pack, given a set of required extensions. Either that, or perhaps the next kitten could have support for removing unneeded extensions, and be distributed as an enormous package containing every extension known to tcl-kind. I dunno, call it kit-chen (short for kitchen sink :) then a lazy man like me could take the gigantic package and remove the excess, leaving just the packages I need.

I guess, then, this implies that kitten/kitchen would benefit from some more policy at the package level, which might not be what you want - it could evolve into something as full featured as Debian's apt-get.

Suggestion2 - VFS might yet be too weak for what we want - perhaps we could use Plan9 overlay-mount.

It occurs to me that one could easily mount a starkit over an existing directory, so the myapp/lib/ could be mounted over /usr/local/lib, and this would mean that normal --prefix compilations would work (so you could almost take any package and compile it straight in, no worries.)

It then occurs to me that one might want to add several starkits together, and have each starkit overlay the previous one.

Plan9 has an idea of directories which add together - mounting in Plan9 can overlay - that's what it does for $PATH, it just overlays /bin, /usr/bin, /usr/local/bin etc (you get the idea.) This should be possible for VFS - would it be of benefit? That way you could use a starkit to augment a standard install, and use each starkit to augment the others in a stack.

Even without Plan9 overlay-mount, mounting over /usr/local/lib would have the advantage that some kind of chroot arrangement would enable you to make install directly into your .vfs file and have some confidence that it'd work out of the box, it shouldn't impact any starkit which was completely self-contained (and that is the idea, no?) It wouldn't have the entropy-decreasing advantage of requiring everything to run relative, but reducing entropy costs energy (in this case, the energy necessary to mod any useful tcl extension and get the mods accepted upstream.)

Man, this page sure has gotten complicated... --Ro

See also: Sexy Starkits