These are the notes I took while trying to use TEA, first on a Windows machine, and later the results of passing the 'TEA' code developed on windows to a Linux developer (and the problems observed there). Feel free to edit! This started life as a text only document so it doesn't yet have any Wiki formatting -- Vince Darley.
Introduction:
There are various goals to TEA (in no particular order):
This document describes my attempts to:
My primary platform (at the moment at least) is Windows, so I am trying to use TEA with the vc++ compiler (not with gcc/mingw/cygwin although that ought to be possible now).
For the sake of consistency this document is in more or less chronological order, which means the more interesting issues/questions are towards the end.
Any feedback on this much appreciated. I hope it helps others avoid some of the pitfalls I encountered, and any help on some of the questions/issues raised would be much appreciated.
Installation is only really an issue for windows users, who are obliged to download the cygwin/autoconf system, which they will otherwise have little use for.
First: read the instructions on scriptics web site, and try to follow them: https://www.tcl-lang.org/doc/tea/windows.html
(1.1) I found the I could not make install autoconf -- it gave some error. This was because I had tried to use it a year ago, and had since changed my cygwin installation. I had to manually delete the file config.cache (make clean isn't enough), and run through the whole ./configure ; make ; make install again. Then it worked. (Addendum: make distclean would clean up that file too, so I should've used that).
I added the contents of vcvars.bat to my cygnus.bat startup file to ensure my environment was correctly setup for use of vc++ as a compiler.
Installation isn't too complicated, although the default seems to be to create various /tmp, /opt, /lib, /usr, /bin directories all over the place.
Now you have the bash shell which is used for all TEA-based activities.
ADDENDUM: The entire cygwin setup/directory-structure etc has completely changed since ajuba's '10 steps to success' was written. It appears to have changed for the better, but still that change makes all the old instructions rather difficult to follow. There are various options for internet vs local installation process etc. Go to Annotated 10 steps to success with TEA for some help with that.
Download the sampleextension. I grabbed this with cvs, to make sure I had the latest.
(2.1) autoconf, configure and make
After cd sampleextension ; autoconf ; mkdir win ; cd win I tried to ../configure which failed --- the tclConfig.sh file could not be found. This file doesn't seem to come with the binary distributions of Tcl. So I download the Tcl8.3.1 sources, and did the following cd tcl8.3.1/win ; autoconf ; mkdir Release ; cd Release ; ../configure ; make and that all worked (phew!). So now I had a directory which contained all the .sh, .lib, .exe's etc. Now go back to sampleextension/win and try again. ../configure still fails, but if you do
../configure --with-tcl=//d/tcl-source/tcl8.3.1/win/Release/
it worked better, but couldn't find tcl.h this time. Fortunately the error message was better, so I then did
../configure --with-tcl=//d/tcl-source/tcl8.3.1/win/Release/ --with-tclinclude=//d/tcl-source/tcl8.3.1/generic/
and this completed! The next step is make which failed, this time with unterminated quoted string. It doesn't like the fact that I used a trailing / on my directory names above (the forward slash was converted to a microsoft backslash which was followed by a quote...). So, back to configure:
../configure --with-tcl=//d/tcl-source/tcl8.3.1/win/Release --with-tclinclude=//d/tcl-source/tcl8.3.1/generic
This works (seemingly the same as before) -- shame the code is fragile in this way. And I try make again... This time it works, and I appear to have a .dll present so compilation seems successful. There is even some advice about how to create documentation from xml files!
(2.2) make install
The next step is make install which will of course fail, since I didn't bother to tell configure where I wanted to install things. Let's try anyway. This tried to dump stuff into /usr/local/bin and /usr/local/lib (why do we have idiotic defaults like that on Windows? If it knows I'm using Windows, why can't it at least try C:/Program Files/Tcl/lib etc??). It also failed with on this:
Making pkgIndex.tcl in d:/usr/local/lib/exampleA
eval pkg_mkIndex exampleA0.2 ../../bin/exampleA02.dll *.tcl
couldn't change working directory to "exampleA0.2": no such file or directory
while executing
"cd $dir"No idea what that's all about, so I ignored it all. Back to configure, this time with a correct prefix/exec-prefix?
../configure --with-tcl=//d/tcl-source/tcl8.3.1/win/Release --with-tclinclude=//d/tcl-source/tcl8.3.1/generic --prefix=//d/progra~1/tcl
Now to avoid problems with spaces which I'm sure will arise (yet another reason not to use autoconf/configure/make as a cross platform tool --- what am I supposed to do if windows didn't provide the peculiar conversion of progra~1 into 'program files'?), I use progra~1. I didn't bother to set the exec-prefix, since I assume that just with prefix set, the installer should know to split the installation into lib and bin as appropriate?
make install
Appears to work...
(2.3) package require
So I startup wish, and type package require exampleA since that seems to be the name of this thing. But no, I discover after reading the pkgIndex file that it is called Tclsha1. What kind of name is that? (I guess, when I read the code it is a package sha1 which makes some sense, except there was a recent thread on comp.lang.tcl suggesting the package names cannot contain numbers, I guess they cannot contain numbers in the middle, perhaps that fragility should be changed!). It seems like a bad idea to have numbers as part of the name (this is version 0.2 of package Tclsha1 ...). Anyway, at least it works:
% package require Tclsha1 0.2
The command it creates is sha1:
% sha1
wrong # args: should be either:
sha1 ?-log2base log2base? -string string
or
sha1 ?-log2base log2base? ?-copychan chanID? -chan chanID
or
sha1 -init (returns descriptor)
sha1 -update descriptor ?-maxbytes n? ?-copychan chanID? -chan chanID
(any number of -update calls, returns number of bytes read)
sha1 ?-log2base log2base? -final descriptor
The default log2base is 4 (hex)So it has worked! Success! Now I have to get my own extension to compile.
Make your directory myextension and copy all *.in, *.m4 and install-sh from the sampleextension to that directory. Edit the *.in files to fill in your own .c/.h information (the instructions are pretty good in the comments of the files). Place your .c/.h files in that directory. (ADDENDUM: you also need mkinstalldirs from the sampleextension -- see below).
Do autoconf. Hopefully it completes ok.
mkdir win ; cd win ../configure --with-tcl=//d/tcl-source/tcl8.3.1/win/Release --with-tclinclude=//d/tcl-source/tcl8.3.1/generic --prefix=//d/progra~1/tcl
again hopefully completes ok. Now make. I got an error:
make: *** No rule to make target 'tclMatrix.obj', needed by 'Matrix01.dll'
This seems surprising. So I go to look at the makefile. It seems I have to manually list the object rules which comprise my shared library. So, back to makefile.in and edit those. Now a quick autoconf, cd win, ../configure ...., and I try to make again. This time it works, and I appear to have a dll! Jolly good. So I try make install and it complains it can't find mkinstalldirs. So I copy that over from the sampleextension and try make install again. It seems to work, although at the end it complains about not finding ../*.n --- not surprising since I don't have any man pages. However what is it doing looking for man pages on Windows?? Very strange.
(3.1) package require Matrix
Anyway, it seems to have copied the important stuff over: .dll, creation of pkgIndex.tcl etc, so I startup Wish and try package require Matrix. That fails. So I notice the pkgIndex.tcl file only contains a long comment, no package ifneeded ... command. Finally I try loading the .dll manually, and there is no _Init proc. It occurs to me that windows requires all that DLL_EXPORT stuff, so I go back to my sources, and add the relevant code (look at exampleA.h, it seems to contain the important stuff). Now I redo make, make install and I can finally package require xxxx and it works.
Note that the first time that make install seemed to work, but didn't, the pkg_mkIndex step failed to find any packages at all, yet didn't signal any error or warning. That is not very helpful. Presumably any call to pkg_mkIndex which results in a pkgIndex.tcl devoid of anything except comments is an error, and should be signalled as such.
I pass the code on to someone who is pretty experienced with Linux, but not so with Tcl, but TEA is designed to make their life easy...
(4.1) basic line-endings problem. Had to convert all \r\n to \n. Eventually found a simple, repeatable solution to this, which would let us pass code back and forth.
(4.2) autoconf, configure, make. This all appeared to work ok (provided configure is given the correct locations for tcl)
(4.3) make install. Again this appeared to work ok.
NOTE: You can make life a bit easier, if you put some files (or symlinks to them) into the right directories (on SuSE linux):
/usr/include/tcl.h /usr/include/tk.h /usr/include/tlcXXX.h ... /usr/include/tkXXX.h /usr/lib/libtclstub.NN.a /usr/lib/libtkstub.NN.a
HRB.
(4.4) package require Matrix.
This failed with "can't find package Matrix". It seems as if the pkgIndex.tcl file is again just full of comments. However the problem is far more subtle than simply forgetting the EXPORT stuff (which was the problem with the same symptoms on Windows). The problem turns out to be that pkg_mkIndex is sourcing the shared library libMatrix0.1.so.1, rather than loading it! This in turn is due to a bug in either the TEA installation on linux, or in the mkIndex.tcl.in installer (or both). The decision to source vs load is made by looking at the file's extension and comparing it with info sharedlibextension, which is .so. However the file's extension is .1. So, we can either fix pkg_compareExtension, or make the installer create a soft link from libMatrix0.1.so.1 to libMatrix0.1.so, or perhaps both.
Finally there is one more problem. If the library is installed somewhere which is not on LD_LIBRARY_PATH, the load also fails, so the installation code needs to do something like:
pushd $(libdir) ; \ LD_LIBRARY_PATH=`pwd` ; \ export LD_LIBRARY_PATH ; \ popd \
before executing mkIndex.tcl.
With these changes the code finally works and package require Matrix does indeed work. Looking at other extensions I see nearly all of them actually have their own manually created pkgIndex.tcl which is installed rather than relying on pkg_mkIndex -- probably a good idea seeing as the code is both buggy (source vs load) and provides no useful feedback at all on whether the process was successful or not.
So your extension has other extensions which rely upon it. You need to create and export your own stub table. I suggest you look at the Trf and Trfcrypt extensions for a good example, and for all the necessary stub/.in files from which you can re-use code (with global replacements of Trf by Mypackage etc).
Problems observed:
(4.2.1) It basically works after several hours work.
(4.2.2) Passing the code over to another platform exposed the usual host of problems. You MUST MUST MUST use the -direct flag when calling pkg_mkIndex on the base shared library (in Tcl 8.3 or newer that is the default, but of course when you pass the code to someone using Tcl 8.2 the base extension will work, but the other one will not). This is my current code (from end of mkIndex.tcl.in):
(base extension with stub table 'Matrix'):
if {$tcl_platform(platform) == "unix"} {
if {[llength $libraryList] > 0} {
set libraryPathList {}
foreach lib $libraryList {
# shared lib extension is usually .so or .sl
if {[regexp "^(.*\\[info sharedlibextension])((\\.\[0-9\]+)+)\$" $lib "" baselib]} {
exec ln -s $lib $baselib
lappend libraryPathList [file join .. $baselib]
} else {
lappend libraryPathList [file join .. $lib]
}
}
puts "eval pkg_mkIndex -direct -verbose $package$version $libraryPathList *.tcl"
eval pkg_mkIndex -direct -verbose $package$version $libraryPathList *.tcl
}
} else {
if {[llength $libraryList] > 0} {
set libraryPathList {}
foreach lib $libraryList {
lappend libraryPathList [file join .. .. bin $lib]
}
puts "eval pkg_mkIndex -direct -verbose $package$version $libraryPathList *.tcl"
eval pkg_mkIndex -direct -verbose $package$version $libraryPathList *.tcl
}
}(dependent extension wanting to use the Matrix extension):
proc pkg_ensurePreloaded {name} {
puts "preloading $name package"
set result [package require $name]
puts "'package require $name' returned $result"
# Make sure it was really loaded
foreach pkg [info loaded] {
if {[lindex $pkg 1] == $name} {
puts "package $name loaded as '$pkg'"
return
}
}
puts "warning: package $name didn't appear to be loaded correctly"
puts "perhaps its pkgIndex was built without the '-direct' flag,"
puts "in any case any use of 'pkg_mkIndex -load ...' will not"
puts "be able to load the '$name' package."
}
if {[llength $libraryList] > 0} {
set libraryPathList {}
if {$tcl_platform(platform) == "unix"} {
foreach lib $libraryList {
# shared lib extension is usually .so or .sl
if {[regexp "^(.*\\[info sharedlibextension])((\\.\[0-9\]+)+)\$" $lib "" baselib]} {
exec ln -sf $lib $baselib
lappend libraryPathList [file join .. $baselib]
} else {
lappend libraryPathList [file join .. $lib]
}
}
} else {
foreach lib $libraryList {
lappend libraryPathList [file join .. .. bin $lib]
}
}
# In case we are installing in a non-standard location
lappend auto_path [pwd]
pkg_ensurePreloaded Matrix
puts "eval pkg_mkIndex -direct -verbose -load Matrix $package$version $libraryPathList *.tcl"
eval pkg_mkIndex -direct -verbose -load Matrix $package$version $libraryPathList *.tcl
}(5.1) If my Extension_Init symbol isn't correctly exported, I get no notification that the pkg_mkIndex didn't find anything. Usually (since packages usually include a XXX_Init function), this is an error. At the very least pkg_mkIndex should output the names and versions of the packages it found (something like: Creating pkgIndex file... added package entry for 'Foo 0.1'... added package entry for 'Blah 2.7'). I've submitted a patch to package.tcl to scriptics which adds better output with pkg_mkIndex -verbose :
Index: package.tcl
===================================================================
RCS file: /cvsroot/tcl/library/package.tcl,v
retrieving revision 1.14
diff -c -r1.14 package.tcl
*** package.tcl 2000/04/23 03:36:51 1.14
--- package.tcl 2000/05/22 16:42:44
***************
*** 328,336 ****
--- 328,344 ----
tclLog "warning: error while $what $file: $msg"
}
} else {
+ set what [$c eval set ::tcl::debug]
+ if {$doVerbose} {
+ tclLog "successful $what of $file"
+ }
set type [$c eval set ::tcl::type]
set cmds [lsort [$c eval array names ::tcl::newCmds]]
set pkgs [$c eval set ::tcl::newPkgs]
+ if {$doVerbose} {
+ tclLog "commands provided were $cmds"
+ tclLog "packages provided were $pkgs"
+ }
if {[llength $pkgs] > 1} {
tclLog "warning: \"$file\" provides more than one package ($pkgs)"
}However, I would suggest going beyond this, to signal a make-time error if nothing is found in the package to install. After all an installation which provides no packages and no commands is rather useless, so it can be used to signal that something has gone wrong. Perhaps another flag to pkg_mkIndex? (Note that the -lazy flag is not properly documented in the comments or usage message of pkg_mkIndex).
(6.2) Just prior to build the pkgIndex, make install gives the following message:
Making pkgIndex.tcl in d:/progra~1/tcl/lib/Matrix
This is incorrect. It is in ...../Matrix0.1 It is fixed below
cd $libdir
puts "Making pkgIndex.tcl in [file join [pwd] $package$version]"
if {$tcl_platform(platform) == "unix"} {
if {[llength $libraryList] > 0} {
set libraryPathList {}
foreach lib $libraryList {
lappend libraryPathList [file join .. $lib]
}
puts "eval pkg_mkIndex -verbose $package$version $libraryPathList *.tcl"
eval pkg_mkIndex -verbose $package$version $libraryPathList *.tcl
}
} else {
if {[llength $libraryList] > 0} {
set libraryPathList {}
foreach lib $libraryList {
lappend libraryPathList [file join .. .. bin $lib]
}
puts "eval pkg_mkIndex -verbose $package$version $libraryPathList *.tcl"
eval pkg_mkIndex -verbose $package$version $libraryPathList *.tcl
}
}
exitThis also adds 'exit', which is required for Tk based extensions, but might as well be there for Tcl based ones too. It also adds the -verbose flag to pkg_mkIndex.
(6.3) On Linux I find that shared libraries are created as blah.so.1 which pkg_mkIndex decides to source and hence generates an empty pkgIndex.tcl file. The following fixes that:
# package.tcl
proc pkg_compareExtension { fileName {ext {}} } {
global tcl_platform
set ext [info sharedlibextension]
if {[string equal $tcl_platform(platform) "windows"]} {
return [string equal -nocase [file extension $fileName] $ext]
} else {
# Some unices add trailing numbers after the .so, so
# we could have something like '.so.1.2'.
set root $fileName
while {1} {
set currExt [file extension $root]
if {![string length $currExt]} {
return 0
}
if {[string equal $currExt $ext]} {
return 1
}
set root [file rootname $root]
}
}
}Alernatively the installer needs to create a link from the .so.1 to .so The following adds such a link.
# mkIndex.tcl.in
if {$tcl_platform(platform) == "unix"} {
if {[llength $libraryList] > 0} {
set libraryPathList {}
foreach lib $libraryList {
if {[regexp {^(.*\.so)((\.[0-9]+)+)$} $lib "" baselib]} {
exec ln -s $lib $baselib
lappend libraryPathList [file join .. $baselib]
} else {
lappend libraryPathList [file join .. $lib]
}
}
puts "eval pkg_mkIndex -verbose $package$version $libraryPathList *.tcl"
eval pkg_mkIndex -verbose $package$version $libraryPathList *.tcl
}
} else ...(6.4) There also seem to be some requirements to set LD_LIBRARY_PATH during installation of the extension. Is this true? Almost every extension out there uses its own hand-made pkgIndex.tcl so problems with pkg_mkIndex/load/sharedlibraries are never seen. An install-binaries which looks like this should help:
install-binaries: binaries install-lib-binaries install-bin-binaries
- $(TCLSH_PROG) mkIndex.tcl $(Matrix_LIB_FILE)
if test "x$(SHARED_BUILD)" = "x1"; then \
- $(TCLSH_PROG) mkIndex.tcl $(Matrix_LIB_FILE); \
+ case "uname -s" in *win32* | *WIN32* | *CYGWIN_NT* | *CYGWIN_98* | *CYGWIN_95*) \
+ ;; \
+ *) \
+ pushd $(libdir) ; \
+ LD_LIBRARY_PATH=`pwd` ; \
+ export LD_LIBRARY_PATH ; \
+ ln -sf $(Matrix_LIB_FILE) lib$(PACKAGE)$(VERSION).so ; \
+ popd \
+ ;; \
+ esac ; \
+ $(TCLSH_PROG) mkIndex.tcl lib$(PACKAGE)$(VERSION).so ; \
fiWill this hard-coding of lib and .so work on all Unix platforms?
(6.1) Shouldn't configure/make be able to cope with directories given by the user with trailing backslashes?
(6.2) When trying to compile Tk using the TEA makefiles on windows, if I do mkdir Release ; cd Release ; ../configure ..., Tk looks for Tcl by default in ../../tcl8.3/ which isn't very helpful, it needs one more set of ../. Also tcl8.3 isn't very helpful either, when the release is 8.3.1, and a tcl8.3.1 directory is what cvs gives me.
(6.3) Why is it using /usr/local/ as the default installation location on Windows? Almost all users will install in Program Files/Tcl on either the C or D drives (possibly with trailing 8.3 or whatever), so why not look there?
(6.4) Why does make install look for *.n files on Windows, and then why does it fail when it can't find any?
(6.5) Why do I have to list a separate .c to .o myfilename.$(OBJEXT): ... in makefile.in for every single source file. Why can't these be generated automatically? (So there's a comment saying this can't be done in a makefile-independent way -- what a feeble system!).
(6.6) In the checking for tclsh... configuration step, one time (with I guess wrong --with-xx arguments), it picked up tclsh.rc instead of tclsh83.exe. Why is it even considering anything which isn't .exe on windows?
(6.7) If I edit the Makefile.in and then don't do configure manually, (when I'm in a Release subdirectory, say), but just type make, then the code tries to rebuild the makefile again (which is good), but it does it wrongly (it cd's up a level, and then things go wrong).
(6.8) Why does this seem to be doing the mkIndex thing twice?
install-binaries: binaries install-lib-binaries install-bin-binaries
$(WISH_PROG) mkIndex.tcl $(Pltk_LIB_FILE)
if test "x$(SHARED_BUILD)" = "x1"; then \
$(WISH_PROG) mkIndex.tcl $(Pltk_LIB_FILE); \
fi(6.9) Now that Scriptics is Ajuba, what is going to happen to the SC_ macros? Why are there so many bugs in TEA if Scriptics is really supporting it?
(7.1) How do I create a set of TEA files to handle a set of multiple extensions/packages? Let's say I have three extensions A, B, C. Extension A is a simple Tcl extension, and TEA works ok (subject to the million problems above). Extension B is a Tcl extension which also requires some code in A. How can I create that? Extension C is a Tk extensions which requires some code in A and B. (This is a real situation for the Plplot graphing library). Do I create one big configure.in/makefile.in (is that possible given the differing requirements of these libraries)? Or do I create 3 different sets of configure/makefile.in? How do I handle the linking issue on Unix? Where is the TEA documentation/examples on this kind of stuff?
The whole process is completely unintuitive to someone not familiar with these kinds of things. Once it works it is quite nice, but there are very large hurdles to be jumped, and the entire system is very fragile. It appears to be a very dumb system, which is trying to look rather clever. I guess it is basically a very simple text and rule based iterative system, which doesn't really have much embedded knowledge about files/directories, the structure of a tcl distribution or whatever. The fact that it contains numerous bugs/fragilities doesn't really help either.
There is, I guess, the possibility of cross-compiling, and that is a pretty good reason for using the system (although, with perhaps one notable exception, one hardly sees a peep on comp.lang.tcl about announcements of binary distributions being available on many platforms through the advances of cross-compilation). Also, if TEA was replaced by something more user-friendly, perhaps people would be willing to compile extensions themselves, and we wouldn't have to worry about cross-compiling anyway.
Jeff has raised the idea of getting rid of the tcl/tkConfig.sh files and embedding that information inside the Tcl/Tk binaries. I think that would be a big step forward. Then if Tcl/Tk are correctly installed (and have headers/libs installed as well in default locations), TEA could automatically work out where everything it requires (and if headers or libs are not present it can give a meaningful message). My suggestion would be to have that in Tcl8.4 as soon as possible. Right now the goal of configure, make, install isn't really possible (at least on windows). Then it might even be possible (for the enterprising ;-, to have package require Blah download a .tar.gz, unpack it, configure, make and install, and continue running). I guess this is the package acquire idea.
Is TEA a good thing? Does it have the support of the community? Are people using it?
My answer is "I'm not sure". I've run into sufficient problems using TEA on both windows and linux that I can't believe many people are really using it that extensively. In particular, associated pieces like pkg_mkIndex are hopelessly uninformative and fragile on all platforms, and almost every extension I've seen actually has its own hand-made pkgIndex.tcl which it installs, rather than building one. The TEA mailing list archive is full of debate, the need for better documentation (especially for windows, but from my own investigation it seems just as lacking on unix), etc. However there is very little action. Well, this is my tiny contribution to that action, but frankly TEA needs some proper support and work which simply doesn't appear to be happening.