Odyce , by Eric Hassold, is a closed-source system for embedding C in Tcl scripts. It is based on Colin McCormack's tcc tcl extension and on tcc itself. Colin is not credited, except possibly in some source code which is no longer publicly available. Furthermore, in the conversation below, Eric never acknowledged the roots of Odyce. This has led to some controversy.
The name is for "Autonomous Dynamic C Environment" ("Au" in French is pronounced like "O"), and is pronouned "odyssey".
Currently available only as part of eTcl binary distribution.
Odyce accomplishes a goal similar to critcl, but without needing a separate compiler or linker to be installed on target machine.
Comparison:
List of supported architectures:
Partial emulation of Critcl on top of Odyce is also available. For now (rc23), download odyce-demo-20071001.zip and, instead of package require critcl, source the file critcl.tcl located within. critcl.
RS 2007-10-01: http://www.evolane.com/software/odyce/index.html says: "This package contains softwares covered by other licenses: TinyCC .. LGPL". Is that tcc? Also, some examples would be very welcome.
That pages mentions that no headers are required - not even <stdio.h> and friends?
EH: System headers, together with Tcl/Tk ones, are embedded into VFS, and system include directory for embedded C compiler is automatically setup to find them there. So you can use #include <stdio.h> and friends in your code without having to care about any header installed on target machine.
wdb 2007-10-01: just downloaded etcl, but sadly, no documentation available. Are there any users out there with some example applications?
EH 2007-10-01: Odyce announce, together with 1.0-rc23, was a very early announce, to let others know about this previously hidden feature. We are working hard this week on providing samples, documentation, and improving critcl emulation (so that all critcl documentation should be adequate too). This page is good place to keep in sync until our official site gets updated.
RS: Eric Hassold himself posted an example today on fr.comp.lang.tcl:
# Inutile de charger odyce
# package require odyce
# Car l'emulation de critcl s'en charge
package require critcl
# Bug stupide dans la version embarquee
# (ai oublie d'initialiser la liste a vide)
#::critcl::csources ""
set ::critcl::critcl(cfiles) [list]
# Un bout de code C a integer
critcl::ccode {
#include <stdio.h>
#include <math.h>
static int fib(int n)
{
if (n <= 2) {
return 1;
} else {
return fib(n-1) + fib(n-2);
}
}
}
# Et la definition de la commande Tcl "fib" a partir
# du code natif
critcl::ccommand fib {dummy interp objc objv} {
int n;
if (objc!=1) {
Tcl_WrongNumArgs(interp,1,objv,"int");
}
if (Tcl_GetIntFromObj(interp,objv[1],&n)!=TCL_OK) {
return TCL_ERROR;
}
Tcl_SetObjResult(interp, Tcl_NewIntObj(fib(n)));
return TCL_OK;
}
# Ca va vite, tres vite...
puts [fib 30] Works by reporting 832040 on stdout - I suppose it's correct :) For comparison, I also did a Tcl version:
proc fib0 n {
if {$n <= 2} {return 1}
expr {[fib0 [expr {$n-1}]] + [fib0 [expr {$n-2}]]}
}
puts Odyce:[fib 30],[time {fib 30}]
puts Tcl:[fib0 30],[time {fib0 30}]which gives the same result, but takes about 171 times longer:
Odyce:832040,10578 microseconds per iteration Tcl:832040,1808314 microseconds per iteration
Next idea: to wrap a C command into a proc that has it compiled at first call:
proc Fib x {
critcl::ccommand Fib {dummy interp objc objv} {
int n;
if (objc!=1) {Tcl_WrongNumArgs(interp,1,objv,"int");}
if (Tcl_GetIntFromObj(interp,objv[1],&n)!=TCL_OK) {return TCL_ERROR;}
Tcl_SetObjResult(interp, Tcl_NewIntObj(fib(n)));
return TCL_OK;
}
uplevel 1 Fib $x
}
puts wrapped:[Fib 30],[time {Fib 30}]which, not surprisingly, fares equally well:
Odyce:832040,10203 microseconds per iteration Tcl:832040,1871121 microseconds per iteration wrapped:832040,10511 microseconds per iteration
The difference here is that only if Fib is called (the first time), the C code is compiled. If nobody calls Fib, we save the (not high) time and memory consumption...
Lars H 2007-10-02: As I recall it, Critcl normally puts the command to actually compile C code into the auto_index array, meaning nothing gets compiled at critcl::ccommand time, but rather the first time the command is called. If odyce does it differently, then that is a difference that is worth observing.
Just tested, the examples run on Windows Mobile 5 too, and with an ever bigger gain of 619 (!):
Odyce:832040,172293 microseconds per iteration Tcl:832040,106723982 microseconds per iteration wrapped:832040,172042 microseconds per iteration
Re available commands, good old introspection to the rescue:
21% info commands critcl::* ::critcl::init ::critcl::cleanname ::critcl::ccode ::critcl::debug ::critcl::clibraries ::critcl::Log ::critcl::ccommand ::critcl::checkname ::critcl::cproc ::critcl::csources ::critcl::tk ::critcl::cheaders
Or, to see which header files come "in the box":
% cd d:/Tcl/etcl/etcl.exe/.etcl/vfs/extensions/odyce0.1/include % lsort [glob *] X11 assert.h dlfcn.h features.h float.h limits.h malloc.h math.h memory.h stab.h stdarg.h stdbool.h stddef.h stdio.h stdlib.h string.h tcl.h tclDecls.h tclPlatDecls.h tk.h tkDecls.h tkIntXlibDecls.h varargs.h % cd X11 % lsort [glob *] X.h Xatom.h Xfuncproto.h Xlib.h Xutil.h cursorfont.h keysym.h keysymdef.h license.terms
RS 2007-10-01: Until odyce supports critcl::cproc properly, the following little "source code generator" can help (or serve as starter) in simple cases:
proc cproc {name argl rtype body} {
set n 0
set cbody "critcl::ccommand $name {dummy interp objc objv} \{\n"
foreach {type var} $argl {
append cbody "$type $var;\n"
incr n
}
append cbody "if (objc!=$n) Tcl_WrongNumArgs(interp,$n,objv,\"int\");\n"
set n 0
foreach {type var} $argl {
append cbody "if (Tcl_Get[tmap $type]FromObj(interp,objv\[[incr n]\],&$var)!=TCL_OK) return TCL_ERROR;\n"
}
foreach line [split $body \n] {
if [regexp {return(.+)} $line -> ret] {
set ret [string trimright $ret ";"]
append cbody "Tcl_SetObjResult(interp, Tcl_New[tmap $rtype]Obj($ret));\n"
} else {append cbody $line\n}
}
append cbody "return TCL_OK;\n\}"
puts $cbody
eval $cbody
}
proc tmap type {
switch -- $type {
int {return Int}
default {error "unknown type $type"}
}
}From the specification
cproc fiba {int x int y} int {
return fib(x)+y;
}it produces, and tcc takes it without complaining:
critcl::ccommand fiba {dummy interp objc objv} {
int x;
int y;
if (objc!=2) Tcl_WrongNumArgs(interp,2,objv,"int");
if (Tcl_GetIntFromObj(interp,objv[1],&x)!=TCL_OK) return TCL_ERROR;
if (Tcl_GetIntFromObj(interp,objv[2],&y)!=TCL_OK) return TCL_ERROR;
Tcl_SetObjResult(interp, Tcl_NewIntObj( fib(x)+y));
return TCL_OK;
}However, experimenting with an unknown function name
cproc fiba {int x int y} int {
return cfib(x)+y;
}crashes eTcl with an unknown exception, instead of reporting the problem, as it does with unknown variables.
EH 2007-10-01: From http://www.evolane.com/download/devel/ , you can get a zip archive containing an updated critcl.tcl package (version 0.1.1) and several demos (fib, md5 are Tcl only, LRI uses Tk). Critcl package included into this snapshot fixes several bugs, and provides an emulation for ::critcl::cproc API. More development snapshots will be made available in next days, until eTcl 1.0-rc24 is made available. Feedback and/or submisions for inclusion into this critcl emulation are welcome.
cche 2007-10-01: Tested the fib example on an older HTC9500 with the "pocketpc 2000/2002 no official support" and the odyce/critcl version was 190x faster than the pure tcl version. This is a really nice addition to eTcl. Thanks a lot!
RS: Many thanks from me too! With the updated critcl.tcl as from above, the fib example boils down to
#package require critcl
source [info script]/../critcl.tcl
# Un bout de code C a integer
critcl::ccode {
static int fib(int n) {
return n <= 2? 1 : fib(n-1) + fib(n-2);
}
}
critcl::cproc fib {int n} int {
return fib(n);
}
#-- version Tcl
proc fib0 n {
if {$n <= 2} {return 1}
expr {[fib0 [expr {$n-1}]] + [fib0 [expr {$n-2}]]}
}
puts tcc:[fib 30],[time {fib 30}]
puts Tcl:[fib0 30],[time {fib0 30}]or even ...
#package require critcl
source [info script]/../critcl.tcl
# Un bout de code C a integer
critcl::ccode {
static int fib(int n) {return n <= 2? 1 : fib(n-1) + fib(n-2);}
}
critcl::cproc fib {int n} int {return fib(n);}
#-- version Tcl
proc fib0 n {
expr {$n <= 2? 1 : [fib0 [expr {$n-1}]] + [fib0 [expr {$n-2}]]}
}
Another example (if we had no [string reverse], i.e. before 8.5):
critcl::ccode {
static size_t strlen(char* s) {
size_t n = 0;
while(*s) {s++; n++;}
return n;
}
}
critcl::cproc strrev {char* s} char* {
char *cp0, *cp1, t;
for (cp0=s, cp1=s+strlen(s)-1; cp1 > cp0; cp0++, cp1--) {
t=*cp0; *cp0=*cp1; *cp1=t;
}
return s;
}
puts [strrev "A man, a plan, a canal - Panama"]
# -> amanaP - lanac a ,nalp a ,nam AI could include <string.h>, but strlen() was still unknown, so I had to roll my own above. eTcl must sure be linked with some lib that provides strlen()...
EH: Odyce uses internally an hash table initialized at startup with all system symbols allowed in C codes. All Tcl/Tk symbols are there, on all platforms (so you could rewrite code above to use DString instead of char*, and use Tcl_DStringLength()). For system symbols (e.g. the one from libc), I took some time defining nearly all available symbols on Linux version of eTcl/Odyce. However, Win32 and WinCE initial tables has been filled only with a very minimal set for testing. This is of course only due to the early stage of the project, and will be fixed very quickly. For information, together with all public Tcl and Tk symbols, odyce version embedded into eTcl 1.0-rc23 for Win32/WinCE declares symbols fprintf, fputc, fputs, fread, fwrite, fscanf, fseek, ftell, ferror, fflush, fgetc, feof, fdopen, fgets, printf, snprintf, sprintf, vfprintf, vprintf, vsnprintf, vsprintf, fileno, putc, putchar, gets, malloc, free, memchr, memcmp, memcpy, memset, memmove, strcmp, tolower and toupper. Next version will provide same symbols table on all architectures.
RS: Further playing with the new critcl.tcl, I propose the following changes:
/Tcl/etcl $ diff critcl.tcl odyce-demo/lib/
Include tcl.h once and for all:
36c36 < set critcl(code) "\#include <tcl.h>\n" --- > set critcl(code) ""
Avoid syntax errors on one-liners like: critcl::ccode {int foo = 42;}
64c64 < append critcl(code) $code\n --- > append critcl(code) $code
Save time by not including tcl.h many times
176c176 < #append code "\#include <tcl.h>" "\n" --- > append code "\#include <tcl.h>" "\n"
One good debugging aid is, in the eTcl console,
% set critcl::critcl(code)
When a syntax error is reported, the given line number minus 3 shows the place:
...
{<string>:32: 'x43' undeclared}
3% lindex [split $::critcl::critcl(code) \n] 29
int foo = x43; Latest news: eTcl and in particular odyce install and run well on my ancient Windows 95 machine (which Tclkit didn't a year or so ago). Of course, at 200MHz you'd better take your time, but odyce is still 286 times faster than Tcl itself:
tcc:832040,201354 microseconds per iteration Tcl:832040,57668098 microseconds per iteration
Big thanks to EH for his robust engineering :^)
Another use is to explore the C side of Tcl interactively, e.g.
20% critcl::cproc gethost {} char* {return Tcl_GetHostName();}
21% gethost
rshomeI don't know where that name comes from (it's not in env), but it fits :^) Or this:
24% critcl::cproc sigid {int i} char* {return Tcl_SignalId(i);}
25% sigid 0
unknown signal
26% sigid 1
unknown signal
27% sigid 2
SIGINT
28% sigid 3
unknown signal
29% sigid 4
SIGILL
30% sigid 5
unknown signal
...
31% critcl::cproc sigmsg {int i} char* {return Tcl_SignalMsg(i);}
32% sigmsg 2
interrupt
33% sigmsg 4
illegal instruction
34% sigmsg 8
floating-point exceptionThis one took me a long time (and much help from the Tcl chatroom) until I got it right, though the spec is simple - increment a given variable by 1:
critcl::cproc myincr {Tcl_Interp* interp char* varname} ok {
Tcl_Obj* var = Tcl_GetVar2Ex(interp,varname, NULL, TCL_LEAVE_ERR_MSG);
int i;
if(var == NULL) return TCL_ERROR;
if(Tcl_GetIntFromObj(interp, var, &i) != TCL_OK) return TCL_ERROR;
Tcl_SetVar2Ex(interp, varname, NULL, Tcl_NewIntObj(i+1), 0);
return TCL_OK;
}
#-- testing with a shared value:
set foo 42
set bar $foo
myincr foo
puts foo:$foo,bar:$barBack to something simple. Even a very plain addition can gain by a factor of >2:
critcl::cproc tcc_add {double a double b} double {return a+b;}
proc tcl_add {a b} {expr {$a + $b}}
puts add/tcc:[time {tcc_add 12.3 34} 10000]
puts add/tcl:[time {tcl_add 12.3 34} 10000]gives:
add/tcc:21.5248 microseconds per iteration add/tcl:49.6773 microseconds per iteration
RS Without documentation, some experimenting gives insights:
% set cc [::odyce::odyce] odyce4 % $cc -type foo bad type "foo": must be memory, exe, dll, or obj
So odyce in principle allows compiling to file. But I could not find out what libs are to be linked, as there is an include but not a lib dir in the VFS:
% $cc -type exe -include [pwd] -o /Tcl/etcl/a.exe "#include <stdio.h>\nvoid main(){printf(\"hello, world!\\n\");}"
{<string>:2: undefined symbol 'printf'}Also, of popular system include files, ctype.h and time.h are not in the VFS.
EH 2007-10-05: EH I have released some unofficial, devel snapshots of eTcl binaries (full version) for linux-x86, win32 and WinCE only, with many improvements and fixes to embedded Odyce package and critcl emulation. Get it at http://www.evolane.com/download/devel/etcl-dev-20071005.zip . Too many changes to list them here. Most important ones are a larger set fo headers embedded into VFS, and automatic resolution of system symbols for WinCE and Win32 (e.g. strlen is automatically available, but also all kernel32 symbols for Win32 API on Win32). Since most C code are now supported out of the box, also added a demo to illustrate how easy it is to extend eTcl dynamically by compiling and loading automatically a TEA package (tDOM has been choosen) using Odyce. To test it, start eTcl binary for your platform, type "package require tdom" and tdom package will be compiled by Odyce and loaded into interpreter. This takes 0.7s on a linux box, but 3s with Win32 version on same hardware, so there is still something to improve on Win32. A universal tdom package in an 300kb large kit! To learn from it, unzip file lib/tdom-0.8.2-1.etk (yes, eTcl kits are just plain ZIP archives) and have a look at tdom.tcl file (expat and generic subdirectories are just exact copies of tDOM sources distribution).
RS: got up early to test, on the old 200 MHz W95 box :^)
1% time {package require tdom}
50929377 microseconds per iteration
2% packa re tdom
0.8.2
3% dom parse <foo><bar>hello</bar></foo>
domDoc00F72AD0
4% set root [domDoc00F72AD0 documentElement]
domNode00F74430
5% $root asXML
<foo>
<bar>hello</bar>
</foo>So, first test very successful (if you can spare 51 seconds)... Win XP:
1% time {package require tdom}
4352723 microseconds per iteration EH: 3-4s on WinXP while 0.6-0.7 on Linux (same machine), so there is really something wrong with Win32. Even on a 200Mhz machine, I guess it is possible to divide time per a factor 10 or so.
Re header files, there are now:
22% cd D:/etcl2/etcl-dev-20071005/etcl-win32.exe/.etcl/vfs/extensions/odyce0.1/include/win32 23% glob * winapi limits.h stddef.h stdio.h stdarg.h dos.h varargs.h stdint.h float.h stdbool.h fenv.h string.h fcntl.h process.h io.h sys tchar.h assert.h math.h stdlib.h direct.h setjmp.h share.h excpt.h wctype.h ctype.h wchar.h errno.h signal.h dirent.h dir.h mem.h unistd.h time.h inttypes.h conio.h locale.h values.h malloc.h _mingw.h memory.h 24% cd ../generic; glob * stddef.h stab.h limits.h stdio.h bits stdarg.h float.h varargs.h dlfcn.h stdbool.h fcntl.h string.h stdlib.h math.h assert.h ctype.h features.h locale.h malloc.h memory.h 25% cd ../tcltk/; glob * tclDecls.h tkDecls.h tcl.h tk.h tclPlatDecls.h X11 tkIntXlibDecls.h
EH 2007-10-06 has made another devel snapshot available at http://www.evolane.com/download/devel/etcl-dev-20071006.zip . Performance issue on Win32/WinCE systems has been fixed, that is tDOM demo is compiled in 0.6s in both Linux and WinXP OS (same hardware), vs. 5-6s with previous snapshot. WinCE port is fully functionnal, all missing symbols have been added, tDOM compilation works fine. Profiling shows performance could be even more improved by caching headers (especially those stored compressed in VFS). To be continued...
RS: Very good news! But, Eric, just curious: how did you manage to cut compilation time on Windows by a factor of 5..10?
EH: Because overhead wasn't due to compilation itself, but to too many I/O operations when searching for includes. On Linux, OS is efficiently managing filesystem cache, but on Win32, this leads to significant slow down. Ratio (5..10) depends more on disk/filesystem eTcl was run on than on CPU.
ramsan: congratulations. This is a very nice addition to etcl. I wanted to use tdom in wince and now it is possible!! One question: is it possible to compile just once and keep some kind of cache like a DLL or símilar to avoid compiling every time that we use the library? EH Soon, hopefully before next (rc24) official release. Saving code into DLL instead of running in memory is nearly ready. ALso note that tDOM is juts one example among many others. Once Odyce is stabilized, It should be possible to extend eTcl with virtually any TEA compliant Tcl/Tk extension from an architecture-agnostic kit. Anyone is welcome to submit kits for other popular extensions.
ramsan: The following example, crashes the program in wince in the etcl-dev-20071006. It works perfectly on Windows:
time { package require tdom }
set data "<internal_data></internal_data>"
set doc [dom parse $data]
set root [$doc documentElement]
set key non/existantkey
set domNodes [$root selectNodes $key typeVar]EH 2007-10-12: Code get stabilized, more packages (itcl, itk, xotcl, etc...) supported and passing all their tests, so tdom problem above should hopefully get fixed in next build. Start a build session, binaries should be available later today. For those wishing to try from build it themselves, sources have been pushed into our public subversion repository, and can be browsed (and downloaded) at http://www.evolane.net/viewvc/odyce/ .
APN 2008-06-20: Is Odyce going to be available as a standalone package (for Windows, not WinCE) outside of eTcl? 2008-06-21
EH: Since sources are available from http://www.evolane.net/viewvc/odyce/ under LGPL license, one may easily generate a loadable DLL for Win32. If you believe there is interest/demand for a standalone binary distribution, I can add a section on evolane's site for download (in a way similar to what has been done for pixane extension, available both as a core component of eTcl and a standalone extension). A lazy and dynamic compilation of your TWAPI extension with Odyce would indeed be a nice application.
APN: That was the thought. Looking at it a bit closer though, there are some logistical/licensing issues because TWAPI used a lot of Windows SDK header files which are not freely distributable.
EH: Doesn't the Mingw headers (which are public domain, see README.w32api) cover most (all?) declarations of WIN32 API required by TWAPI?
APN: Hmm, I didn't know about the Mingw headers. I will take a look at some point, debating if there is an advantage in using either odyce or ffidl as a replacement for the swig layer.
2008-08-14 MSA Problem with WM5
1% package req critcl
0.1.3
2% critcl::cproc pr {} ok {double a=2.0; return TCL_OK; }
3% pretcl disappears with no message. If critcl::cproc pr {} ok {int a=2;return TCL_OK; } - OK. Same for float.