Speed Tables

https://www.tcl-lang.org/community/tcl2007/proceedings/databases/speedtables-paper-2.pdf

Speed tables provides an interface for defining tables containing zero or more rows, with each row containing one or more fields. The speed table compiler reads a table definition (similar to a C struct definition) and generates C code to create and manage tables where the rows are such structures, producing a set of C access routines and a C language extension for Tcl to create, access and manipulate those tables. It then compiles the extension, links it as a shared library, and makes it loadable on demand via Tcl's "package require" mechanism.

Speed tables are well-suited for applications for which this table/row/field abstraction is useful, with row counts from the dozens to the tens of millions, for which the performance requirements for access, search and/or update frequency exceed those of the available SQL database, and the application does not require “no transaction loss” behavior in the event of a crash.

Current development: https://github.com/flightaware/speedtables

Old project link (historic): http://sourceforge.net/projects/speedtables

APN 2012/10/17 I have a forked version at https://github.com/apnadkarni/speedtables intended for porting to Windows and hopefully merged upstream at some future (stable) point. Make sure to work off the windows branch.

jima (2011-01-27)

I have been able to set up Speed Tables for my Linux box. As the authors stated in their documentation their main focus has been BSD OS. I have encountered some tricky points in my way and I wanted to reflect here my steps just in case this could be of value to other people interested in this project.

I hope this will also help authors and contributors to find a better way of packaging this so as to cover BDS and Linux in the same go. Unfortunately I don't know anything about autotools to help in a better way than simply pointing to the differences between both OS'es.

Thanks to all people in the Tcl chat that helped me in my way to success.

The original scheme of things:

  • In Speed Tables one can write a Tcl script that is the seed to create a set of (mainly) C files in a build directory. These files are compiled and built into a Tcl extension. This building process is done by a Tcl script which invokes the compiler and linker by means of exec command.

Main difficulties in my Linux (debian lenny) box:

  • (I) In general, some options (like using tclsh8.4) seem to be hardcoded in the files that guide the building process.
  • (II) The compiler seems to be writing warning to stderr. This causes exec command to stop execution.
  • (III) Speed Tables use C function qsort_r that has been developed originally in BSD. This function can be found in the C standard library for Linux but, alas, with a different order used for its parameters (as seen on the web ).
  • (IV) Tcl would not find the entry point for the extension.

Workarounds:

For (I):

In file ctables/Makefile I had to set PREFIX to my install dir:

  • PREFIX=/mnt/tis/ins/src

In file ctables/config.tcl I tweaked the following directives:

  • set showCompilerCommands 1
  • At the end of the file: set withPgtcl 0

In file ctables/mkConfig.sh I did:

  • tcl_versions="8.6"
  • tcl_prefixes="/mnt/tis/ins/src/lib"

By the way, instead of letting the Makefile produce the sysconfig.tcl from config.tcl I was somehow forced to do it manually: mkConfig.sh > sysconfig.tcl.

After the generation of sysconfig.tcl I tweaked it to:

  • Add to the list of define directives: -DCTABLE_NO_SYS_LIMITS=1
  • Declare the linker in a proper way: set sysconfig(ld) {gcc -shared}

Last but not least:

In file ctables/tests/Makefile I just told the system to use my tclsh instead of a hardwired one:

  • TCLSH=tclsh

For (II)

In file ctables/gentable.tcl just tweak a proc called myexec:

  • eval exec -ignorestderr -- $command

For (III)

First, I had to update my system to have a libc that included function qsort_r by grabbing latest version of it from debian unstable.

In file ctables/ctable.h:

  • I had to load the ugly __USE_GNU directive ahead of stdlib to allow loading of qsort_r.
 #define __USE_GNU
 #include <stdlib.h>
  • I had to distinguish between the loading of the BDS version or the Linux one: 
 #ifdef JIMA_QSORT_R_BSD     
    int (*sort_compare) (void *clientData, const void *pointer1, const void *pointer2);
 #else
    int (*sort_compare) (const void *pointer1, const void *pointer2, void *clientData);
 #endif

In file ctables/ctable_search.c:

This is where the actual call to qsort_r is done so I had to flag here as well:

 #ifdef JIMA_QSORT_R_BSD     
      qsort_r (search->tranTable, search->matchCount, sizeof (ctable_HashEntry *), &search->sortControl, creator->sort_compare);
 #else
      qsort_r (search->tranTable, search->matchCount, sizeof (ctable_HashEntry *), creator->sort_compare, &search->sortControl);
 #endif

Not only parameters for qsort_r have different orders for both OS'es. The comparison function being passed to qsort_r have different orders for its parameters too.

This I hacked in a very dirty way in file: ctables/gentable.tcl at the point of variable sortCompareHeaderSource:

 // JIMA_NON_BSD
 int ${table}_sort_compare(const void *vRow1, const void *vRow2, void *clientData)

For (IV)

In file ctables/init-exten.c-subst:

  • Change mentions to EXTERN int by int DLLEXPORT.

TODO's

Obviously this is a very dirty set of workarounds... a better way to do it would require probably find the best point to insert C preprocessor directives and let all code be governed properly by them.

I haven't yet tested the install procedure and checked on shared memory goodies in a detailed way.

Category Package
Updated 2012-10-17 13:44:37