Version 57 of Extending Tcl

Updated 2011-09-16 09:18:24 by dkf

The goal of this document is to teach you how to extend Tcl. It covers the various methods of doing this, and the pros and cons of each method.


Please note that after appropriate parts of this document are finalized, they should be incoroprated into TIP 66. http://www.tcl.tk/cgi-bin/tct/tip/66.html


Reasons for extending Tcl:

  • to expose an API function to Tcl
  • to increase the speed of a CPU intensive calculation

Methods of extending Tcl:

Overview of the methods: With executable modules (EM), exposure of the Tcl C API is not needed. Building an executable module is generally easier than writing a C extension for Tcl. The Tcl exec command can be used to start a module and send it initial data. For most EM however the open command is more useful, because you can use it to create a two-way pipeline. Debugging is easier with an EM, because Tcl isn't involved in the function of the module. It's also reasonable to assume that an EM works with most scripting languages, and is thus not dependent on Tcl.

The Tcl C API is powerful and for something like a new data type for a tree, threads, or other things that can not be easily done with executable modules it's a good solution.

Critcl is a newcomer to the world of Tcl. It compiles an extension on the fly based on C code embedded in a Tcl script. It can't be used for all extension work, but for many things it's good. [todo findout more about how well it works]

A variation on the methods employed by Critcl allows Fortran programmers to write extensions for Tcl - it is called Critclf (not very imaginative :), I admit) and is currently under development. Contact AM for more information.

Mktclapp has an API for writing extensions. It can be used to generate a stand-alone application by generating C code, or a load'able shared object library. It can also be used to convert a Tcl script or scripts into C code for compilation with a C compiler, so its use is beyond just extensions.

SWIG automates the task of generating an extension. It can generate an interface for Tcl and other languages, which can then be compiled. [How well does it work? What is it good at doing?]

xWizard can generate template C/C++ code for Tcl extension. It is a GUI program written by pure Tcl/Tk so it works for cross-platform. More details in http://www.neatware.com/myrmecox/professional/wizard.html


Executable Modules (EM)

An EM is one of the easiest methods for extending Tcl. It's possible to build an EM from other Tcl scripts, Perl scripts [but why would anyone want to do that?], or some other language [surely this third option is the most common?], but sometimes a program usable as an EM already exists and then one doesn't even have to create it. An EM typically communicates with a Tcl pipe created with open. Tcl supports binary data over two-way pipes, via fconfigure and -translation binary, so theoretically any data can be transferred. It's also possible to use shared memory.

Pro:

  1. Any language may be used for an EM.
  2. Debugging may be easier, because programs have an extra memory boundary.
  3. Testing may be easier, because the program can be tested alone (interactively in many cases).

Con:

  1. Performance may be an issue if lots of I/O is required.
  2. Shared state can be a problem. A possible solution is shared memory.
  3. Complex/structured data can be a problem. Tcl handles serializing very well, but other languages are often weak in this area.
/*
 * By George Peter Staplin
 * This is version 2 of a simple executable-module written in C.
 * This is simply a demonstration.
 */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <limits.h>
 
#define SEPFMT " \t\n"
 
void get_input (char *buf, size_t s) {
  if (NULL == fgets (buf, s, stdin)) {
   if (feof (stdin)) {
    exit (EXIT_SUCCESS);
   } 
   exit (EXIT_FAILURE);
  }
}

int expect_int (void) {
  char *t = strtok (NULL, SEPFMT);
  char *end;
  long l;
 
  if (NULL == t) {
   fprintf (stderr, "premature end of token stream.\n");
   exit (EXIT_FAILURE);
  }
 
  l = strtol (t, &end, 10);
 
  if ((LONG_MIN == l && ERANGE == errno)
    || (end < (t + (strlen (t) - 1)))) {
   fprintf (stderr, "invalid integer '%s'\n", t);
   exit (EXIT_FAILURE);
  }
 
  return (int) l;
}

int main (int argc, char *argv[]) {
  char buf[1024];
  char *tok;
 
 again:
  get_input (buf, sizeof (buf));
 
  tok = strtok (buf, SEPFMT);
 
  if (NULL == tok || 1 != strlen (tok)) {
   fprintf (stderr, "invalid operator: '%s'\n", tok);
   exit (EXIT_FAILURE);
  }
 
  switch (tok[0]) {
   case '+':
    printf ("%d\n", expect_int () + expect_int ());
   break;
  
   case '-':
   {
    int a, b;
    a = expect_int ();
    b = expect_int ();
    printf ("%d\n", a - b);
   }
   break;
 
   default:
    fprintf (stderr, "unknown operator: '%c'\n", tok[0]);
    exit (EXIT_FAILURE);
  }
  goto again;
 
  return EXIT_FAILURE;
}

To use the code above as a module we will use a two-way pipe. Tcl uses the | character in the open command to create a pipe. We will open the pipe for w+ also known as read-and-write mode. The gets command is used to retrieve the result. The read command could also be used, although it would require a fixed-length format, non-blocking I/O, or a read of a single character until \n is reached.

  #!/bin/tclsh8.4
  
  set ::mathModule [open {|./modmath} w+]
    
  proc modMath {args} {
   puts -nonewline "$args is "
   puts $::mathModule $args
   flush $::mathModule
   return [gets $::mathModule]
  }
  
  proc main {} {
   puts [modMath + 200 300]
   puts [modMath - 5600 1243]
   puts Done
  }
  main

Tcl C API

The Tcl C API is quite powerful. There are at least 2 approaches to extending Tcl via the C API. A new tclsh-like shell can be created, or a loadable extension can be loaded into an interpreter.

Our first example will be a new interactive shell that performs simple addition. This will teach you how to create a Tcl interpreter, initialize Tcl, and create a simple command for addition.

Pro:

  1. The C API is powerful and some things can be done that are impossible to do with an EM.
  2. The C API can be used as a portable abstract layer for I/O, and other uses.
  3. Performance is generally excellent -- provided that you choose the right algorithms.

Con:

  1. Tcl_Obj management can be tricky for beginners. Bad Tcl_Obj reference counting practice can result in crashes in other parts of Tcl that can be difficult to diagnose.
  2. C is the only language binding for the Tcl API (although "C" in this case typically includes derivatives such as C++), unless you count some tools that use assembly language or things like ffidl.
  3. The C API may change overtime, and tends to change more than the Tcl script-level interface.
  4. Depending on what part of the API you use or require (for an extension library), you may find that Tclkit doesn't work, because the tclInt.h functions used are not in the stubs table.
  #include <stdio.h>
  #include <stdlib.h>
  #include <tcl.h>
  
  int AddObjCmd (ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) {
    long n1;
    long n2;
    Tcl_Obj *res;
  
    /*The command name is objv[0] so 3 arguments are expected.*/
    if (3 != objc) {
      Tcl_WrongNumArgs (interp, 1, objv, "n1 n2");
      return TCL_ERROR;
    }
  
    if (TCL_OK != Tcl_GetLongFromObj (interp, objv[1], &n1)) {
      /*The error result should be set by Tcl_GetLongFromObj.*/
      return TCL_ERROR;
    }
  
    if (TCL_OK != Tcl_GetLongFromObj (interp, objv[2], &n2)) {
      return TCL_ERROR;
    }
    
    res = Tcl_NewLongObj (n1 + n2);
  
    Tcl_SetObjResult (interp, res);
  
    return TCL_OK;
  }
  
  int main (int argc, char *argv[]) {
    Tcl_Interp *interp;
  
  
    /*
     * This finds Tcl's library files and performs some initialization.
     */
    Tcl_FindExecutable (argv[0]);
  
    interp = Tcl_CreateInterp ();
   
    if (TCL_OK != Tcl_Init (interp)) {
      fprintf (stderr, "Tcl_Init error: %s\n", Tcl_GetStringResult (interp));
      exit (EXIT_FAILURE);
  
    }
  
    Tcl_CreateObjCommand (interp, "+", AddObjCmd, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL);
  
    while (1) {
      char cmd[1024];
    
      fgets (cmd, sizeof (cmd), stdin);
  
      if (TCL_OK != Tcl_Eval (interp, cmd)) {
        fprintf (stderr, "error: %s\n", Tcl_GetStringResult (interp));
        continue;
      }
  
      printf ("result is: %s\n", Tcl_GetStringResult (interp));
    }  
  
    return EXIT_SUCCESS;  
  }

Links:

  1. SampleExtension - the official example
  2. BOOK Practical Programming in Tcl and Tk, Third edition - Chapter 44 on the Tcl C API is available on-line [L1 ].
  3. http://www.cs.man.ac.uk/%7Efellowsd/tcl/funcproc.c
  4. ftp://www.tcl.tk/pub/tcl/misc/example.tar.gz
  5. http://www.iki.fi/mjk/files/test.c.txt

[explain how to build a load'able extension]
[explain how to build an export list for a DLL]
[explain how to link using MSVC++]


Tcl C API using C++

Using C++ with Tcl is similar to using it with C. There are a few tricks to using methods within a class. Within the class we create a wrapper which calls a static function within the class.

  #include <stdio.h>
  #include <stdlib.h>
  
  /*This is specifically for C++.*/
  extern "C" {
    #include <tcl.h>
  }
  
  /*Thanks to Kevin Kenny for his help with this.*/
  
  class Math {
    public:
    static int AddObjCmd (ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) {
      return (reinterpret_cast<Math*>(clientData))->AddMethod (interp, objc, objv);
    }
    int AddMethod (Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]);
  };
  
  int Math::AddMethod (Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) {
    long n1;
    long n2;
    Tcl_Obj *res;
  
    /*The command name is objv[0] so 3 arguments are expected.*/
    if (3 != objc) {
      Tcl_WrongNumArgs (interp, 1, objv, "n1 n2");
      return TCL_ERROR;
    }
  
    if (TCL_OK != Tcl_GetLongFromObj (interp, objv[1], &n1)) {
      /*The error result should be set by Tcl_GetLongFromObj.*/
      return TCL_ERROR;
    }
  
    if (TCL_OK != Tcl_GetLongFromObj (interp, objv[2], &n2)) {
      return TCL_ERROR;
    }
    
    res = Tcl_NewLongObj (n1 + n2);
  
    Tcl_SetObjResult (interp, res);
  
    return TCL_OK;
  }
  
  int main (int argc, char *argv[]) {
    Tcl_Interp *interp;
  
  
    /*This finds Tcl's library files and performs some initialization.*/
    Tcl_FindExecutable (argv[0]);
  
    interp = Tcl_CreateInterp ();
  
  
    if (TCL_OK != Tcl_Init (interp)) {
      fprintf (stderr, "Tcl_Init error: %s\n", Tcl_GetStringResult (interp));
      exit (EXIT_FAILURE);
    }
  
    Math *inst = new Math;
  
    Tcl_CreateObjCommand (interp, "+", inst->AddObjCmd, (ClientData) inst, (Tcl_CmdDeleteProc *) NULL);
  
    while (1) {
      char cmd[1024];
    
      fgets (cmd, sizeof (cmd), stdin);
  
      if (TCL_OK != Tcl_Eval (interp, cmd)) {
        fprintf (stderr, "error: %s\n", Tcl_GetStringResult (interp));
        continue;
      }
  
      printf ("result is: %s\n", Tcl_GetStringResult (interp));
    }  
  
    return EXIT_SUCCESS;  
  }

[perhaps someone more knowledgeable about C++ could improve this]

SLB I'll just comment that the 'tricks' are dangerous. Compiling the above code with Sun's compiler reports:

  "tclmain.cpp", line 61: Warning (Anachronism): Formal argument proc of type extern "C" int(*)(void*,Tcl_Interp*,int,Tcl_Obj*const*)
  in call to Tcl_CreateObjCommand(Tcl_Interp*, const char*, extern "C" int(*)(void*,Tcl_Interp*,int,Tcl_Obj*const*), void*,
   extern "C" void(*)(void*)) is being passed int(*)(void*,Tcl_Interp*,int,Tcl_Obj*const*).

I don't have a reference to the relavent clause in the C++ standard to hand but warnings about portability problems from treating C and C++ function pointers as interchangeable are widespread. See for example [L2 ]

Links:

  1. ftp://ftp.forwiss.uni-passau.de/pub/Os/Unix/languages/tcl/extensions/tkmin/ [Does this URL work, just very slowly, or is it no longer valid?]
  2. http://www.uni-frankfurt.de/%7Efp/Tcl/tcl-c++.txt [This URL doesn't appear to be working]
  3. http://prdownloads.sourceforge.net/incrtcl/sampleItclExtension1.1.zip?download

Tcl C API using Objective-C

To expose an Objective-C class to Tcl we can create a wrapper in much the same was as was done with C++ earlier.

Compile this code using something like:

gcc tcl_objc.m -lobjc -I/usr/pkg/include -L/usr/pkg/lib -ltcl84 -lm -lpthread -Wl,-rpath -Wl,/usr/pkg/lib

tcl_objc.h:

 /*
  * By George Peter Staplin
  * This is version 2, and much better than the previous code I put here...
  */
 #include <stdio.h>
 #include <stdlib.h>
 #include <tcl.h>
 #include <objc/Object.h>
 
 
 @interface Adder : Object {
  /* This is just a pointless instance variable. */
  int lastresult; 
 }
 - (int) add:(Tcl_Interp *)interp x:(Tcl_Obj *)x  y:(Tcl_Obj *)y;
 @end
 int AdderWrapper (ClientData, Tcl_Interp *, int, Tcl_Obj *CONST[]);
 
'''tcl_objc.m:'''
 #include "tcl_objc.h"
 
 @implementation Adder
 
 - (int)add:(Tcl_Interp *)interp x:(Tcl_Obj *)x y:(Tcl_Obj *)y {
  int a, b;
 
  if (TCL_OK != Tcl_GetIntFromObj (interp, x, &a))
   return TCL_ERROR;
  
  if (TCL_OK != Tcl_GetIntFromObj (interp, y, &b))
   return TCL_ERROR;
 
  Tcl_SetObjResult (interp, Tcl_NewIntObj (lastresult = a + b));
 
  return TCL_OK;
 }
 @end
 
 int AdderWrapper (
   ClientData cdata, 
   Tcl_Interp *interp,
   int objc, 
   Tcl_Obj *CONST objv[]) 
 {
  id inst = (id) cdata;
 
  if (TCL_OK != [inst add: interp x: objv[1] y: objv[2]]) {
   return TCL_ERROR;
  }
 
  return TCL_OK;
 }
 
 int main (int argc, char *argv[]) {
  Tcl_Interp *interp;
  id inst;
 
  Tcl_FindExecutable (argv[0]);
 
  interp = Tcl_CreateInterp ();
 
  if (TCL_OK != Tcl_Init (interp)) {
   fprintf (stderr, "Tcl_Init error: '%s'\n", Tcl_GetStringResult (interp));
   return EXIT_FAILURE;
  }
 
  inst = [Adder new];
 
  Tcl_CreateObjCommand (interp, "add", AdderWrapper, (ClientData) inst,
    (Tcl_CmdDeleteProc *) NULL);
 
  if (TCL_OK != Tcl_Eval (interp, "add 200 123")) {
   fprintf (stderr, "Tcl_Eval error: '%s'\n", Tcl_GetStringResult (interp));
   return EXIT_FAILURE;
  }
 
  printf ("Success: the result of add 200 123 is: %s\n", Tcl_GetStringResult (interp));
 
  return EXIT_SUCCESS;
 }

15jan03 jcw - Your "add" example above in Critcl:

  package require critcl
  critcl::cproc addup {long a long b} long {
    return a + b;
  }
  interp alias {} + {} addup
  puts "123 + 864 = [+ 123 864]"

If you have tclkit and the critcl starkit, do:

  $ critcl add.tcl 
  123 + 864 = 987
  $

If you want to turn this into a loadable package, use this code instead:

  package provide add 1.0
  package require critcl
  critcl::cproc addup {long a long b} long {
    return a + b;
  }

Then use the -pkg flag:

  $ critcl -pkg add.tcl 
  Source: add.tcl 
  Library: add.so
  Package: /home/jcw/lib/add
  $

The result is a self-contained auto-generated package:

  $ ls -lR lib/add/
  lib/add/:
  total 9
  drwxr-xr-x    2 jcw      users         104 Jan 15 18:52 Linux-x86
  -rw-r--r--    1 jcw      users        1506 Jan 15 18:52 critcl.tcl
  -rw-r--r--    1 jcw      users          64 Jan 15 18:52 pkgIndex.tcl

  lib/add/Linux-x86:
  total 12
  -rwxr-xr-x    1 jcw      users        4660 Jan 15 18:50 add.so
  -rw-r--r--    1 jcw      users         157 Jan 15 18:52 critcl.tcl
  $

Somewhere, RS wrote some code that talks about embedding C code into your Tcl scripts... C code generators


What about extending via mechanisms like CORBA, remote RPC, etc.? Are those kinds of ways, where you are invoking code over a network, considered on topic here?

George Peter Staplin Any kind of RPC would be on topic. On a separate page linked to from here, or within this page. RPC is an interesting topic, and I look forward to hearing about it. :)

LV Inventory of IPC methods covers a variety of means of communicating between two "executable modules". However, I don't personally think of that as being a method of "extending" Tcl. "Extending Tcl", to me, means Tcl having new commands / APIs at its disposal.


How to invoke a C function from Tcl