Embedding TCL program in DLL

HaO 2020-05-29: Embed a program written in Tcl in a DLL.

The Task

A program written in Tcl. A customer wants to embedd the program in its own application and requires a DLL for a Microsoft compiler.

Additional requirements:

  • only one binary package: tdom
  • interface has the following calls: init, evaluate, clear
  • no event loop, no file events, no sockets, no threads
  • nevertheless, leave it generic enough to be relatively universal and extendable. So include dde, registry, Thread.
  • try to get everything in one DLL, no additional files or dependent DLL's.

I am a medium level programmer and totally new to this subject.

Credits

I want to start by all the people helped within the journey:

KitDLL

KitDLL is binary distribution of a dll with an embedded tcl & bundles.

KitCreator/KitDLL contains the following options:

  • Starkits: read a file (the binary or dll) and locate a metakit file data base and mount it to the Tcl interpreter
  • ZIPKit: same as Starkits, but using a zip archive instead the metakit
  • C-VFS: includes scripts in a C source at compile time and allows to access them on runtime.

All those options were added to the Starkit framework still containing many bug-fixes for Tcl 8.4/5 and support for old platforms like Windows-CE.

The build system downloads many packages, applies provided patches, compiles them and packs all scripts in the C-VFS and statically binds all binary extensions.

I tried KitDLL binary from [L1 ]. When I download the binary distribution, copied the tclsh.exe beside the binary and started it, I got "<sys/stat.h> not compatible with VC". It is cross-compiled on Linux for Windows. I already tried MingW and those distributions and always ran into subtile bugs, like inexact calculations, wrong colors in Widgets etc. Thus, I decided to compile on my own and use a Microsoft compiler.

I am only interested in the C-VFS part and not in ZIP-VFS or MKVFS.

Embedding Tcl

Embedding Tcl consists of the following steps:

  • Call Tcl_FindExecutable(NULL) to set a global variable with the current executable file path. In case of a DLL, this is the path to the executable using the dll. This is very important for starkits and zib-vfs, as those file systems want to open this file to find the virtual file system. This does not work for a DLL, as the path to the calling exe, and not to the dll is provided.
  • Call Tcl_CreateInterp() to create a Tcl interpreter.
  • Optionally register a script to run in Tcl_Init() (source init.tcl) by TclSetPreInitScript(). This is an internal Tcl library function. This is used to setup the vfs.
  • Call Tcl_Main() and pass a custom AppInit procedure. AppInit() typically initializes statically linked packages.

Internally, Tcl_Main() (in tcl8.6.10/generic/tclMain.c) will do for each created interpreter:

  • do initialization
  • call the function passed as AppInit() parameter. The function Tcl_Init() may be invoked in AppInit() to find and source an init.tcl file.
  • call Tcl command "exit" at the end of the script and end the process.

As Tcl_Main() does not return, it is not a suited procedure for this task. Its tasks must be done manually.

The vfs is to be initialized for each created interpreter (and thread or slave) including search path setup etc.

After initialization, the interpreter handle may be used to invoke commands in the interpreter using Tcl_Eval*() functions.

At end of life, Tcl_DeleteInterp() is called.

Static Tcl

To build a static Tcl lib, the makefile.vc is used. I decided to use the community edition of MS-VisualC2015. So, first choose in the start menu: "Visual Studio 2015 -> Windows Desktop Command -> VS2015 x86 Native Tools-Eingabeaufforderung". Tcl 8.6.10 source distribution is unzipped in: "C:\test\tcl8.6.10"

Tcl is compiled with the following options:

  • static: to not require a sub-dll
  • staticpkg: to link dde and reg binary packages in the library
  • msvcrt: the final result is a DLL. So, link with the DLL C library, which is typically for a DLL (but unusal for a static build).
  • nostubs: any bundled package should not use stubs. It only takes time and no advantage, as they are linked together anyway.
  • symbols (only for development) - crucial for a learner to be able to trace through everything.
> cd test\tcl8.6.10\win
> nmake -f Makefile.vc release OPTS=static,staticpkg,msvcrt,nostubs,symbols
> nmake -f Makefile.vc install OPTS=static,staticpkg,msvcrt,nostubs,symbols INSTALLDIR=c:\test\tcl8610

Load statically linked packages in interpreter

Statically linked packages are loaded in the interpreter in a different way. First, the start address of the package is passed to the Tcl system:

Tcl_StaticPackage(interp,pkgName, initProc, safeInitProc (=NULL) )

The parameter "interp" may be NULL to register it to all interpreters.

As a second step, each interpreter executes a load command with an empty filename and the announced package name:

load {} pkgName

Compile to link with static Tcl lib

If a custom package should later be linked with the static tcl lib, the define STATIC_BUILD must be defined before the inclusion of "tcl.h". This does not apply if stubs are used (define USE_TCL_STUBS).

Tcl (Win) startup

Here is the program flow of tclsh.exe and the concerned files:

  • main() in tcl8.6.10/win/tclAppInit.c
  • Tcl_Main() in tcl8.6.10/generic/tclMain.c
  • Tcl_AppInit() in tcl8.6.10/win/tclAppInit.c

All those are sources of inspiration for the own solution.

C-VFS

Download: I took the trunk from [L2 ] dated 2020-01-22. The folder "kitsh\buildsrc\kitsh-0.0" contains the kit programs. Change to this folder.

First, the file tree with the scripts to include is set-up in the folder "C:\test\starpack.vfs". I added the tcl lib folder (excluding most encodings and time zones and tests) and the file "boot.tcl" from the upper "kitsh.0-0"-folder.

A C-VFS file containing this file tree is created by (see aclocal.m4):

tclsh dir2c.tcl tcl c:/test/starpack.vfs --obsfucate > cvfs_data_tcl.c

"--obfuscate" folds the data with a constant key to avoid clear text in the DLL. It works and takes considerably more time on packing.

To compile this file, I used a MSVC2015 dll project with the following additional defines:

HAVE_STRING_H;HAVE_STDLIB_H;CVFS_MAKE_LOADABLE,STATIC_BUILD;UNICODE;_UNICODE;TCLKIT_CAN_SET_ENCODING

First I tried using MS-VC6. The dir2c outputs C99 code, not C89. So this patch is applied: [L3 ]. When I included Tcl itself, I ran into memory issues. So I changed to MS VC2015.

Then I ran into a compiler limit (C1091), that string constants may not exceed 65535 bytes. A solution is an unsigned char byte array, with the limit of size_t (0x7CFFFFFF).

This lead to the following patch of dir2c.tcl: [L4 ].

Then, the file "cvfs_data_tcl.c" is successfully compiled.

DLL main

My DLL main program with the 3 exported functions is as follows:

#define STRICT
#define DLL_BUILD
#include <windows.h>
#include <windowsx.h>
#include <tchar.h>
#include <string.h>
#include <stdlib.h>
#include <math.h>

#include <tcl.h>
#include "tclkit.h"

#define ERROR_MESSAGE_MAX 1024
// >>> local Prototypes
__declspec(dllexport) void my_release();
static void TclGetError();

// The pointer to the tcl interpreter
static Tcl_Interp *fg_interp = NULL;

static WCHAR fg_w_error_msg[ERROR_MESSAGE_MAX + 1];

// >>>>> DllMain
BOOL APIENTRY DllMain(HANDLE hModule,
    DWORD  ul_reason_for_call,
    LPVOID lpReserved)
{
    switch (ul_reason_for_call) {
    case DLL_PROCESS_DETACH:
        // >>> DLL is unloaded
        my_release();
    }
    return TRUE;
}
__declspec(dllexport) WCHAR * my_init()
{
    // >> Release eventual present interpreter
    if (fg_interp != NULL)
        my_release();

    // >> from tcl/generic/TclMain.c Tcl_Main()
    Tcl_FindExecutable(NULL);
    fg_interp = Tcl_CreateInterp();
    Tcl_InitMemory(fg_interp);
    Tcl_SetVar2Ex(fg_interp, "tcl_interactive", NULL, Tcl_NewIntObj(0), TCL_GLOBAL_ONLY);

    // from tcl/win/TclAppinit.c Tcl_AppInit()

    // >> from kitInit.c Tcl_Init()
    _Tclkit_Init();
    // >> Source the init.tcl script
    if (Tcl_Init(fg_interp) != TCL_OK) {
        TclGetError();
        my_release();
        return fg_w_error_msg;
    }
    return NULL;
}
void __declspec(dllexport) my_release()
{
    if (fg_interp != NULL)
        Tcl_DeleteInterp(fg_interp);
    fg_interp = NULL;
}
// >>>>> my_cmd
__declspec(dllexport) WCHAR * my_cmd(WCHAR *pwCmd, int *pfError)
{
    WCHAR * pwError;
    Tcl_DString dCmd;
    int Res;
    if (fg_interp == NULL) {
        pwError = my_init();
        if (pwError != NULL)
            *pfError = 1;
        return pwError;
    }
    if (Tcl_InterpActive(fg_interp)) {
        *pfError = 0;
        return NULL;
    }
    // >> Call command
    Tcl_DStringInit(&dCmd);
    Tcl_WinTCharToUtf(pwCmd, -1, &dCmd);
    Res = Tcl_Eval(fg_interp, Tcl_DStringValue(&dCmd));
    Tcl_DStringFree(&dCmd);
    TclGetError();
    *pfError = (Res != TCL_OK);
    return fg_w_error_msg;
}
// >>>>> TclGetError
static void TclGetError()
{
    Tcl_DString dErrorMessage;
    Tcl_DString dErrorMessageWide;
    int ErrorLength;
    Tcl_DStringInit(&dErrorMessage);
    Tcl_DStringInit(&dErrorMessageWide);
    Tcl_DStringGetResult(fg_interp, &dErrorMessage);
    Tcl_WinUtfToTChar(Tcl_DStringValue(&dErrorMessage),
        Tcl_DStringLength(&dErrorMessage), &dErrorMessageWide);
    // Limit length to size of error buffer
    ErrorLength = min(Tcl_DStringLength(&dErrorMessageWide)*2, ERROR_MESSAGE_MAX);
    wmemcpy(fg_w_error_msg, (const wchar_t *)Tcl_DStringValue(&dErrorMessageWide), ErrorLength);
    // set end zero
    fg_w_error_msg[ErrorLength] = 0;
}

The interpreter is initialized, commands may be executed and the interpreter may be released.

c-vfs initialization

The c-vfs initialization happens in the file kitInit.c, while the data and implementation is in the generated file cvfs_data_tcl.c.

The Tclkit initialization is announced with the prototype file "tclkit.h":

void _Tclkit_Init(void);

It is a matter of taste to rename this function and remove the leading "_" which indicates an internal function.

The initialization tcl file cvfs.tcl is prepared for embedding by:

tclsh stringify.tcl cvfs.tcl > cvfs.tcl.h

Now, the file kitInit.c is ready for compilation within the DLL project.

I personally have removed from the file:

  • anything for metakit and zip file system
  • the bugfixes for "FindExecutable()".
  • anything for Tk

So, the following defines are set:

_WIN32;KIT_STORAGE_CVFS;CVFS_MAKE_LOADABLE;TCL_THREADS;TCLKIT_DLL
  • The gcc directive "__attribute__((constructor))" is just removed.
  • FindAndSetExecName(interp) is just removed.

There is a magic to replace Tcl_Init() in the TCL lib by Tcl_InitReal() and _Tcl_Init() by _Tcl_InitReal() (see Makefile.kitdll.in). Then, a new Tcl_Init() function is defined as follows:

int Tcl_InitReal(Tcl_Interp *interp);

int Tcl_Init(Tcl_Interp *interp) {
    _Tclkit_Init();
    return(Tcl_InitReal(interp));
}
  • All this is removed from "kitInit.c".

To compile, the internal C headers are required. So include:

  • c:\test\tcl8.6.10\generic
  • c:\test\tcl8.6.10\win

Encodings

Encodings live in encoding files in the tcl tree. Nevertheless, the following encodings are created internally without those files: utf-8, unicode, iso8859-1 (see tcl8.6.10/generic/tclEncoding.c TclInitEncodingSubsystem() ).

The kit initialization routine has additional encoding initialization with the define TCLKIT_CAN_SET_ENCODING. This define depends on defines HAVE_TCL_GETENCODINGNAMEFROMENVIRONMENT and HAVE_TCL_SETSYSTEMENCODING. They are set by aclocal.a and say, that the TCL library function Tcl_GetEncodingNameFromEnvironment() and Tcl_SetSystemEncoding() are available. This is the case in TCL8.6.

The implemented steps in kitInit.c are nearly identical to function TclpSetInitialEncodings() in "tcl8.6.10/win/tclWinInit.c". In addition, the TCL variable tclkit_system_encoding is set with the found system encoding. Nevertheless, the included call to Tcl_SetSystemEncoding() fails, as the c-vfs is not available jet.

Without the define TCLKIT_CAN_SET_ENCODING, my system encoding reports "iso8859-1". It should be "cp1252" and this encoding file is present.

So try to recompile with the define set: HAVE_TCL_GETENCODINGNAMEFROMENVIRONMENT;HAVE_TCL_SETSYSTEMENCODING This did not help. To make the variable tclkit_system_encoding appear, the TCL_GLOBAL_ONLY flag must be added:

So change from:

Tcl_SetVar(interp, "tclkit_system_encoding", Tcl_DStringValue(&encodingName), 0 );

to

Tcl_SetVar(interp, "tclkit_system_encoding", Tcl_DStringValue(&encodingName), TCL_GLOBAL_ONLY);

The routine in kitInit.c correctly detects cp1252, but a later "encoding system" returns "iso8859-1". An "encoding system cp1252" succeeds, so it is defined to early. Actually, "boot.tcl" is caring about this case when the encoding is still "identity". "Identity" was the fallback until TCL8.5. In TCL8.6, it is "iso8859-1".

So I changed in boot.tcl:

if {[encoding system] eq "identity"} {
    if {[info exists ::tclkit_system_encoding] && $::tclkit_system_encoding != ""} {
        catch {
            encoding system $::tclkit_system_encoding
        }
    }
}

to

if {[info exists ::tclkit_system_encoding] && $::tclkit_system_encoding != "" && [encoding system] ne $::tclkit_system_encoding } {
    catch {
        encoding system $::tclkit_system_encoding
    }
}

List of statically linked binary packages

By the build system,aclocal.m4 creates "kitInit-libs.h" with content for each contained binary package (excluding build, kitsh and common). In addition, an init function _Tclkit_GenericLib_Init() is created.

We create this file manually with empty contents:

static void _Tclkit_GenericLib_Init(void) {
}

In a later chapter, the call to TDOM init is added here

rechan

The rechan package is included. I don't know, if it is used by C-VFS. Just include the file "rechan.c".

vfs

The VFS package is included by default. I suppose, it is required for C-VFS.

Get the root branch from: [L5 ]

And add file vfs.c in folder generic to the project. Fix bug [L6 ].

Required defines: HAVE_SYS_STAT_H

Linking

Now add the following libraries to the project:

  • c:\test\tcl8610\lib\tclstub86.lib
  • c:\test\tcl8610\lib\tcl86tsgx.lib
  • c:\test\tcl8610\lib\thread2.8.5\thread285tsgx.lib
  • Netapi32.lib

test

In a 2nd project targeting a console application, I have a test file like that:

// scanlink_dll_test.cpp : Defines the entry point for the console application.
//

#pragma warning(disable : 4201 4214 4514)
#define STRICT
#ifndef UNICODE
#define UNICODE
#define _UNICODE
#endif

#include <windows.h>
#include <tchar.h>

#include <string.h>
#include <stdlib.h>
#include <stdio.h>
#include <math.h>
#include "my_dll.h"

#define BUFLEN 256

int main(int argc, char* argv[])
{
    WCHAR *pwErrorMsg;
    pwErrorMsg = my_init();
    if (pwErrorMsg != NULL) {
        wprintf(L"%s", pwErrorMsg);
        return 1;
    }
    for (;;) {
        WCHAR pwIn[BUFLEN];
        WCHAR *pwRes;
        int fError;
        printf("cmd, x to exit %% ");
        fflush(stdout);
        fgetws(pwIn, BUFLEN, stdin);
        if (wcscmp(pwIn, L"x\n") == 0) {
            my_release();
            return 0;
        }
        pwRes = my_cmd(pwIn, &fError);
        if (fError) {
            wprintf(L"Error: %s\n", pwRes);
        }
        else {
            wprintf(L"%s\n", pwRes);
        }
    }
}

Link it with the result of the dll project to find out, if something is missing. And run it ;-)

Boot.tcl script

The boot.tcl file is sourced as startup script by the system.

The file first loads the files lib/vfs/vfsUtils.tcl and lib/vfs/vfslib.tcl. So copy those files from "tclvfs/library" to "c:/test/starpack.vfs/lib/vfs".

Thread package

Copy the files "pckIndex.tcl and ttrace.tcl from the TCL installation to starpack.vfs/lib/thread2.8.5.

The load command in pckIndex.tcl is:

package ifneeded Thread 2.8.5 [list load [file join $dir thread285tsgx.lib]]

modify to:

package ifneeded Thread 2.8.5 [list load {} Thread]

and a "package require Thread" succeeds.

Own script

The user script "starpack.vfs/main.tcl" should be executed after startup.

There are two places, where this is done in kitCreator:

  • kitInit.c: The "initScript" sources main.tcl. This is executed in the function "TclKit_AppInit", which is not included, if define "TCLKIT_DLL" is true (which is the case).
  • kitCreator/kitsh/buildsrc/kitsh-0.0/zipvfs.tcl: Calls main.tcl in proc "cfs::zip::Execute".

They are both not applicable to my case.

I have no idea, how this should work. It is not so important, as I can call "eval{source $::starkit::tclkitroot/main.tcl}" manually.

By the way, the mountpoint of the c-vfs is hardcoded to "/.KITDLL_TCL". This path is contained in the variable "::starkit::tclkitroot".

tdom

TDOM 0.9.1 source is downloaded from [L7 ] and unzipped to c:\test\tdom0.9.1.

Do this modification: [L8 ].

Now, compile it with VC2015:

nmake -f Makefile.vc opts=static,staticpkg,msvcrt,nostubs,symbols TCLDIR=c:\test\tcl8610
nmake -f Makefile.vc install opts=static,staticpkg,msvcrt,nostubs,symbols TCLDIR=c:\test\tcl8610 INSTALLDIR=c:\test\tdom

Copy the following files from the installation to the virtual file system:

  • c:\test\tdom\tdom0.9.1\pkgIndex.tcl -> starpack/lib/tdom0.9.1
  • c:\test\tdom\tdom0.9.1\tdom.tcl -> starpack/lib/tdom0.9.1

Change "c-vfs/kitInit-libs.h" to:

Tcl_AppInitProc        Tdom_Init;
static void _Tclkit_GenericLib_Init(void) {
        Tcl_StaticPackage(0, "Tdom", Tdom_Init, NULL);
}

Change the following load command within "starpack/lib/tdom0.9.1/pckIndex.tcl from:"

[list load [file join $dir tdom091tsgx.lib] tdom]

to:

[list load "" tdom]

Link with "c:\test\tdom\tdom0.9.1\tdom091tsgx.lib".

argv

The variable "argv" does not exist. I added a

set argv {}

to set it in my "main.tcl" file.

glob

I have issues with the glob command within the cvfs:

% glob -directory $::starkit::tclkitroot -nocomplain -types f *.tcl
C:/.KITDLL_TCL/main.tcl C:/.KITDLL_TCL/runme.tcl
% glob -directory $::starkit::tclkitroot -nocomplain -types f run*.tcl

% glob -directory $::starkit::tclkitroot -nocomplain -types f {*.t[xc][tl]}
C:/.KITDLL_TCL/main.tcl C:/.KITDLL_TCL/runme.tcl C:/.KITDLL_TCL/runme.txt

So if there is a pattern with "abc*.txt", nothing is found, but the pattern "*.txt" works.

I have observed this already in starkits, so it is not a cvfs-only bug. To investigate...

Event loop

Calling "update" in a repeated manner executes the event loop. The method "Tcl_DoOneEvent" may also be used, but I did not try it.

64bit

HaO 2021-04-16: I have tried it today with 64 bit and TCl 8.6.11. It also works without any issues.

Note: I did not update TDOM to already released 0.9.2.

Further steps

Here are additional ideas for the project: