[HaO] 2020-04-23: This is the log of the creation of a project to embedd a program written in TCL in a DLL.

**The Task**

I have a program written in TCL (plus some C components).
A customer wants to embedd the program in its own application and requires a DLL for a Microsoft compiler.

Additional requirements:

   *   only binary package: [tdom]
   *   interface has the following calls: init, evaluate, clear
   *   no event loop, no file events, no sockets, no threads
   *   nevertheless, leave it generic enough to be relatively universal and extendable. So include dde, registry, Thread.
   *   try to get everything in one DLL.

I am a medium level programmer and totally new to this subject.

**Credits**

I want to start by all the people helped within the journey:

   *   "Christian Gollwitzer" to give the hint of c-vfs on clt
   *   [Helmut Giese] for the proposal to use a similar approach xmktclapp.tcl (by D. Hipp)
   *   [Roy Keene] to author KitDLL and c-vfs.
   *   [Ashok Nadkarni] to help me with the linking
   *   [Jean-Claude Wippler] to author TCLKit & Friends.

**KitDLL**

KitDLL is binary distribution of a dll with an embedded tcl & bundles.

KitCreator/KitDLL contains the following options:

   * Starkits: read a file (the binary or dll) and locate a [metakit] file data base and mount it to the TCL interpreter
   * ZIPKit: same as Starkits, but using a zip archive instead the metakit
   * C-VFS: includes scripts in a C source at compile time and allows to access them on runtime.

All those options were added to the Starkit framework still containing many bug-fixes for TCL8.4/5 and support for weired platforms like Windows-CE.

The build system downloads many packages, applies provided patches, compiles them and packs all scripts in the C-VFS and statically binds all binary extensions.

I tried KitDLL binary from [https://kitcreator.rkeene.org/fossil/wiki?name=KitDLL].
When I download the binary distribution, copied the tclsh.exe beside the binary and started it, I got "<sys/stat.h> not compatible with VC".
It is cross-compiled on Linux for Windows.
I already tried MingW and those distributions and always ran into subtile bugs, like inexact calculations, wrong colors in Widgets etc. Thus, I decided to compile on my own and use a Microsoft compiler.

I am only interested in the C-VFS part and want to build on my own.

**Embedding TCL**

Embedding TCL consists of the following steps:

   * Call Tcl_FindExecutable(NULL) to set a global variable with the current executable file path. In case of a DLL, this is the path to the executable using the dll. This is very impoortant for starkits and zib-vfs, as those file systems want to open this file to find the virtual file system. This does not work for a DLL, as the path to the calling exe, and not to the dll is provided.
   * Call Tcl_CreateInterp() to create a TCL interpreter.
   * Optionally register a script to run in Tcl_Init() (source init.tcl) by TclSetPreInitScript(). This is an internal Tcl library function. This is used to setup the vfs.
   * Call '''Tcl_Main()''' and pass a custom AppInit procedure. AppInit() typically initializes static linked packages.

Internally, Tcl_Main() (in tcl8.6.10/generic/tclMain.c) will do for each created interpreter:

   * do initialization
   * call the function passed as AppInit() parameter. The function Tcl_Init() may be invoked in AppInit() to find and source an init.tcl file.
   * call Tcl command "exit" at the end of the script and end the process.

As '''Tcl_Main()''' does not return, it is not a suited procedure for this task.
Its tasks must be done manually.

For a vfs, this procedure must be done for each created interpreter, to have the vfs installed in.

After initialization, the interpreter handle may be used to invoke commands in the interpreter using Tcl_Eval*() functions.

At end of life, Tcl_DeleteInterp() is called.

**Static TCL**

To build a static TCL lib, the makefile.vc is used.
I decided to use the community edition of MS-VisualC2015.
So, first choose in the start menu: "Visual Studio 2015 -> Windows Desktop Command -> VS2015 x86 Native Tools-Eingabeaufforderung".
TCL8.6.10 source distribution is unzipped in: "C:\test\tcl8.6.10"

TCL is compiled with the following options:

   * static: to not require a sub-dll
   * staticpkg: to link dde and reg binary packages in the library
   * msvcrt: the final result is a DLL. So, link with the DLL C library, which is typically for a DLL (but unusal for a static build).
   * nostubs: any bundled package should not use stubs. It only takes time and no advantage, as they are linked together anyway.
   * symbols (only for development) - crucial for a learner to be able to trace through everything.

======
> cd test\tcl8.6.10\win
> nmake -f Makefile.vc release OPTS=static,staticpkg,msvcrt,nostubs,symbols
> nmake -f Makefile.vc install OPTS=static,staticpkg,msvcrt,nostubs,symbols INSTALLDIR=c:\test\tcl8610
======


***Load statically linked packages in interpreter***

Staticly added packages are loaded in the interpreter in a different way.
Mostly they are added to interpreter awareness by the C command on setup time:

======
Tcl_StaticPackage(interp,pkgName, initProc, safeInitProc (=NULL) )
======

and later loaded in the script by:

======
load "" pkgName
======


***Compile to link with static TCL lib***

If a custom package should later be linked with the static tcl lib, the define "STATIC_BUILD" must be defined before the inclusion of "tcl.h". This does not apply if stubs are used (define USE_TCL_STUBS).

**Tcl (Win) startup**

Here is the program flow of tclsh.exe and the concerned files:

   *   main() in tcl8.6.10/win/tclAppInit.c
   *   Tcl_Main() in tcl8.6.10/generic/tclMain.c
   *   Tcl_AppInit() in tcl8.6.10/win/tclAppInit.c

All those are sources of inspiration for the own solution.

**C-VFS**

Download: I took the trunk from [https://kitcreator.rkeene.org/fossil/wiki?name=KitDLL] dated 2020-01-22.
The folder "kitsh\buildsrc\kitsh-0.0" contains the kit programs. Change to this folder.

First, the file tree with the scripts to include is set-up in the folder "C:\test\starpack.vfs".
I added the tcl lib folder (excluding most encodings and time zones and tests) and the file "boot.tcl" from the upper "kitsh.0-0"-folder.

A C-VFS file containing this file tree is created by (see aclocal.m4):

======
tclsh dir2c.tcl tcl c:/test/starpack.vfs --obsfucate > cvfs_data_tcl.c
======

I did not use "--obfuscate" at the beginning, which does not put the scripts in clear text into the DLL.

To compile this file, I used a MSVC2015 dll project with the following additional defines:

======
HAVE_STRING_H;HAVE_STDLIB_H;CVFS_MAKE_LOADABLE,STATIC_BUILD;UNICODE;_UNICODE;TCLKIT_CAN_SET_ENCODING
======

First I tried using MS-VC6. The dir2c outputs C99 code, not C89. So this patch is applied: [https://kitcreator.rkeene.org/fossil/info/127ac40147407d70].
(ToDo as ticket site is down: The patch has a bug that struct member size and type are mixed-up (initialized in the wrong order)).
When I included TCL itself, I ran into memory issues. So I changed to MS VC2015.

Then I ran into a compiler limit (C1091), that string constants may not exceed 65535 bytes.
A solution is an unsigned char byte array, with the limit of size_t (0x7CFFFFFF).

This lead to the following patch of dir2c.tcl: [http://kitcreator.rkeene.org/fossil/tktview?name=bd6188edd4].

Then, the file "cvfs_data_tcl.c" is successfully compiled.


**DLL main**

My DLL main program with the 3 exported functions is as follows:

<<discussion>>
======
#define STRICT
#define DLL_BUILD
#include <windows.h>
#include <windowsx.h>
#include <tchar.h>
#include <string.h>
#include <stdlib.h>
#include <math.h>

#include <tcl.h>
#include "tclkit.h"

#define ERROR_MESSAGE_MAX 1024
// >>> local Prototypes
__declspec(dllexport) void my_release();
static void TclGetError();

// The pointer to the tcl interpreter
static Tcl_Interp *fg_interp = NULL;

static WCHAR fg_w_error_msg[ERROR_MESSAGE_MAX + 1];

// >>>>> DllMain
BOOL APIENTRY DllMain(HANDLE hModule,
    DWORD  ul_reason_for_call,
    LPVOID lpReserved)
{
    switch (ul_reason_for_call) {
    case DLL_PROCESS_DETACH:
        // >>> DLL is unloaded
        my_release();
    }
    return TRUE;
}
__declspec(dllexport) WCHAR * my_init()
{
    // >> Release eventual present interpreter
    if (fg_interp != NULL)
        my_release();

    // >> from tcl/generic/TclMain.c Tcl_Main()
    Tcl_FindExecutable(NULL);
    fg_interp = Tcl_CreateInterp();
    Tcl_InitMemory(fg_interp);
    Tcl_SetVar2Ex(fg_interp, "tcl_interactive", NULL, Tcl_NewIntObj(0), TCL_GLOBAL_ONLY);

    // from tcl/win/TclAppinit.c Tcl_AppInit()

    // >> from kitInit.c Tcl_Init()
    _Tclkit_Init();
    // >> Source the init.tcl script
    if (Tcl_Init(fg_interp) != TCL_OK) {
        TclGetError();
        my_release();
        return fg_w_error_msg;
    }
    return NULL;
}
void __declspec(dllexport) my_release()
{
    if (fg_interp != NULL)
        Tcl_DeleteInterp(fg_interp);
    fg_interp = NULL;
}
// >>>>> my_cmd
__declspec(dllexport) WCHAR * my_cmd(WCHAR *pwCmd, int *pfError)
{
    WCHAR * pwError;
    Tcl_DString dCmd;
    int Res;
    if (fg_interp == NULL) {
        pwError = my_init();
        if (pwError != NULL)
            *pfError = 1;
        return pwError;
    }
    if (Tcl_InterpActive(fg_interp)) {
        *pfError = 0;
        return NULL;
    }
    // >> Call command
    Tcl_DStringInit(&dCmd);
    Tcl_WinTCharToUtf(pwCmd, -1, &dCmd);
    Res = Tcl_Eval(fg_interp, Tcl_DStringValue(&dCmd));
    Tcl_DStringFree(&dCmd);
    TclGetError();
    *pfError = (Res != TCL_OK);
    return fg_w_error_msg;
}
// >>>>> TclGetError
static void TclGetError()
{
    Tcl_DString dErrorMessage;
    Tcl_DString dErrorMessageWide;
    int ErrorLength;
    Tcl_DStringInit(&dErrorMessage);
    Tcl_DStringInit(&dErrorMessageWide);
    Tcl_DStringGetResult(fg_interp, &dErrorMessage);
    Tcl_WinUtfToTChar(Tcl_DStringValue(&dErrorMessage),
        Tcl_DStringLength(&dErrorMessage), &dErrorMessageWide);
    // Limit length to size of error buffer
    ErrorLength = min(Tcl_DStringLength(&dErrorMessageWide)*2, ERROR_MESSAGE_MAX);
    wmemcpy(fg_w_error_msg, (const wchar_t *)Tcl_DStringValue(&dErrorMessageWide), ErrorLength);
    // set end zero
    fg_w_error_msg[ErrorLength] = 0;
}
======
<<enddiscussion>>

The interpreter is initialized, commands may be executed and the interpreter may be released.

**c-vfs initialization**

The c-vfs initialization happens in the file '''kitInit.c''', while the data and implementation is in the generated file '''cvfs_data_tcl.c'''.

The Tclkit initialization is announced with the prototype file "tclkit.h":

======
void _Tclkit_Init(void);
======

It is a matter of taste to rename this function and remove the leading "_" which indicates an internal function.

The initialization tcl file cvfs.tcl is prepared for embedding by:

======
tclsh stringify.tcl cvfs.tcl > cvfs.tcl.h
======

Now, the file kitInit.c is ready for inclusion.

I personally have removed from the file:

   * anything for metakit and zip file system
   * the bugfixes for "FindExecutable()".
   * anything for Tk

So, the following defines are set:

======
_WIN32;KIT_STORAGE_CVFS;CVFS_MAKE_LOADABLE;TCL_THREADS;TCLKIT_DLL
======

   * The gcc directive "__attribute__((constructor))" is just removed.
   * "FindAndSetExecName(interp)" is replaced by "Tcl_GetNameOfExecutable()"

There is a magic to replace ''Tcl_Init()'' in the TCL lib by ''Tcl_InitReal()'' and ''_Tcl_Init()'' by ''_Tcl_InitReal()'' (see Makefile.kitdll.in).
Then, a new ''Tcl_Init()'' function is defined as follows:

======
int Tcl_InitReal(Tcl_Interp *interp);

int Tcl_Init(Tcl_Interp *interp) {
    _Tclkit_Init();
    return(Tcl_InitReal(interp));
}
======

   *   All this is removed from "kitInit.c".

To compile, the internal C headers are required. So include: 

   * c:\test\tcl8.6.10\generic
   * c:\test\tcl8.6.10\win


***Encodings***

Encodings live in encoding files in the tcl tree.
Nevertheless, the following encodings are created internally without those files: utf-8, unicode, iso8859-1 (see tcl8.6.10/generic/tclEncoding.c TclInitEncodingSubsystem() ).

The kit initialization routine has additional encoding initialization with the define "TCLKIT_CAN_SET_ENCODING".
This define depends on defines "HAVE_TCL_GETENCODINGNAMEFROMENVIRONMENT" and "HAVE_TCL_SETSYSTEMENCODING".
I did not find out, where those defines are set, but probably they say, that the TCL library function "Tcl_GetEncodingNameFromEnvironment()" and "Tcl_SetSystemEncoding()" are available.

The implemented steps in kitInit.c are nearly identical to function "TclpSetInitialEncodings()" in "tcl8.6.10/win/tclWinInit.c". In addition, the TCL variable "tclkit_system_encoding" is set with the found system encoding.
Nevertheless, the included call to "Tcl_SetSystemEncoding()" fails, as the c-vfs is not available jet.

Without the define "TCLKIT_CAN_SET_ENCODING", my system encoding reports "iso8859-1". It should be "cp1252" and this encoding file is present. So try to recompile with the define set.

HAVE_TCL_GETENCODINGNAMEFROMENVIRONMENT;HAVE_TCL_SETSYSTEMENCODING

This did not help.
To make the variable "tclkit_system_encoding" appear, the TCL_GLOBAL_ONLY flag must be added:

So change from:


======
Tcl_SetVar(interp, "tclkit_system_encoding", Tcl_DStringValue(&encodingName), 0 );
======

to

======
Tcl_SetVar(interp, "tclkit_system_encoding", Tcl_DStringValue(&encodingName), TCL_GLOBAL_ONLY);
======

The routine in kitInit.c correctly detects cp1252, but a later "encoding system" returns "iso8859-1".
An "encoding system cp1252" succeeds, so it is defined to early.
Actually, "boot.tcl" is caring about this case when the encoding is still "identity".
I suppose, "identity" is replaced by "iso8859-1" as fallback, so we get this.

So I changed in boot.tcl:

======
if {[encoding system] eq "identity"} {
    if {[info exists ::tclkit_system_encoding] && $::tclkit_system_encoding != ""} {
        catch {
            encoding system $::tclkit_system_encoding
        }
    }
}
======

to

======
if {[info exists ::tclkit_system_encoding] && $::tclkit_system_encoding != "" && [encoding system] ne $::tclkit_system_encoding } {
    catch {
        encoding system $::tclkit_system_encoding
    }
}
======

**List of statically linked binary packages**

By the build system,aclocal.m4 creates "kitInit-libs.h" with content for each contained binary package (excluding build, kitsh and common). In addition, an init function "_Tclkit_GenericLib_Init" is created.

We create this file manually with empty contents:
======
static void _Tclkit_GenericLib_Init(void) {
}
======

**rechan**

The rechan package is included. I don't know, if it is used by C-VFS.
Just include the file "rechan.c".

**vfs**

The VFS package is included by default. I suppose, it is required for C-VFS.

Get the root branch from: [https://core.tcl-lang.org/tclvfs/info/995426198338747b]

And add file vfs.c in folder generic to the project.
Fix bug [https://core.tcl-lang.org/tclvfs/tktview?name=c6829e8f45].

Required defines: HAVE_SYS_STAT_H

**Linking**

Now add the following libraries to the project:

   * c:\test\tcl8610\lib\tclstub86.lib
   * c:\test\tcl8610\lib\tcl86tsgx.lib
   * c:\test\tcl8610\lib\thread2.8.5\thread285tsgx.lib
   * Netapi32.lib

**test**

In a 2nd project targeting a console application, I have a test file like that:

<<discussion>>
======
// scanlink_dll_test.cpp : Defines the entry point for the console application.
//

#pragma warning(disable : 4201 4214 4514)
#define STRICT
#ifndef UNICODE
#define UNICODE
#define _UNICODE
#endif

#include <windows.h>
#include <tchar.h>

#include <string.h>
#include <stdlib.h>
#include <stdio.h>
#include <math.h>
#include "my_dll.h"

#define BUFLEN 256

int main(int argc, char* argv[])
{
    WCHAR *pwErrorMsg;
    pwErrorMsg = my_init();
    if (pwErrorMsg != NULL) {
        wprintf(L"%s", pwErrorMsg);
        return 1;
    }
    for (;;) {
        WCHAR pwIn[BUFLEN];
        WCHAR *pwRes;
        int fError;
        printf("cmd, x to exit %% ");
        fflush(stdout);
        fgetws(pwIn, BUFLEN, stdin);
        if (wcscmp(pwIn, L"x\n") == 0) {
            my_release();
            return 0;
        }
        pwRes = my_cmd(pwIn, &fError);
        if (fError) {
            wprintf(L"Error: %s\n", pwRes);
        }
        else {
            wprintf(L"%s\n", pwRes);
        }
    }
}
======
<<enddiscussion>>

Link it with the result of the dll project to find out, if something is missing.
And run it ;-)

**Boot.tcl script**

The '''boot.tcl''' file is sourced as startup script by the system.

The file first loads the files lib/vfs/vfsUtils.tcl and lib/vfs/vfslib.tcl.
So copy those files from "tclvfs/library" to "c:/test/starpack.vfs/lib/vfs".

**Own script**

I am now (2020-05-08) at the state that I can execute commands with the test application.
Thread is not loading jet, which will be the subject of the next session.
Unfortunately, kitcreator fossil is still down, so I may not report any issues and hugs.

Read you again next week,
Harald