[HaO] 2020-04-23: This is the log of the creation of a project to embedd a program written in TCL in a DLL. **Work in progress** **My Task** I have a program written in TCL (plus some C components). A customer wants to embedd the program in its own application and requires a DLL for a Microsoft compiler. Additional requirements: * only binary package: [tdom] * interface has the following calls: init, evaluate, clear * no event loop, no file events, no sockets, no threads * nevertheless, leave it generic enough to be relatively universal and extendable. So include dde, registry, Thread. * try to get everything in one DLL. I am a medium level programmer and totally new to this subject. **Credits** I want to start by all the people helped within the journey: * [auriocus] to give the hint of c-vfs on clt * [Helmut Giese] for the proposal to use a similar approach xmktclapp.tcl (by D. Hipp) * [Roy Keene] to author KitDLL and c-vfs. * [Ashok Nadkarni] to help me with the linking * [Jean Claude Wippler] to author TCLKit & Friends. **KitDLL** KitDLL is binary distribution of a dll with an embedded tcl & bundles I tried KitDLL binary from [https://kitcreator.rkeene.org/fossil/wiki?name=KitDLL]. When I download the binary distribution, copied the tclsh.exe beside the binary and started it, I got " not compatible with VC". It is cross-compiled on Linux for Windows. I already tried MingW and those distributions and always ran into subtile bugs, like inexact calculations, wrong colors in Widgets etc. Thus, I decided to compile on my own and use a Microsoft compiler. **Static TCL** To build a static TCL lib, the makefile.vc is used. I decided to use the community edition of MS-VisualC2015. So, first choose in the start menu: "Visual Studio 2015 -> Windows Desktop Command -> VS2015 x86 Native Tools-Eingabeaufforderung". TCL8.6.10 source distribution is unzipped in: "C:\test\tcl8.6.10" TCL is compiled with the following options: * static: to not require a sub-dll * staticpkg: to link dde and reg binary packages in the library * msvcrt: the final result is a DLL. So, link with the DLL C library, which is typically for a DLL (but unusal for a static build). * nostubs: any bundled package should not use stubs. It only takes time and no advantage, as they are linked together anyway. * symbols (only for development) - crucial for a learner to be able to trace through everything. ====== > cd test\tcl8.6.10\win > nmake -f Makefile.vc release OPTS=static,staticpkg,msvcrt,nostubs,symbols > nmake -f Makefile.vc install OPTS=static,staticpkg,msvcrt,nostubs,symbols INSTALLDIR=c:\test\tcl8610 ====== Staticly added packages are loaded in the interpreter in a different way. Mostly they are added to interpreter awareness by the C command on setup time: ====== Tcl_StaticPackage(interp,pkgName, initProc, safeInitProc (=NULL) ) ====== and later loaded in the script by: ====== load "" pkgName ====== **C-VFS** C-VFS includes scripts in a C source and allows to access them on runtime. But KitDLL contains more options: * Starkits: read a file (the binary or dll) and locate a [metakit] file data base and mount it to the TCL interpreter * ZIPKit: same as Starkits, but using a zip archive instead the metakit * C-VFS: as described above All those options were added to the Starkit framework still containing many bug-fixes for TCL8.4/5 and support for weired platforms like Windows-CE. I am only interested in the C-VFS part and want to build on my own. The original build system downloads many packages, applies provided patches, compiles them and packs all scripts in the C-VFS and statically binds all binary extensions. This is far to complex for me. Download: I took the trunk from [https://kitcreator.rkeene.org/fossil/wiki?name=KitDLL] dated 2020-01-22. The folder "kitsh\buildsrc\kitsh-0.0" contains the kit programs. Change to this folder. First, the file tree with the scripts to include is set-up in the folder "C:\test\starpack.vfs". I added the tcl lib folder (excluding most encodings and time zones and tests) and the file "boot.tcl" from the upper "kitsh.0-0"-folder. A C-VFS file containing this file tree is created by (see aclocal.m4): ====== tclsh dir2c.tcl tcl c:/test/starpack.vfs --obsfucate > cvfs_data_tcl.c ====== I did not use "--obfuscate" at the beginning, which does not put the scripts in clear text into the DLL. To compile this file, I used a MSVC2015 dll project with the following additional defines: ====== HAVE_STRING_H;HAVE_STDLIB_H;CVFS_MAKE_LOADABLE,STATIC_BUILD;UNICODE;_UNICODE ====== First I tried using MS-VC6. The dir2c outputs C99 code, not C89. So this patch is applied: [https://kitcreator.rkeene.org/fossil/info/127ac40147407d70]. When I included TCL itself, I ran into memory issues. So I changed to MS VC2015. Then I ran into a compiler limit (C1091), that string constants may not exceed 65535 bytes. A solution is an unsigned char byte array, with the limit of size_t (0x7CFFFFFF). This lead to the following patch of dir2c.tcl: [http://kitcreator.rkeene.org/fossil/tktview?name=bd6188edd4]. Then, the file "cvfs_data_tcl.c" is successfully compiled. **DLL main** My DLL main program with the 3 exported functions is as follows: ====== #define STRICT #define DLL_BUILD #include #include #include #include #include #include #include #include "tclkit.h" #define RET_ERR_STRING -1 #define RET_WIN_ERROR -2 #define ERROR_MESSAGE_MAX 1024 // >>> local Prototypes __declspec(dllexport) void my_release(); static void TclGetError(); // The pointer to the tcl interpreter static Tcl_Interp *fg_interp=NULL; static WCHAR fg_w_error_msg[ERROR_MESSAGE_MAX+1]; // >>>>> DllMain BOOL APIENTRY DllMain( HANDLE hModule, DWORD ul_reason_for_call, LPVOID lpReserved) { switch (ul_reason_for_call) { case DLL_PROCESS_DETACH: // >>> DLL is unloaded my_release(); } return TRUE; } __declspec(dllexport) WCHAR * my_init() { // >> Release eventual present interpreter if (fg_interp != NULL) scanlink_release(); Tcl_FindExecutable(NULL); fg_interp = Tcl_CreateInterp(); // >> Init Tcl Kit Tclkit_Init(); // >> Source the init.tcl script if (Tcl_Init(fg_interp) != TCL_OK) { TclGetError(); return fg_w_error_msg; } return NULL; } // >>>>> scanlink_release void __declspec(dllexport) scanlink_release() { if ( fg_interp != NULL) Tcl_DeleteInterp(fg_interp); fg_interp = NULL; } // >>>>> my_cmd __declspec(dllexport) WCHAR * my_cmd(WCHAR *pwCMD) { WCHAR * pwError; Tcl_DString dCmd; int Res; if (fg_interp == NULL) { pwError = my_init(); if (pwError != NULL) return pwError; } if (Tcl_InterpActive(fg_interp)) { return 0; } // >> Call initialization routine Tcl_DStringInit(&dCmd); Tcl_WinTCharToUtf(pwCmd,-1,&dCmd); oArguments[1] = Tcl_NewStringObj(Tcl_DStringValue(&dProfileName), Tcl_DStringLength(&dProfileName)); Tcl_DStringFree(&dProfileName); // > Call directly without compilation Res = Tcl_Eval(fg_interp,dCmd); if (Res != TCL_OK) { TclGetError(); return fg_w_error_msg; } return NULL; } // >>>>> TclGetError static void TclGetError() { Tcl_DString dErrorMessage; Tcl_DString dErrorMessageWide; int ErrorLength; Tcl_DStringInit(&dErrorMessage); Tcl_DStringInit(&dErrorMessageWide); Tcl_DStringGetResult(fg_interp, &dErrorMessage); Tcl_WinUtfToTChar(Tcl_DStringValue(&dErrorMessage), Tcl_DStringLength(&dErrorMessage), &dErrorMessageWide); // Limit length to size of error buffer ErrorLength = min(Tcl_DStringLength(&dErrorMessageWide),ERROR_MESSAGE_MAX*2); memcpy(fg_w_error_msg,Tcl_DStringValue(&dErrorMessageWide),ErrorLength); // Be sure we have a wide end 0 somewhere. fg_w_error_msg[ERROR_MESSAGE_MAX] = 0; // >> If there is nothing in the interpreter, put an E in. if (fg_w_error_msg[0] == 0) { fg_w_error_msg[0] = 'E'; fg_w_error_msg[1] = 0; } } ====== The interpreter is initialized, commands may be executed and the interpreter may be released. **c-vfs initialization** I have no idea how it works, but the C file tree is made available to the script. The Tclkit initialization is announced with the prototype file "tclkit.h": ====== void Tclkit_Init(void); ====== The initialization tcl file cvfs.tcl is prepared for embedding by: ====== tclsh stringify.tcl cvfs.tcl > cvfs.tcl.h ====== Now, the file kitint.c is ready for inclusion. I personally have removed from the file: * anything for metakit and zip file system * the bugfixes for "FindExecutable()". * anything for Tk So, the following defines are set: ====== _WIN32;KIT_STORAGE_CVFS;TCL_THREADS;TCLKIT_DLL ====== * The gcc directive "__attribute__((constructor))" is just removed. * "FindAndSetExecName(interp)" is replaced by "Tcl_GetNameOfExecutable()" * Tcl_Init() is not replaced by own function, we call this manually anyway. To compile, the internal C headers are required. So include: * c:\test\tcl8.6.10\generic * c:\test\tcl8.6.10\win **List of statically linked binary packages** By the build system,aclocal.m4 creates "kitInit-libs.h" with content for each contained binary package (excluding build, kitsh and common). In addition, an init function "_Tclkit_GenericLib_Init" is created. We create this file manually with empty contents: ====== static void _Tclkit_GenericLib_Init(void) { } ====== **rechan** The rechan package is included. I don't know, if it is used by C-VFS. Just include the file "rechan.c". **vfs** The VFS package is included by default. I suppose, it is required for C-VFS. Get the root branch from: [https://core.tcl-lang.org/tclvfs/info/995426198338747b] And add file vfs.c in folder generic to the project. Fix bug [https://core.tcl-lang.org/tclvfs/tktview?name=c6829e8f45]. Required defines: HAVE_SYS_STAT_H **Linking** Now add the following libraries to the project: * c:\test\tcl8610\lib\tclstub86.lib * c:\test\tcl8610\lib\tcl86tsgx.lib * c:\test\tcl8610\lib\thread2.8.5\thread285tsgx.lib * Netapi32.lib **test** In a 2nd project targeting a console application, I have a test file like that: ====== int main(int argc, char* argv[]) { WCHAR *pwErrorMsg; pwErrorMsg = scanlink_init(); if (pwErrorMsg != NULL) { wprintf(L"%s",pwErrorMsg); return 1; } return 0; } ====== Link it with the result of the dll project to find out, if something is missing. And run it ;-) **Boot.tcl script** ToDo **Own script** ToDo