HaO 2020-05-29: Embed a program written in Tcl in a DLL.
A program written in Tcl. A customer wants to embedd the program in its own application and requires a DLL for a Microsoft compiler.
Additional requirements:
I am a medium level programmer and totally new to this subject.
I want to start by all the people helped within the journey:
KitDLL is binary distribution of a dll with an embedded tcl & bundles.
KitCreator/KitDLL contains the following options:
All those options were added to the Starkit framework still containing many bug-fixes for Tcl 8.4/5 and support for old platforms like Windows-CE.
The build system downloads many packages, applies provided patches, compiles them and packs all scripts in the C-VFS and statically binds all binary extensions.
I tried KitDLL binary from [L1 ]. When I download the binary distribution, copied the tclsh.exe beside the binary and started it, I got "<sys/stat.h> not compatible with VC". It is cross-compiled on Linux for Windows. I already tried MingW and those distributions and always ran into subtile bugs, like inexact calculations, wrong colors in Widgets etc. Thus, I decided to compile on my own and use a Microsoft compiler.
I am only interested in the C-VFS part and not in ZIP-VFS or MKVFS.
Embedding Tcl consists of the following steps:
Internally, Tcl_Main() (in tcl8.6.10/generic/tclMain.c) will do for each created interpreter:
As Tcl_Main() does not return, it is not a suited procedure for this task. Its tasks must be done manually.
The vfs is to be initialized for each created interpreter (and thread or slave) including search path setup etc.
After initialization, the interpreter handle may be used to invoke commands in the interpreter using Tcl_Eval*() functions.
At end of life, Tcl_DeleteInterp() is called.
To build a static Tcl lib, the makefile.vc is used. I decided to use the community edition of MS-VisualC2015. So, first choose in the start menu: "Visual Studio 2015 -> Windows Desktop Command -> VS2015 x86 Native Tools-Eingabeaufforderung". Tcl 8.6.10 source distribution is unzipped in: "C:\test\tcl8.6.10"
Tcl is compiled with the following options:
> cd test\tcl8.6.10\win > nmake -f Makefile.vc release OPTS=static,staticpkg,msvcrt,nostubs,symbols > nmake -f Makefile.vc install OPTS=static,staticpkg,msvcrt,nostubs,symbols INSTALLDIR=c:\test\tcl8610
Statically linked packages are loaded in the interpreter in a different way. First, the start address of the package is passed to the Tcl system:
Tcl_StaticPackage(interp,pkgName, initProc, safeInitProc (=NULL) )
The parameter "interp" may be NULL to register it to all interpreters.
As a second step, each interpreter executes a load command with an empty filename and the announced package name:
load {} pkgNameIf a custom package should later be linked with the static tcl lib, the define STATIC_BUILD must be defined before the inclusion of "tcl.h". This does not apply if stubs are used (define USE_TCL_STUBS).
Here is the program flow of tclsh.exe and the concerned files:
All those are sources of inspiration for the own solution.
Download: I took the trunk from [L2 ] dated 2020-01-22. The folder "kitsh\buildsrc\kitsh-0.0" contains the kit programs. Change to this folder.
First, the file tree with the scripts to include is set-up in the folder "C:\test\starpack.vfs". I added the tcl lib folder (excluding most encodings and time zones and tests) and the file "boot.tcl" from the upper "kitsh.0-0"-folder.
A C-VFS file containing this file tree is created by (see aclocal.m4):
tclsh dir2c.tcl tcl c:/test/starpack.vfs --obsfucate > cvfs_data_tcl.c
"--obfuscate" folds the data with a constant key to avoid clear text in the DLL. It works and takes considerably more time on packing.
To compile this file, I used a MSVC2015 dll project with the following additional defines:
HAVE_STRING_H;HAVE_STDLIB_H;CVFS_MAKE_LOADABLE,STATIC_BUILD;UNICODE;_UNICODE;TCLKIT_CAN_SET_ENCODING
First I tried using MS-VC6. The dir2c outputs C99 code, not C89. So this patch is applied: [L3 ]. When I included Tcl itself, I ran into memory issues. So I changed to MS VC2015.
Then I ran into a compiler limit (C1091), that string constants may not exceed 65535 bytes. A solution is an unsigned char byte array, with the limit of size_t (0x7CFFFFFF).
This lead to the following patch of dir2c.tcl: [L4 ].
Then, the file "cvfs_data_tcl.c" is successfully compiled.
My DLL main program with the 3 exported functions is as follows:
#define STRICT
#define DLL_BUILD
#include <windows.h>
#include <windowsx.h>
#include <tchar.h>
#include <string.h>
#include <stdlib.h>
#include <math.h>
#include <tcl.h>
#include "tclkit.h"
#define ERROR_MESSAGE_MAX 1024
// >>> local Prototypes
__declspec(dllexport) void my_release();
static void TclGetError();
// The pointer to the tcl interpreter
static Tcl_Interp *fg_interp = NULL;
static WCHAR fg_w_error_msg[ERROR_MESSAGE_MAX + 1];
// >>>>> DllMain
BOOL APIENTRY DllMain(HANDLE hModule,
DWORD ul_reason_for_call,
LPVOID lpReserved)
{
switch (ul_reason_for_call) {
case DLL_PROCESS_DETACH:
// >>> DLL is unloaded
my_release();
}
return TRUE;
}
__declspec(dllexport) WCHAR * my_init()
{
// >> Release eventual present interpreter
if (fg_interp != NULL)
my_release();
// >> from tcl/generic/TclMain.c Tcl_Main()
Tcl_FindExecutable(NULL);
fg_interp = Tcl_CreateInterp();
Tcl_InitMemory(fg_interp);
Tcl_SetVar2Ex(fg_interp, "tcl_interactive", NULL, Tcl_NewIntObj(0), TCL_GLOBAL_ONLY);
// from tcl/win/TclAppinit.c Tcl_AppInit()
// >> from kitInit.c Tcl_Init()
_Tclkit_Init();
// >> Source the init.tcl script
if (Tcl_Init(fg_interp) != TCL_OK) {
TclGetError();
my_release();
return fg_w_error_msg;
}
return NULL;
}
void __declspec(dllexport) my_release()
{
if (fg_interp != NULL)
Tcl_DeleteInterp(fg_interp);
fg_interp = NULL;
}
// >>>>> my_cmd
__declspec(dllexport) WCHAR * my_cmd(WCHAR *pwCmd, int *pfError)
{
WCHAR * pwError;
Tcl_DString dCmd;
int Res;
if (fg_interp == NULL) {
pwError = my_init();
if (pwError != NULL)
*pfError = 1;
return pwError;
}
if (Tcl_InterpActive(fg_interp)) {
*pfError = 0;
return NULL;
}
// >> Call command
Tcl_DStringInit(&dCmd);
Tcl_WinTCharToUtf(pwCmd, -1, &dCmd);
Res = Tcl_Eval(fg_interp, Tcl_DStringValue(&dCmd));
Tcl_DStringFree(&dCmd);
TclGetError();
*pfError = (Res != TCL_OK);
return fg_w_error_msg;
}
// >>>>> TclGetError
static void TclGetError()
{
Tcl_DString dErrorMessage;
Tcl_DString dErrorMessageWide;
int ErrorLength;
Tcl_DStringInit(&dErrorMessage);
Tcl_DStringInit(&dErrorMessageWide);
Tcl_DStringGetResult(fg_interp, &dErrorMessage);
Tcl_WinUtfToTChar(Tcl_DStringValue(&dErrorMessage),
Tcl_DStringLength(&dErrorMessage), &dErrorMessageWide);
// Limit length to size of error buffer
ErrorLength = min(Tcl_DStringLength(&dErrorMessageWide)*2, ERROR_MESSAGE_MAX);
wmemcpy(fg_w_error_msg, (const wchar_t *)Tcl_DStringValue(&dErrorMessageWide), ErrorLength);
// set end zero
fg_w_error_msg[ErrorLength] = 0;
}The interpreter is initialized, commands may be executed and the interpreter may be released.
The c-vfs initialization happens in the file kitInit.c, while the data and implementation is in the generated file cvfs_data_tcl.c.
The Tclkit initialization is announced with the prototype file "tclkit.h":
void _Tclkit_Init(void);
It is a matter of taste to rename this function and remove the leading "_" which indicates an internal function.
The initialization tcl file cvfs.tcl is prepared for embedding by:
tclsh stringify.tcl cvfs.tcl > cvfs.tcl.h
Now, the file kitInit.c is ready for compilation within the DLL project.
I personally have removed from the file:
So, the following defines are set:
_WIN32;KIT_STORAGE_CVFS;CVFS_MAKE_LOADABLE;TCL_THREADS;TCLKIT_DLL
There is a magic to replace Tcl_Init() in the TCL lib by Tcl_InitReal() and _Tcl_Init() by _Tcl_InitReal() (see Makefile.kitdll.in). Then, a new Tcl_Init() function is defined as follows:
int Tcl_InitReal(Tcl_Interp *interp);
int Tcl_Init(Tcl_Interp *interp) {
_Tclkit_Init();
return(Tcl_InitReal(interp));
}To compile, the internal C headers are required. So include:
Encodings live in encoding files in the tcl tree. Nevertheless, the following encodings are created internally without those files: utf-8, unicode, iso8859-1 (see tcl8.6.10/generic/tclEncoding.c TclInitEncodingSubsystem() ).
The kit initialization routine has additional encoding initialization with the define TCLKIT_CAN_SET_ENCODING. This define depends on defines HAVE_TCL_GETENCODINGNAMEFROMENVIRONMENT and HAVE_TCL_SETSYSTEMENCODING. They are set by aclocal.a and say, that the TCL library function Tcl_GetEncodingNameFromEnvironment() and Tcl_SetSystemEncoding() are available. This is the case in TCL8.6.
The implemented steps in kitInit.c are nearly identical to function TclpSetInitialEncodings() in "tcl8.6.10/win/tclWinInit.c". In addition, the TCL variable tclkit_system_encoding is set with the found system encoding. Nevertheless, the included call to Tcl_SetSystemEncoding() fails, as the c-vfs is not available jet.
Without the define TCLKIT_CAN_SET_ENCODING, my system encoding reports "iso8859-1". It should be "cp1252" and this encoding file is present.
So try to recompile with the define set: HAVE_TCL_GETENCODINGNAMEFROMENVIRONMENT;HAVE_TCL_SETSYSTEMENCODING This did not help. To make the variable tclkit_system_encoding appear, the TCL_GLOBAL_ONLY flag must be added:
So change from:
Tcl_SetVar(interp, "tclkit_system_encoding", Tcl_DStringValue(&encodingName), 0 );
to
Tcl_SetVar(interp, "tclkit_system_encoding", Tcl_DStringValue(&encodingName), TCL_GLOBAL_ONLY);
The routine in kitInit.c correctly detects cp1252, but a later "encoding system" returns "iso8859-1". An "encoding system cp1252" succeeds, so it is defined to early. Actually, "boot.tcl" is caring about this case when the encoding is still "identity". "Identity" was the fallback until TCL8.5. In TCL8.6, it is "iso8859-1".
So I changed in boot.tcl:
if {[encoding system] eq "identity"} {
if {[info exists ::tclkit_system_encoding] && $::tclkit_system_encoding != ""} {
catch {
encoding system $::tclkit_system_encoding
}
}
}to
if {[info exists ::tclkit_system_encoding] && $::tclkit_system_encoding != "" && [encoding system] ne $::tclkit_system_encoding } {
catch {
encoding system $::tclkit_system_encoding
}
}By the build system,aclocal.m4 creates "kitInit-libs.h" with content for each contained binary package (excluding build, kitsh and common). In addition, an init function _Tclkit_GenericLib_Init() is created.
We create this file manually with empty contents:
static void _Tclkit_GenericLib_Init(void) {
}In a later chapter, the call to TDOM init is added here
The rechan package is included. I don't know, if it is used by C-VFS. Just include the file "rechan.c".
The VFS package is included by default. I suppose, it is required for C-VFS.
Get the root branch from: [L5 ]
And add file vfs.c in folder generic to the project. Fix bug [L6 ].
Required defines: HAVE_SYS_STAT_H
Now add the following libraries to the project:
In a 2nd project targeting a console application, I have a test file like that:
// scanlink_dll_test.cpp : Defines the entry point for the console application.
//
#pragma warning(disable : 4201 4214 4514)
#define STRICT
#ifndef UNICODE
#define UNICODE
#define _UNICODE
#endif
#include <windows.h>
#include <tchar.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>
#include <math.h>
#include "my_dll.h"
#define BUFLEN 256
int main(int argc, char* argv[])
{
WCHAR *pwErrorMsg;
pwErrorMsg = my_init();
if (pwErrorMsg != NULL) {
wprintf(L"%s", pwErrorMsg);
return 1;
}
for (;;) {
WCHAR pwIn[BUFLEN];
WCHAR *pwRes;
int fError;
printf("cmd, x to exit %% ");
fflush(stdout);
fgetws(pwIn, BUFLEN, stdin);
if (wcscmp(pwIn, L"x\n") == 0) {
my_release();
return 0;
}
pwRes = my_cmd(pwIn, &fError);
if (fError) {
wprintf(L"Error: %s\n", pwRes);
}
else {
wprintf(L"%s\n", pwRes);
}
}
}Link it with the result of the dll project to find out, if something is missing. And run it ;-)
The boot.tcl file is sourced as startup script by the system.
The file first loads the files lib/vfs/vfsUtils.tcl and lib/vfs/vfslib.tcl. So copy those files from "tclvfs/library" to "c:/test/starpack.vfs/lib/vfs".
Copy the files "pckIndex.tcl and ttrace.tcl from the TCL installation to starpack.vfs/lib/thread2.8.5.
The load command in pckIndex.tcl is:
package ifneeded Thread 2.8.5 [list load [file join $dir thread285tsgx.lib]]
modify to:
package ifneeded Thread 2.8.5 [list load {} Thread]and a "package require Thread" succeeds.
The user script "starpack.vfs/main.tcl" should be executed after startup.
There are two places, where this is done in kitCreator:
They are both not applicable to my case.
I have no idea, how this should work. It is not so important, as I can call "eval{source $::starkit::tclkitroot/main.tcl}" manually.
By the way, the mountpoint of the c-vfs is hardcoded to "/.KITDLL_TCL". This path is contained in the variable "::starkit::tclkitroot".
TDOM 0.9.1 source is downloaded from [L7 ] and unzipped to c:\test\tdom0.9.1.
Do this modification: [L8 ].
Now, compile it with VC2015:
nmake -f Makefile.vc opts=static,staticpkg,msvcrt,nostubs,symbols TCLDIR=c:\test\tcl8610 nmake -f Makefile.vc install opts=static,staticpkg,msvcrt,nostubs,symbols TCLDIR=c:\test\tcl8610 INSTALLDIR=c:\test\tdom
Copy the following files from the installation to the virtual file system:
Change "c-vfs/kitInit-libs.h" to:
Tcl_AppInitProc Tdom_Init;
static void _Tclkit_GenericLib_Init(void) {
Tcl_StaticPackage(0, "Tdom", Tdom_Init, NULL);
}Change the following load command within "starpack/lib/tdom0.9.1/pckIndex.tcl from:"
[list load [file join $dir tdom091tsgx.lib] tdom]
to:
[list load "" tdom]
Link with "c:\test\tdom\tdom0.9.1\tdom091tsgx.lib".
The variable "argv" does not exist. I added a
set argv {}to set it in my "main.tcl" file.
I have issues with the glob command within the cvfs:
% glob -directory $::starkit::tclkitroot -nocomplain -types f *.tcl
C:/.KITDLL_TCL/main.tcl C:/.KITDLL_TCL/runme.tcl
% glob -directory $::starkit::tclkitroot -nocomplain -types f run*.tcl
% glob -directory $::starkit::tclkitroot -nocomplain -types f {*.t[xc][tl]}
C:/.KITDLL_TCL/main.tcl C:/.KITDLL_TCL/runme.tcl C:/.KITDLL_TCL/runme.txtSo if there is a pattern with "abc*.txt", nothing is found, but the pattern "*.txt" works.
I have observed this already in starkits, so it is not a cvfs-only bug. To investigate...
Calling "update" in a repeated manner executes the event loop. The method "Tcl_DoOneEvent" may also be used, but I did not try it.
HaO 2021-04-16: I have tried it today with 64 bit and TCl 8.6.11. It also works without any issues.
Note: I did not update TDOM to already released 0.9.2.
Here are additional ideas for the project: