pkgDeps

Introduction

DDG 2020-02-25: Below is a small utility script which writes all package require calls to stdout. This is useful to check if making a release of your application which packages are required for your application. It shows recursively all packages on the terminal. I use this to create standalone files which contains all embedded modules before the actual application script using the script mk_tm.tcl at the bottom of the Wiki page Another Tcl module maker. So this helps in creating single file Tcl applications where the application is the last file and all modules files are added before.

Here is the code:

#!/usr/bin/env tclsh
# file pkgDeps.tcl

rename package package.orig

proc package {args} {
    set subcmd [lindex $args 0]
    if {$subcmd eq "require"} {
        puts stderr "package require $args"
    }
    package.orig {*}$args
}

if {[info exists argv0] && $argv0 eq [info script]} {
    if {[llength $argv] == 0} {
        puts "Usage: [info script] app.tcl ?other args?"
        exit 0
    }
    set ::argv0 [lindex $argv 0]
    set ::argv [lrange $argv 1 end]
    source $argv0 
}

Example: standalone mkdoc

Example call for the package mkdoc::mkdoc:

$ ./pkgDeps.tcl mkdoc.tcl
package require require Tcl 8.4
package require require Markdown
package require require Tcl 8.4
package require require Tcl 8.2
package require require Tcl
package require require Tcl
package require require textutil
package require require Tcl 8.2
package require require textutil::string
package require require Tcl 8.2
package require require textutil::repeat
package require require Tcl 8.2
package require require textutil::adjust
package require require Tcl 8.2
package require require textutil::repeat
package require require textutil::string
package require require textutil::split
package require require Tcl 8.2
package require require textutil::tabify
package require require Tcl 8.2
package require require textutil::repeat
package require require textutil::trim
package require require Tcl 8.2
...

Now my Makefile code to create the standalone application with all required packages in one file:

# Makefile
app:
        tclsh ../tools/mk_tm.tcl mkdoc 0.4 \
                any-script ../libs/textutil/string.tcl \
                any-script ../libs/textutil/repeat.tcl \
                any-script ../libs/textutil/adjust.tcl \
                any-script ../libs/textutil/expander.tcl \
                any-script ../libs/textutil/split.tcl \
                any-script ../libs/textutil/tabify.tcl \
                any-script ../libs/textutil/trim.tcl \
                any-script ../libs/textutil/textutil.tcl \
                any-script ../libs/markdown/markdown.tcl \
                any-script mkdoc.tcl

Using a tclkit interpreter, which does not(!) use the Tcl library on the current machine, I can check if the order of the files in the output follows the inverse calling order in package require. Means that because "Markdown" needs "textutils" the latter must be added first.

The Makefile can be further simplified as we don't have any binary extensions. So we can just use as well cat instead of the mk_tm.tcl script:

# Makefile
VERSION=0.4
mkdoc-module:
        cat ../libs/textutil/string.tcl \
                ../libs/textutil/repeat.tcl \
                ../libs/textutil/adjust.tcl \
                ../libs/textutil/expander.tcl \
                ../libs/textutil/split.tcl \
                ../libs/textutil/tabify.tcl \
                ../libs/textutil/trim.tcl \
                ../libs/textutil/textutil.tcl \
                ../libs/markdown/markdown.tcl \
                mkdoc.tcl > mkdoc-$(VERSION).tm
        echo "#!/usr/bin/env tclsh" > mkdoc-$(VERSION).app
        cat mkdoc-$(VERSION).tm >> mkdoc-$(VERSION).app

Tracing the source command

Let's extend the script a little bit further by tracing as well the source command.

Tool: pkgDeps.tcl

#!/usr/bin/env tclsh
# file pkgDeps.tcl

rename package package.orig
rename source source.orig

proc package {args} {
    set subcmd [lindex $args 0]
    if {$subcmd eq "require"} {
        puts stderr "package require $args"
    }
    package.orig {*}$args
}

proc source {args} {
    set file [lindex $args end]
    # be sure do show only source commands from not standard Tcl/Tk files
    if {[file tail $file] ne "pkgIndex.tcl"} {
        puts stderr "source $args"
    }
    uplevel 1 [list source.orig {*}$args]
}

if {[info exists argv0] && $argv0 eq [info script]} {
    if {[llength $argv] == 0} {
        puts "Usage: [info script] app.tcl ?other args?"
        exit 0
    }
    set ::argv0 [lindex $argv 0]
    set ::argv [lrange $argv 1 end]
    source $argv0 
}

Tool: autoPather.tcl

We now see as well which files are sourced. Please note that all the pkgIndex.tcl sourcings are not displayed. As said before I prefer for testing during making standalone applications the Tcl script with a tclkit interpreter. The advantage using this interpreter is, that it ignores the system available Tcl libraries as well as the TCLLIBPATH variable. This at the same time a problem in testing. I often solved this by adding temporarily an manipulation of the auto_path to my scripts and remove it later. Sometimes you might however forget this removal. Similar to the approach above however there is better solution. Let's write another small dispatcher script which manipulates the auto_path variable.

#!/usr/bin/env tclsh
# file: autoPather.tcl
if {[info exists argv0] && $argv0 eq [info script]} {
    if {[llength $argv] == 0} {
        puts "Usage: [info script] pathname app.tcl ?other args?"
        exit 0
    }
    lappend auto_path [lindex $argv 0]
    set ::argv0 [lindex $argv 1]
    set ::argv [lrange $argv 2 end]
    source $argv0 
}

This small script accepts a pathname to be added to the auto_path variable and then calls the actual application script updating the necessary variables $argv0 and $argv.

So you can call if in libs are your local library files for instance with:

 tclkit autoPather.tcl ../libs myapp.tcl arg1 arg2

That's nice already, but even better we can combine it with our script pkgDeps.tcl

 tclkit autoPather.tcl ../libs pkgDeps.tcl myapp.tcl arg1 arg2

Now we see which packages are required and which files are sourced. To see only the files which are used within ../libs we can as well filter the stderr channel.

tclkit autoPather.tcl ../libs pkgDeps.tcl myapp.tcl arg1 arg2 2>&1 >/dev/null | grep '../libs'

As my programm needs the snit package we see for instance the lines:

source ../libs/snit/snit2.tcl
source ../libs/snit/main2.tcl
source ../libs/snit/validate.tcl

Example: standalone hyperhelp

Now let's do a sample standalone application for the hyperhelp library/application.

tclkit-8.6.10-min autoPather.tcl ../libs pkgDeps.tcl hyperhelp.tcl hyperhelp-docu.txt 2>&1 >/dev/null | grep '../libs'
source ../libs/dgtools/shistory-0.2.tm
source ../libs/snit/snit2.tcl
source ../libs/snit/main2.tcl
source ../libs/snit/validate.tcl
source ../libs/dgw/dgwutils-0.2.tm

This lists our required files.

Our final application script is basically just merge of all the files listed above and finally our application script. So we must disable all source calls within the files listed above. So create our Makefile task:

app-hyperhelp:
        # so let's strip of source commands from non-pkgIndex.tcl files
        grep -vE '\s*source' ../libs/snit/snit2.tcl > snit2.tmp
        # strip comments and documentation from helperfiles
        grep -vE "^\s*#'?" ../dgtools/shistory.tcl > shistory.tmp
        grep -vE "^\s*#'?" dgwutils.tcl > dgwutils.tmp
        cat snit2.tmp \
                ../libs/snit/main2.tcl \
                ../libs/snit/validate.tcl \
                shistory.tmp \
                dgwutils.tmp \
                hyperhelp.tcl > hyperhelp-0.8.1.tm
        echo "#!/usr/bin/env tclsh" > hyperhelp-0.8.1.app
        cat hyperhelp-0.8.1.tm >> hyperhelp-0.8.1.app
        # clean up files
        -rm *.tmp

This produces a standalone file containing the snit library as well as code from the shistory.tcl and the dgwutils.tcl file. At the end our actual hyperhelp code is added:

 [bariuke dgw]$ ls -lth hyperhelp-0.8.1.app 
 -rwxr-xr-x. 1 grth grth 237K Feb 28 07:23 hyperhelp-0.8.1.app

To test we repeat our tclkit command but without the autoPather.tcl script:

 tclkit hyperhelp-0.8.1.app hyperhelp-docu.txt 

This would as well work on a machine using the standard tclsh/wish interpreter where only the standard libraries are installed. So this will as well work:

 chmod 755 hyperhelp-0.8.1.app
 cp hyperhelp-0.8.1.app ~/bin/hyperhelp
 hyperhelp hyperhelp-docu.txt 

The resulting application can be downloaded from here: https://chiselapp.com/user/dgroth/repository/tclcode/wiki?name=releases

TODO's

  • can we do compression of the app-file?
  • example with more complex packages
  • example with embedded binary files, images etc.

---

Discussion

Please discuss here:

bll - 2020-02-25: I actually have a script that traverses all of my modules, creates a dependency list, runs a topological sort on it (using 'tsort'), then rewrites each module's list of 'package require' statements in dependency order. It warns of missing dependencies and unused dependencies. It's by no means perfect, but generally works well enough for occasional use. It doesn't handle classes and instantiation very well, there's some hard-coded data in there to handle some of my classes and other oddities. It's also rather dependent on my style of fully qualifying my procedure calls. I use '::uiutils::initUI' for example, I don't use namespace imports.

DDG - 2020-02-25: Yes, my approach above certainly only is useful for small apps with a few dependencies. For larger applications your approach, wrapping into starkits, freewrap, thatch etc. might be more useful. However I felt that in many cases this simple sample script above might be a good starter for small console applications.

DDG - 2020-02-28: Extending pkgDeps.tcl with source, adding autoPather.tcl and tutorial for packing hyperhelp, replacing mk_tm.tcl with simple cat command.