[FF] 2007-30-05 - The discussion evolved in [ToW Tcl on WebFreeWay] has generate some interest, and here is an effort for making a '''minimal''' template engine. I would like to use this page for collecting ideas from people. Fell free to change everything, fix bugs and revolutionize the concept - it is the purpose of the page. [FF] 2007-06-02 - I rewrote the engine in a OO-like way (still haven't used any OO system, neither build my OO system) (as a rule, methods beginning with _ are meant to be ''private''), improving some features. Very important is that from now it could run in a recursive way, allowing to design modular/nested templates. (note that since the rewrite I changed all the examples) '''Description:''' Really, the engine works by converting a template (which is a string, or a file) into a Tcl script, and then running it. Each line of text encountered will be returned as is. Exception is text between '''<%''' ... '''%>''' which is treated as Tcl code (the eval happens into a '''safe interp''', see [Safe Interps]). This make it very flexible and it leaves you freedom of doing what you want, dealing with any data and putting how much code you want in the template (although the template should possibly hold minimum code - according to [MVC], just the presentation logic, no code related to controller logic, state persistance, or data-retrieval logic). Also for convenience there is a '''<%=''' ... '''%>''' for spooling output (the proc _defaultSpool takes care of sending the text to the right place; you can change that: see '''Preprocessor Commands'''). '''Usage:''' first, let's create a template file (templtest.tmpl): <% for {set i 0} {$i < 4} {incr i} { %> <% } %>
<%= $i %>
the above template (in HTML language) generates a 4-row table, and an unordered list, taking values from a ''$cityList'' variable, that we're going to define. First let's load the code: source TemplaTcl now create a parser instance: TemplaTcl::create tt and parse our template: tt parseFile templtest.tmpl set the variables used in our template: tt setVar cityList {Ragusa Ravenna Rieti Rimini Rome Rovigo} and render the template: puts [tt render] Here's the output that gets produced:
1
2
3
4
'''Preprocessor commands:''' Running into a safe interp you cannot source your Tcl function library. There's a special markup command: '''<%@''' ... '''%>''' which allows you to do special things, i.e. setting parser/interp options. That happens before the parser, and before running the interpreter. To source a file you will do: <%@ source = poo.tcl %> Instead, for '''including''' a template: <%@ include = template.tmpl %> Also for changing the default spooler (mentioned above) do: <%@ printCommand = puts -nonewline %> (that replaces the _defaultSpool spooler, which allows you to get the content as the return value of TemplaTcl::render. ---- #!/usr/bin/env tclsh package require Tcl 8.5 # 8.5 required cause the {*} in proc create and proc method package require struct # tcllib required namespace eval ::TemplaTcl { variable obj proc method {name args body} { proc $name [list self {*}$args] "variable obj ; $body" } method create {} { # create and setup a safe interpreter # for running template's tcl code catch {if [interp exists $obj($self:interp)] { interp delete $obj($self:interp)}} set obj($self:interp) [interp create -safe] interp share {} stdout $obj($self:interp) interp eval $obj($self:interp) { proc _defaultSpool {txt {cmd {}}} { global content if {$cmd == "clear"} {set content {}; return} if {$cmd == "get"} {return $content} append content $txt } } uplevel "proc $self {method args} {namespace eval ::TemplaTcl \[list \$method $self {*}\$args\]}" return $self } method parseFile {filename} { # read the template into $rawl - list of chars set fh [open $filename r] set raw [read $fh] close $fh return [$self parse $raw] } method parse {template} { $self _setMode raw #$self setOption printCommand "puts -nonewline" $self setOption printCommand "_defaultSpool" $self setVar * {} set q [::struct::queue] set rawl [split $template {}] foreach ch $rawl { # we work char-by-char :| $q put $ch # max block to compare (<%=) is 3 chars long: if {[$q size] >= 3} { set s3 [join [$q peek 3] {}] set s2 [join [$q peek 2] {}] switch $obj($self:mode) { raw { if {$s3 == "<%="} { # <%= is a shorthand for puts ... $q get 3; $self _setMode code; append obj($self:buf:$obj($self:mode)) "$obj($self:options:printCommand) " continue } elseif {$s3 == "<%@"} { # <%@ is for setting preprocessor options $q get 3; $self _setMode opt; continue } elseif {$s2 == "<%"} { # <% indicates begin of a code block $q get 2; $self _setMode code; continue } } code { if {$s2 == "%>"} { # and %> is the end of code block $q get 2; $self _setMode raw; continue } } opt { if {$s2 == "%>"} { # option parser $q get 2; $self _parseOptions $obj($self:buf:opt) set obj($self:buf:opt) {} $self _setMode raw; continue } } } append obj($self:buf:$obj($self:mode)) [$q get] } } # finish processing the queue: while {[$q size] > 0} { append obj($self:buf:$obj($self:mode)) [$q get] } $self _setMode flush # cleanup: foreach v {buf:code buf:opt buf:raw mode modeprev} {catch {unset obj($self:$v)}} } method render {} { # run the template script set tclBuf "" foreach l $obj($self:data) { set t [lindex $l 0] set d [lindex $l 1] switch $t { raw {append tclBuf "$obj($self:options:printCommand) [list $d]\n"} code {append tclBuf "$d\n"} } } foreach {var val} $obj($self:variables) {$obj($self:interp) eval [list set $var $val]} #puts $tclBuf;return if {$obj($self:options:printCommand) == "_defaultSpool"} { $obj($self:interp) eval {_defaultSpool {} clear} } $obj($self:interp) eval $tclBuf if {$obj($self:options:printCommand) == "_defaultSpool"} { set x [$obj($self:interp) eval {_defaultSpool {} get}] $obj($self:interp) eval {_defaultSpool {} clear} return $x } } method setOption {opt value} { switch $opt { printCommand {set obj($self:options:$opt) $value} include { set o inc$value create $o $o parseFile $value set prevMode [$self _setMode raw] append obj($self:buf:raw) [$o render] $self _setMode $prevMode } source {$obj($self:interp) invokehidden source $value} default {return -code error -errorinfo "Unknown option: $opt"} } } method setVar {var value} { if {$var == "*" && $value == ""} {set obj($self:variables) {}; return} lappend obj($self:variables) $var lappend obj($self:variables) $value } method _setMode {m} { set modenew {} switch $m {code - raw - opt {set modenew $m}} if {$modenew != {}} { if [catch {set obj($self:mode)}] {set obj($self:mode) $modenew} set obj($self:modeprev) $obj($self:mode) set obj($self:mode) $modenew set obj($self:buf:$obj($self:mode)) {} if {$obj($self:modeprev) == {}} { set obj($self:modeprev) $obj($self:mode) } } if {$m == "flush"} { set obj($self:modeprev) $obj($self:mode) set obj($self:mode) _ } if {$obj($self:mode) != $obj($self:modeprev)} { lappend obj($self:data) [list $obj($self:modeprev) $obj($self:buf:$obj($self:modeprev))] set obj($self:buf:$obj($self:modeprev)) {} return $obj($self:modeprev) } } method _parseOptions {o} { set optlist [split $o "\n;"] foreach opt $optlist { set pair [split $opt =] set opt_ [string trim [lindex $pair 0]] if {$opt_ == {}} continue $self setOption $opt_ [string trim [lindex $pair 1]] } } } ---- ''[escargo]'' - I think it's a mistake to have parse read a file; it would be potentially more flexible to use if it just took a string to process as input, and let the caller determine where the string comes from. ''[FF]'' - good point! (changed) [APW] Just found textutil::expander[http://tcllib.sourceforge.net/doc/expander.html] package in [tcllib], which is included in ActiveTcl 8.4.14.0. Have the feeling it does something similar, maybe it's worth looking at it. [FF] but is not much versatile. Just tried this: % package require textutil::expander % ::textutil::expander myexp % set a 64 % myexp expand {[if {$a > 50} {] put some text [} {] put some other text [}]} Error in macro: [if {$a > 50} {] put some text [} {] put some other text [}] --> missing close-bracket and I realized I feel well with TemplaTcl O:) [APW] I think if you look closer to the manual you will see that you have to set the left and right bracket command i.e. "<%" and "%>". Next the token directly following the left bracket is interpreted as the command for handling the contents, so if you have "<%= hello %>" you will need a proc with name "=" for handling the code inside the macro. % package require textutil::expander % ::textutil::expander te % te lb "<%" % te rb "%> % proc = {args} { return $args } % te expand {this is a text which says <%= hello %> to you} this is a text which says hello to you Additionally for constructs like a "if" clause it is possible to push and pop the context and at the end to evaluate the code in the macro, normally you will need some tokens for marking begin and end of macro, means a construct like: <% code if { ...} { %> # some other code here <% /code %> proc code {args} { te cpush .... # some code here } proc /code {args} { set str [te cpop ...] # evaluate the code of the macro and set result return result } You can also define the number of passes to run for the same text to be able to do nested expansion if you use the expand.tcl directly from the author: [William Duquette] [http://www.wjduquette.com/expand] ---- ''[escargo]'' - I think you want to give some thought into what scope the template code executes at. Right now it seems to be run in proc dump. Two likely alternatives are to run it in the caller's scope or at global scope. ''[FF]'' running in the caller scope might seem a convenient idea, but can potentially generate trouble, since the template is a tcl script that would run just between '''TemplaTcl::dump''' and the next command, potentially altering/damaging the caller's scope (if there are subsequent commands after the TemplaTcl::dump command); in order to make it more strict, I make it run in global scope (hoping that global is not polluted or has variable that can make the application unstable); but ''ideally'' it should run in a separate interp, copying the needed vars into that interpreter. Don't you think so? ''[escargo]'' - Running in a safe interp would be the right thing, but the question might be what data can the code executing as part of template processing access (or access easily). One could reasonable argue that it should only access values provided by a supplied data object. I could see wanting to supply maybe an environment data object as well (for access to info like date and time, host name, user name, and other info that would not be part of the data coming out of a table to be published). ''[FF]'' added safe interp, setVar command, and also a new markup command for special use (see above). ''[escargo] 1 Jun 2007'' - See also [subst]. In some respects, subst is a better model than eval, since it assumes text with variable references and commands embedded in it, instead of valid programs with text embedded in the commands. The ability to intercept variable references and command executions that you don't know ahead of time would simplify some things that you want to do, but those aren't available yet. (Maybe what's needed is a subst that has been extended to take a -variables and/or -commands arguments, which are the names of commands to call when there are variable references or command references respectively.) ''[FF]'' 2007-06-02 - changed much things (almost '''all'''). I implemented a pseudo OO system. This allows running recursive/nested templates. If you were interested in this project, maybe you might be interested in re-reading this page from the beginning. [CMcC] 2Jun07 just doesn't get it - why create a language interpreter to substitute variables and function calls when tcl has a full-function command to do that, in [subst]? What does the new [litle language] for templating buy you? [APW] Would that also work for the following example? I don't think so! <%if {$a != $b} { %>

abc

<%if {$a != 1} {%>
  • <%= $var1 %>
  • <%} else {%>
  • <%= $var2 %>
  • <% } %> <% } else { %>
  • <% myobj getResult %>
  • Hello there
    <% } %> [CMcC] no, this would work better: expr { $a != $b ? "[

    abc][
  • [expr {$a != 1 ? $var1: $var2}]]" : "[
  • [myobj getResult]] [ [ "Hello there"]]" } This would also work: if {$a != $b} { subst {

    abc

    [expr {$a != 1 ? "
  • $var1
  • " : "
  • $var2
  • "}] } } else { subst {
  • [myobj getResult]
  • Hello there
    } } (note: [if] returns a value!) But nothing would work as well as refactoring it so it wasn't quite as messy in the first place. [APW] from the discussion on [ToW Tcl on WebFreeWay] I have the feeling that people would like to have a template which looks like a HTML page (maybe also XML), so the could see what goes to the client and to have a little bit (or some more) code spread in to drive some output. That's also what Ruby on Rails (short RoR) does. So having Tcl code to produce HTML is not what is wanted in that case (my interpretation). Also for a Tcl programmer that might be more readable I agree. But I think the audience should be more peaple which do not know much about Tcl but are able tho enter some variables and simple constructs in Tcl to get input from a data source for example a database. [CMcC] This is getting silly. Here's a fragment of the stuff that generates this very page you're reading - it's done using [subst]: # page template for standard page decoration variable pageT {title: $name [div container { [div header {

    $Title

    }] [div {wrapper content} {

    $C

    }]
    [div footer "

    [join $menu { - }]

    "] }] } If you ''really'' wanted to, you could replace the [[div]] calls with straight
    ...
    text. If you feel that [subst] can't be used on text which resembles an HTML page, you really should have a careful look at what [subst] does. [APW] That works if you have defined a proc named div when using [subst], but if you want to parse a XML template where you don't know all the tags and you only want to handle specially the part included in <% ... %> and handle all the rest as a text/string? Could you give me an example for that using [subst]? my company <% if {[string first $mycompany "DE"]} { %> <%= [getmycompany DE] %> my department <% } else { %> <%= [getmycompany OTHERS] %> <% } %> This is just a very simple example, which could be built differently to produce the same output, I know. How would the template file look like for being able to use [subst]? The output shall be again XML being that conversion the first pass of multiple passes, conversion to HTML being done in a later pass. [CMcC] Since you asked for [subst] usage: [subst { my company [expr {[string first $mycompany "DE"] ? "[getmycompany DE] my department " : "[getmycompany OTHERS]" }] }] But I think I'd do this instead in that case: " my company [expr {[string first $mycompany "DE"] ? "[getmycompany DE] my department " : "[getmycompany OTHERS]" }]" Or, by preference, a bit of refactoring: set cclass [expr {[string first $mycompany "DE"]?DE:OTHERS}] "[ {my company}] [ [list class $cclass] {[getmycompany $cclass]}] [if {$cclass eq "DE"} { {my department}}]" [APW] that's ok, but it does'nt look like an XML page template any more, and that was what some people including me would like to have! The last example you have can also be coded using Tcl code without any [subst] (and then it might be even more readable :-)) ), but that was not the intention. [CMcC] Ok, so using subst-if: [subst { my company [If {[string first $mycompany "DE"]} { [getmycompany DE] my department } else { [getmycompany OTHERS] }] }] will now work. Does that look enough like an XML template for the purpose? [APW] Meanwhile [jcw] has suggested an interesting solution for what I would like to have see below. ---- ''[escargo]'' - [subst] does not have a notion of iteration to expand things inside the string being operated on, for one thing. [CMcC] tcl has the concept of iteration, though. Use [map], or it's easy to write a function which iterates (think [foreach] which returns its iterated results). For example, the thing above to turn a list into an HTML list could be simply represented as
    • [[join $list
    • ]]
    , which is cleaner (I think) ... or even define a proc (such as is already in the tcllib html package) to perform the wrapping. The page [HTML generator] provides some light-weight facilities along those lines. [FF] the above mentioned systems sure are good for some things. But my target is a very specific case, where those systems maybe lack some things. TemplaTcl works with templates rich of static text, but also rich of interruptions due to code or putting variables (the typical case is generating markup for a web page). Dealing with that using other systems can possibly result being counter-productive (the TemplaTcl examples found in this page are too simple to demonstrate that). [CMcC] do you mean that you can stop the template expansion midway? [subst] can also do that. [FF] Even if right now it does its job nicely - just now I added template inclusion (<%@include = ..%>), and tcl source-ing (<%@source = ...%>) - I'll keep adding missing features to satisfy the case of code generation, with a particular regard to X(HT)ML page templating, so don't expect it to be just that tiny set of features. [CMcC] What I'm interested in is examples of some useful things TemplateTcl does better than Tcl, because that's a sign of something which could be improved in Tcl. [FF] what I mean is: your example is more Tcl code than markup, the markup is hidden and split across the Tcl commands. A template designer (or an X(HT)ML editor) shouldn't care about that. If you don't see advantages in that, maybe you don't need this engine. Note: I'm not stating Tcl has some lack; I'm simply doing an application in Tcl. It is NOT doing "something '''better than Tcl'''"! It's doing something "'''with Tcl'''" - that sounds more appropriate ;) [CMcC] I was hoping there was some lack, so it could be repaired. 03jun07 [jcw] - While subst is indeed great for templating IMO, it is unfortunate that it is completely linear. When you use expr and self-defined [[...]] calls inside to control flow and repetition, you end up nesting things ''twice'' each time: first to embed the operation, and again to protect/return the body of that operation. Even Tcl has two modes for that because it becomes unwieldy: the command mode where each command starts on a newline (or after ";"), and embedded calls with [[...]]. [CMcC] I agree with you, jcw. So I think I'd tend to define an [if] command which evaluated the test using [expr], and then evaluated the consequent or else using [subst] instead of [eval], thus: [If {$fred} {

    header $fred

    <..> } else {

    header not $fred

    }] That would be an hour's work, would fit nicely alongside a modified [foreach] and sit inside a namespace. It was 20 minutes' work, and it's on the [HTML generator] page. Thanks, jcw, for cutting to the nub of the problem. Subst-foreach is next. What other facilities would be useful? [APW] What about Subst-while? [CMcC] Too easy :) It's right after Subst-if on the other page. ---- 03jun07 [jcw] - But I don't see the point in embedding an escape layer using XML. Here's a visually simpler approach: % if {$a != $b} {

    abc

  • [expr {$a != 1 ? $var1 : $var2}]
  • % } else {
  • [myobj getResult]
  • Hello there
    % } What you get is the ability to use Tcl's subst where convenient and an outer-level line-by-line escape for larger amounts of code. Just like in Tcl, where we use ''both'' if's and expr's. The example is contrived and has relatively much flow control, so it doesn't come out as clearly as real-world examples would. The above isn't an arbitrary choice btw - it's what [Mason] uses. As it turns out, less than a dozen lines of Tcl are needed to convert the above into a normal Tcl script which consists mostly of [subst] calls. In fact, I use a second level of <%...%> escapes as well for structuring the entire files into "components". This is the sort of thing [Mason] has worked out long ago, driven by ''practical'' use, not mere preference of a specific technology or language construct. Before taking this discussion further, my suggestion would be to take ''some'' notation and run with it for a while to see just well it works out. You can't discuss these design choices without using them in several decently-sized real-world website applications. Well, not meaningfully anyway - IMO. [APW] That approach looks very good to me and makes sense in that the final HTML layout is still visible in the template! Thanks to [jcw] for bringing that in. [jcw] Here's the code for that approach: proc substify {in {var OUT}} { set pos 0 foreach pair [regexp -line -all -inline -indices {^%.*$} $in] { lassign $pair from to set s [string range $in $pos [expr {$from-2}]] append script "append $var \[" [list subst $s] "]\n" \ [string range $in [expr {$from+1}] $to] "\n" set pos [expr {$to+2}] } set s [string range $in $pos end] append script "append $var \[" [list subst $s] "]\n" } You give it a template, and it returns a script you can eval to apply that template. I turn such scripts into procs, but that's just one way to use this. [APW] Great work [jcw], that's exactly, what I am looking for. [FF] I noticed a little problem with switch: % switch what { % case1 { something % } % what { something other % } % } seems that two adjacent % lines generate an empty append line, and a construct like switch don't like that [APW] Here is a slightly modified version of substify which should handle switch correctly: proc substify {in {var OUT}} { set script "" set pos 0 foreach pair [regexp -line -all -inline -indices {^%.*$} $in] { foreach {from to} $pair break set s [string range $in $pos [expr {$from-2}]] if {[string length $s] > 0} { append script "append $var \[" [list subst $s] "]\n" } append script "[string range $in [expr {$from+1}] $to]\n" set pos [expr {$to+2}] } set s [string range $in $pos end] if {[string length $s] > 0} { append script "append $var \[" [list subst $s] "]\n" } return $script } ''Good catch, thanks! -[jcw]'' FYI, fixed a missing init of $script in the above (when $in is empty). ''[escargo] 3 Jun 2007'' - I think one of the reasons for templates is that you can do some of the work of layout in [WYSIWYG] [HTML] tools, and then decide what parts to replace with parameters and control structures to get a more generalized form. Certainly you would want to be using [CSS] wherever possible to separate structure from presentation, but for abstracting certain content details, having a templating system allows decomposition into something that is closer to the original source than replacing final [HTML] with calls to the [tcllib] html package or with hand-coded Tcl to generate the HTML. ---- [[ [Category Application] ]]