FF 2007-30-05 - The discussion evolved in ToW Tcl on WebFreeWay has generate some interest, and here is an effort for making a minimal template engine. I would like to use this page for collecting ideas from people. Fell free to change everything, fix bugs and revolutionize the concept - it is the purpose of the page.
FF 2007-06-02 - I rewrote the engine in a OO-like way (still haven't used any OO system, neither build my OO system) (as a rule, methods beginning with _ are meant to be private), improving some features. Very important is that from now it could run in a recursive way, allowing to design modular/nested templates. (note that since the rewrite I changed all the examples)
Description:
Really, the engine works by converting a template (which is a string, or a file) into a Tcl script, and then running it. Each line of text encountered will be returned as is. Exception is text between <% ... %> which is treated as Tcl code (the eval happens into a safe interp, see Safe Interps). This make it very flexible and it leaves you freedom of doing what you want, dealing with any data and putting how much code you want in the template (although the template should possibly hold minimum code - according to MVC, just the presentation logic, no code related to controller logic, state persistance, or data-retrieval logic).
Also for convenience there is a <%= ... %> for spooling output (the proc _defaultSpool takes care of sending the text to the right place; you can change that: see Preprocessor Commands).
Usage:
first, let's create a template file (templtest.tmpl):
<table><%
for {set i 0} {$i < 4} {incr i} { %>
<tr>
<td><%= $i %></td>
</tr><%
} %>
</table>
<ul>
<% foreach item $cityList {
%><li><%= $item %>
<% } %>
</ul>the above template (in HTML language) generates a 4-row table, and an unordered list, taking values from a $cityList variable, that we're going to define. First let's load the code:
source TemplaTcl
now create a parser instance:
TemplaTcl::create tt
and parse our template:
tt parseFile templtest.tmpl
set the variables used in our template:
tt setVar cityList {Ragusa Ravenna Rieti Rimini Rome Rovigo}and render the template:
puts [tt render]
Here's the output that gets produced:
<table>
<tr>
<td>1</td>
</tr>
<tr>
<td>2</td>
</tr>
<tr>
<td>3</td>
</tr>
<tr>
<td>4</td>
</tr>
</table>
<ul>
<li>Ragusa
<li>Ravenna
<li>Rieti
<li>Rimini
<li>Rome
<li>Rovigo
</ul>
Preprocessor commands:
Running into a safe interp you cannot source your Tcl function library. There's a special markup command: <%@ ... %> which allows you to do special things, i.e. setting parser/interp options. That happens before the parser, and before running the interpreter.
To source a file you will do:
<%@ source = poo.tcl %>
Instead, for including a template:
<%@ include = template.tmpl %>
Also for changing the default spooler (mentioned above) do:
<%@ printCommand = puts -nonewline %>
(that replaces the _defaultSpool spooler, which allows you to get the content as the return value of TemplaTcl::render.
#!/usr/bin/env tclsh
package require Tcl 8.5
# 8.5 required cause the {*} in proc create and proc method
package require struct
# tcllib required
namespace eval ::TemplaTcl {
variable obj
proc method {name args body} {
proc $name [list self {*}$args] "variable obj ; $body"
}
method create {} {
# create and setup a safe interpreter
# for running template's tcl code
catch {if [interp exists $obj($self:interp)] {
interp delete $obj($self:interp)}}
set obj($self:interp) [interp create -safe]
interp share {} stdout $obj($self:interp)
interp eval $obj($self:interp) {
proc _defaultSpool {txt {cmd {}}} {
global content
if {$cmd == "clear"} {set content {}; return}
if {$cmd == "get"} {return $content}
append content $txt
}
}
uplevel "proc $self {method args} {namespace eval ::TemplaTcl \[list \$method $self {*}\$args\]}"
return $self
}
method parseFile {filename} {
# read the template into $rawl - list of chars
set fh [open $filename r]
set raw [read $fh]
close $fh
return [$self parse $raw]
}
method parse {template} {
$self _setMode raw
#$self setOption printCommand "puts -nonewline"
$self setOption printCommand "_defaultSpool"
$self setVar * {}
set q [::struct::queue]
set rawl [split $template {}]
foreach ch $rawl {
# we work char-by-char :|
$q put $ch
# max block to compare (<%=) is 3 chars long:
if {[$q size] >= 3} {
set s3 [join [$q peek 3] {}]
set s2 [join [$q peek 2] {}]
switch $obj($self:mode) {
raw { if {$s3 == "<%="} {
# <%= is a shorthand for puts ...
$q get 3; $self _setMode code;
append obj($self:buf:$obj($self:mode)) "$obj($self:options:printCommand) "
continue
} elseif {$s3 == "<%@"} {
# <%@ is for setting preprocessor options
$q get 3; $self _setMode opt; continue
} elseif {$s2 == "<%"} {
# <% indicates begin of a code block
$q get 2; $self _setMode code; continue
} }
code { if {$s2 == "%>"} {
# and %> is the end of code block
$q get 2; $self _setMode raw; continue
} }
opt { if {$s2 == "%>"} {
# option parser
$q get 2;
$self _parseOptions $obj($self:buf:opt)
set obj($self:buf:opt) {}
$self _setMode raw; continue
} }
}
append obj($self:buf:$obj($self:mode)) [$q get]
}
}
# finish processing the queue:
while {[$q size] > 0} {
append obj($self:buf:$obj($self:mode)) [$q get]
}
$self _setMode flush
# cleanup:
foreach v {buf:code buf:opt buf:raw mode modeprev} {catch {unset obj($self:$v)}}
}
method render {} {
# run the template script
set tclBuf ""
foreach l $obj($self:data) {
set t [lindex $l 0]
set d [lindex $l 1]
switch $t {
raw {append tclBuf "$obj($self:options:printCommand) [list $d]\n"}
code {append tclBuf "$d\n"}
}
}
foreach {var val} $obj($self:variables) {$obj($self:interp) eval [list set $var $val]}
#puts $tclBuf;return
if {$obj($self:options:printCommand) == "_defaultSpool"} {
$obj($self:interp) eval {_defaultSpool {} clear}
}
$obj($self:interp) eval $tclBuf
if {$obj($self:options:printCommand) == "_defaultSpool"} {
set x [$obj($self:interp) eval {_defaultSpool {} get}]
$obj($self:interp) eval {_defaultSpool {} clear}
return $x
}
}
method setOption {opt value} {
switch $opt {
printCommand {set obj($self:options:$opt) $value}
include {
set o inc$value
create $o
$o parseFile $value
set prevMode [$self _setMode raw]
append obj($self:buf:raw) [$o render]
$self _setMode $prevMode
}
source {$obj($self:interp) invokehidden source $value}
default {return -code error -errorinfo "Unknown option: $opt"}
}
}
method setVar {var value} {
if {$var == "*" && $value == ""} {set obj($self:variables) {}; return}
lappend obj($self:variables) $var
lappend obj($self:variables) $value
}
method _setMode {m} {
set modenew {}
switch $m {code - raw - opt {set modenew $m}}
if {$modenew != {}} {
if [catch {set obj($self:mode)}] {set obj($self:mode) $modenew}
set obj($self:modeprev) $obj($self:mode)
set obj($self:mode) $modenew
set obj($self:buf:$obj($self:mode)) {}
if {$obj($self:modeprev) == {}} {
set obj($self:modeprev) $obj($self:mode)
}
}
if {$m == "flush"} {
set obj($self:modeprev) $obj($self:mode)
set obj($self:mode) _
}
if {$obj($self:mode) != $obj($self:modeprev)} {
lappend obj($self:data) [list $obj($self:modeprev) $obj($self:buf:$obj($self:modeprev))]
set obj($self:buf:$obj($self:modeprev)) {}
return $obj($self:modeprev)
}
}
method _parseOptions {o} {
set optlist [split $o "\n;"]
foreach opt $optlist {
set pair [split $opt =]
set opt_ [string trim [lindex $pair 0]]
if {$opt_ == {}} continue
$self setOption $opt_ [string trim [lindex $pair 1]]
}
}}
escargo - I think it's a mistake to have parse read a file; it would be potentially more flexible to use if it just took a string to process as input, and let the caller determine where the string comes from. FF - good point! (changed)
APW Just found textutil::expander[L1 ] package in tcllib, which is included in ActiveTcl 8.4.14.0. Have the feeling it does something similar, maybe it's worth looking at it.
FF but is not much versatile. Just tried this:
% package require textutil::expander
% ::textutil::expander myexp
% set a 64
% myexp expand {[if {$a > 50} {] put some text [} {] put some other text [}]}
Error in macro:
[if {$a > 50} {] put some text [} {] put some other text [}]
--> missing close-bracketand I realized I feel well with TemplaTcl O:)
APW I think if you look closer to the manual you will see that you have to set the left and right bracket command i.e. "<%" and "%>". Next the token directly following the left bracket is interpreted as the command for handling the contents, so if you have "<%= hello %>" you will need a proc with name "=" for handling the code inside the macro.
% package require textutil::expander
% ::textutil::expander te
% te lb "<%"
% te rb "%>
% proc = {args} { return $args }
% te expand {this is a text which says <%= hello %> to you}
this is a text which says hello to youAdditionally for constructs like a "if" clause it is possible to push and pop the context and at the end to evaluate the code in the macro, normally you will need some tokens for marking begin and end of macro, means a construct like:
<% code if { ...} { %>
# some other code here
<% /code %>
proc code {args} {
te cpush ....
# some code here
}
proc /code {args} {
set str [te cpop ...]
# evaluate the code of the macro and set result
return result
}You can also define the number of passes to run for the same text to be able to do nested expansion if you use the expand.tcl directly from the author: William Duquette [L2 ]
escargo - I think you want to give some thought into what scope the template code executes at. Right now it seems to be run in proc dump. Two likely alternatives are to run it in the caller's scope or at global scope.
FF running in the caller scope might seem a convenient idea, but can potentially generate trouble, since the template is a tcl script that would run just between TemplaTcl::dump and the next command, potentially altering/damaging the caller's scope (if there are subsequent commands after the TemplaTcl::dump command); in order to make it more strict, I make it run in global scope (hoping that global is not polluted or has variable that can make the application unstable); but ideally it should run in a separate interp, copying the needed vars into that interpreter. Don't you think so?
escargo - Running in a safe interp would be the right thing, but the question might be what data can the code executing as part of template processing access (or access easily). One could reasonable argue that it should only access values provided by a supplied data object. I could see wanting to supply maybe an environment data object as well (for access to info like date and time, host name, user name, and other info that would not be part of the data coming out of a table to be published).
FF added safe interp, setVar command, and also a new markup command for special use (see above).
escargo 1 Jun 2007 - See also subst. In some respects, subst is a better model than eval, since it assumes text with variable references and commands embedded in it, instead of valid programs with text embedded in the commands. The ability to intercept variable references and command executions that you don't know ahead of time would simplify some things that you want to do, but those aren't available yet. (Maybe what's needed is a subst that has been extended to take a -variables and/or -commands arguments, which are the names of commands to call when there are variable references or command references respectively.)
FF 2007-06-02 - changed much things (almost all). I implemented a pseudo OO system. This allows running recursive/nested templates. If you were interested in this project, maybe you might be interested in re-reading this page from the beginning.
CMcC 2Jun07 just doesn't get it - why create a language interpreter to substitute variables and function calls when tcl has a full-function command to do that, in subst? What does the new litle language for templating buy you?
APW Would that also work for the following example? I don't think so!
<%if {$a != $b} { %>
<h3> abc </h3>
<%if {$a != 1} {%>
<li> <%= $var1 %> </li>
<%} else {%>
<li> <%= $var2 %> </li>
<% } %>
<% } else { %>
<li> <% myobj getResult %> </li>
<table>
<tr> Hello there </tr>
</table>
<% } %>
CMcC no, this would work better:
expr {
$a != $b
? "[<h3> abc][<li> [expr {$a != 1 ? $var1: $var2}]]"
: "[<li> [myobj getResult]] [<table> [<tr> "Hello there"]]"
}This would also work:
if {$a != $b} {
subst {<h3> abc </h3>
[expr {$a != 1
? "<li> $var1 </li>"
: "<li> $var2 </li>"}]
}
} else {
subst {
<li> [myobj getResult] </li> <table> <tr> Hello there </tr> </table>
}
}(note: if returns a value!)
But nothing would work as well as refactoring it so it wasn't quite as messy in the first place.
escargo - subst does not have a notion of iteration to expand things inside the string being operated on, for one thing.
CMcC tcl has the concept of iteration, though. Use map, or it's easy to write a function which iterates (think foreach which returns its iterated results). For example, the thing above to turn a list into an HTML list could be simply represented as <ul><li>[join $list </li><li>]</li></ul>, which is cleaner (I think) ... or even define a proc (such as is already in the tcllib html package) to perform the wrapping.
The page HTML generator provides some light-weight facilities along those lines.
FF the above mentioned systems sure are good for some things. But my target is a very specific case, where those systems maybe lack some things.
TemplaTcl works with templates rich of static text, but also rich of interruptions due to code or putting variables (the typical case is generating markup for a web page). Dealing with that using other systems can possibly result being counter-productive (the TemplaTcl examples found in this page are too simple to demonstrate that).
CMcC do you mean that you can stop the template expansion midway? subst can also do that.
FF Even if right now it does its job nicely - just now I added template inclusion (<%@include = ..%>), and tcl source-ing (<%@source = ...%>) - I'll keep adding missing features to satisfy the case of code generation, with a particular regard to X(HT)ML page templating, so don't expect it to be just that tiny set of features.
CMcC What I'm interested in is examples of some useful things TemplateTcl does better than Tcl, because that's a sign of something which could be improved in Tcl.