Version 16 of Templates and subst

Updated 2012-11-12 18:24:38 by pooryorick

APW 2007-06-07 The contents of this page has been moved from TemplaTcl: a Tcl Template engine to here, to be able to reference that code. It is an example on how to use a simple templating language which can have Tcl code interspersed.

03jun07 jcw - But I don't see the point in embedding an escape layer using XML. Here's a visually simpler approach

  % if {$a != $b} {
      <h3> abc </h3>
      <li> [expr {$a != 1 ? $var1 : $var2}] </li>
  % } else {
      <li> [myobj getResult] </li>
      <table>
         <tr> Hello there </tr>
      </table>
  % }

What you get is the ability to use Tcl's subst where convenient and an outer-level line-by-line escape for larger amounts of code. Just like in Tcl, where we use both if's and expr's.

The example is contrived and has relatively much flow control, so it doesn't come out as clearly as real-world examples would. The above isn't an arbitrary choice btw - it's what Mason uses. As it turns out, less than a dozen lines of Tcl are needed to convert the above into a normal Tcl script which consists mostly of subst calls. In fact, I use a second level of <%...%> escapes as well for structuring the entire files into "components". This is the sort of thing Mason has worked out long ago, driven by practical use, not mere preference of a specific technology or language construct.

Before taking this discussion further, my suggestion would be to take some notation and run with it for a while to see just well it works out. You can't discuss these design choices without using them in several decently-sized real-world website applications. Well, not meaningfully anyway - IMO.

APW That approach looks very good to me and makes sense in that the final HTML layout is still visible in the template! Thanks to jcw for bringing that in.

jcw Here's the code for that approach:

  proc substify {in {var OUT}} {
    set pos 0  
    foreach pair [regexp -line -all -inline -indices {^%.*$} $in] {
      lassign $pair from to
      set s [string range $in $pos [expr {$from-2}]]
      append script "append $var \[" [list subst $s] "]\n" \
                    [string range $in [expr {$from+1}] $to] "\n"
      set pos [expr {$to+2}]
    }
    set s [string range $in $pos end]
    append script "append $var \[" [list subst $s] "]\n"
  }

You give it a template, and it returns a script you can eval to apply that template. I turn such scripts into procs, but that's just one way to use this.

APW Great work jcw, that's exactly, what I am looking for.

pyk: AMG highly recommends JCW's [substify], and uses a variant of it to drive his Web site [L1 ], including a custom search engine made for a friend. See his code here: [L2 ].

  • First look at template [L3 ]
  • Then look at [emit_template] in site.tcl [L4 ]
  • Which is called near the bottom of script [L5 ].

FF I noticed a little problem with switch:

 % switch what {
 % case1 {
 something
 % }
 % what {
 something other
 % }
 % }

seems that two adjacent % lines generate an empty append line, and a construct like switch don't like that

APW Here is a slightly modified version of substify which should handle switch correctly:

  proc substify {in {var OUT}} {
    set script ""
    set pos 0
    foreach pair [regexp -line -all -inline -indices {^%.*$} $in] {
      foreach {from to} $pair break
      set s [string range $in $pos [expr {$from-2}]]
      if {[string length $s] > 0} {
        append script "append $var \[" [list subst $s] "]\n"
      }
      append script "[string range $in [expr {$from+1}] $to]\n"
      set pos [expr {$to+2}]
    }
    set s [string range $in $pos end]
    if {[string length $s] > 0} {
      append script "append $var \[" [list subst $s] "]\n"
    }
    return $script
  }

Good catch, thanks! -jcw

FYI, fixed a missing init of $script in the above (when $in is empty).

merlin, 2007-11-02, 17:00 GMT: I think, it is simple to use a native tcl interpreter to parse templates. We have a simple template syntax, which is easy to convert ("compile" a template) into proper and correct tcl code. Doing subst on that code ("fetch" a template), we get a rendered page.

Compiling can be done without any variables set, which are used in the template. "Fetching" requires these variables to be set in the current scope (else we get a tcl error, about using unset variables).

  # ====== Compiler ======
  # compilng section block
  proc parse_section {arg} {return "\[_section [lindex $arg 0] [parse [lindex $arg 1]] \{"}
  proc parse_end_section {} {return "\}\]"}

  # compiling if-else blocks
  proc parse_if {arg} {return "\[if [parse $arg] \{_echo \""}
  proc parse_else {arg} {return "\"\} else \{_echo \""}
  proc parse_end_if {} {return "\"\}\]"}

  # curly braces
  proc parse_lbr {arg} {return "\\\{"}
  proc parse_rbr {arg} {return "\\\}"}

  # Determines block {...} type, passed to this proc as str parameter
  # and calls a proc to change that content into complement tcl-code.
  # Returns a new line to substitute {...} block
  proc parse {str} {
          if [string equal -length 1 $str \$] {
                  # Block starting with $ is a command to put variable value
                  if [regexp {\$(.+)\[(\d+)\]} $str trash vh vi] {
                          # If there are square brackets with an index in there, like [3],
                          # replacing that recursively with call to list index
                          set str "\[lindex [parse "$$vh"] $vi\]"
                  }
          } elseif [string equal -length 1 $str /] {
                  # Block starting with / is an end for some block.
                  # It can't contain any parameters
                  set cmd [string range $str 1 end]
                  set str [parse_end_$cmd]
          } else {
                  if [regexp {^(\w+)(?: (.+))?$} $str trash cmd arg] {
                          # If the block begins with the word, that word is considered as
                          # command name, the rest is parameters of that command
                          set str [parse_$cmd $arg]
                  }
                  # !TODO!: allow "else" with expressions parsing
          }
          return $str
  }

  # Find and replace any {...} to [parse ...], then subst the result
  proc compile {tpl} {
          upvar template template

  # --- Save ", ], [, $ between blocks ---
          # Find coordinates of that "betweens"
          set beg 0
          set escapes {}
          while {[regexp -start $beg -indices {\{([^\}]+)\}} $tpl res]} {
                  set escapes [concat $beg [expr [lindex $res 0]-1] $escapes]
                  set beg [expr [lindex $res 1]+1]
          }
          set escapes [concat $beg [expr [string length $tpl]-1] $escapes]
          # $escapes now contain a reversed list of blocks to escape

          # Escape rules
          set escamap {\" \\\\\\\" \$ \\\\\\\$ \[ \\\\\\\[ \] \\\\\\\]}
          # " (extra \ for subst)

          foreach {i j} $escapes {
                  set tpl [string replace $tpl $i $j [string map $escamap [string range $tpl $i $j]]]
          }

          set template $tpl

          # Regexp below finds any {...} and replaces them to [parse ...].
          # then subst calls there parses, and strips some backslashes
          uplevel {return [subst [regsub -all {\{([^\}]+)\}} $template {[parse {\1}]}]]}
  # !TODO! rewrite a compile to more clean code,
  # allowing a nested {} in the {...} blocks,
  # which will enable syntax like {section i {a b c}}<li>{$i}</li>{/section}
  }

  # ====== "Fetcher" ======
  proc _echo {val} {return $val}
  proc _section {name list block} {
          set result {}
          foreach $name $list {append result [subst $block]}
          return $result
  }

  proc fetch {ctpl} {
    upvar template template
    set template $ctpl
    uplevel {return [subst $template]}
  }

Usage of this code is demonstrated below (assuming code is in the file cp.tcl):

  source "cp.tcl"

  # A template
  set tpl {Some{if $e}<ul>{section i $cycle}<li>"{$i}"</li>{/section}</ul>{else}"{$cycle[1]}"{/if}tails}

  # Data for that template
  set e 1
  set cycle {one two three}

  # ====== Call of compiler and fetcher ======
  puts "Source template: $tpl"
  puts "Compiled template: [compile $tpl]"
  puts "Filled-in block: [fetch [compile $tpl]]"

Everything (fetch, compile) is done with subst =)


AMG: [substify] is great stuff. Thanks, JCW! See Web Templating for more information on how I use it.

AMG again: Here is [substify] adapted to work as a simple Unix-style text filter.

package require Tcl 8.5
proc applytemplate {template} {
    set script ""
    set pos 0
    foreach pair [regexp -line -all -inline -indices {^%.*$} $template] {
        lassign $pair from to
        set str [string range $template $pos [expr {$from - 2}]]
        if {$str ne ""} {
            append script "puts \[" [list subst $str] \]\n
        }
        append script [string range $template [expr {$from + 1}] $to]\n
        set pos [expr {$to + 2}]
    }
    set str [string range $template $pos end]
    if {$str ne ""} {
        append script "puts -nonewline \[" [list subst $str] \]
    }
    uplevel 1 $script
}
while {[lindex $argv 0] eq "-file"} {
    set chan [open [lindex $argv 1]]
    set argv [concat [lrange $argv 2 end] [read $chan]]
    close $chan
    unset chan
}
dict with argv {applytemplate [read stdin]}

Example usage:

$ cat enums.c.tmpl
#include <stdlib.h>
typedef enum {
% foreach enum $enums {
    $enum,
% }
} ${name}_t;
char *${name}_names\[\] = {
% foreach enum $enums {
    "$enum",
% }
    NULL
};
% # This is a comment.
$ ./template.tcl enums "a b c d e" name "letter" < enums.c.tmpl > enums.c
$ cat enums.c
#include <stdlib.h>
typedef enum {
    a,
    b,
    c,
    d,
    e,
} letter_t;
char *letter_names[] = {
    "a",
    "b",
    "c",
    "d",
    "e",
    NULL
};

Lordmundi: [substify] is great. I'm using it to serve web templates like I'm guessing many others. However, I ran into an issue where the pages I'm serving have javascript in them, and therefore, lots of dollar signs("$") in them, which are getting interpreted by subst. I can backslash them, but this tends to make it very messy. Is there an easy way to edit substify such that another char, possible "#" at the beginning denotes pure text much in the way that % marks pure tcl? I tried editing it a couple of times and am not having much success.


Bezoar : I ran in to same problem as Lordmundi above. I was able to just declare tcl variables to resubstitute themselves back in to the script. Thank goodness the {} is a valid array name. For example:

% array set {} { document "$(document)" "'#sqlresult'" "$('#sqlresult')" }
    <script>
      $(document).ready(function() {
       $('#sqlresult').dataTable();
      } );
     $(document).ready( function () {
       $('#sqlresult').dataTable( {
        "bPaginate": false
       } );
      } );
     $(document).ready( function () {
     $('#sqlresult').dataTable( {
        "bLengthChange": true
       } );
      } );
   </script>

Will result in the script block going through unchanged.