pandoc-tcl-filter

NAME

pandoc-tcl-filter - filter for pandoc to embed Tcl-code within Markdown documents and show the output. Provides as well an infrastructure to create your own filters for other tools using Tcl. Code examples for embedding GraphViz's dot code, Pikchr diagrams, PIC diagrams, EQN or LaTeX equations, R plots and embedding the svg creator package tsvg (Thingy SVG creator) are given as well.

DESCRIPTION

The pandoc application is an advanced Markdown processor that can convert Markdown documents into other document formats like pdf, html or docx. As Markdown itself offers rather limited capabilities in formatting and no advanced features like references, dynamic embedding of other tools like flow chart tools or programming languages etc., the pandoc tool allows filtering of the Markdown processors before final document conversion using user written extensions. Pandoc supports per default filters using Lua. See here for the pandoc documentation about this: https://pandoc.org/lua-filters.html . Based on the code of TR on the pandoc Wiki page I implemented an initial version for a Tcl filter coded in Tcl that allows the embedding of Tcl code and evaluation of this code inside Markdown documents. This can be seen as an simpler alternative to the tmdoc::tmdoc package that is easier to embed into the pandoc universe. The tmdoc::tmdoc package in contrast can be used without using pandoc at all to convert such Tcl documents into HTML using just the Tcl packages tmdoc::tmdoc and tcllib's Markdown package. So pandoc-tcl-filter is a filter in the pandoc universe whereas tmdoc::tmdoc is a Tcl-only solution, which is independent of pandoc. Since version 0.4.0 pandoc-tcl-filter.tcl can be as well used directly to extract Markdown documentation and process it directly.

LINKS

Filters

The following filters are currently available:

For more code examples and general comments about installation and usage, see the Readme.html file of the distribution.

Here an image about the infrastructure taken from the manual page :

pandoc-tcl-filter-infrastructure

Installation

All the filters mentioned above are added to a single file Tcl script that contains as well the require *rl_json* libraries for Linux and Windows. This file can be downloaded from here https://github.com/mittelmark/DGTcl/releases/download/latest/pandoc-tcl-filter.tapp

Copy this file to a folder in your PATH and make it executable. Using the following command line you can then convert, for instance, Markdown documents with embedded Tcl code or code for the other filters into an other document formats like HTML, DOCX, PDF, etc.

$ pandoc -s -o output.html --filter pandoc-tcl-filter.tapp input.md

pandoc-tcl-filter.tcl

Below is the Tcl code for the first version; the latest code can be retrieved using the links above.

#!/usr/bin/env tclsh
package require rl_json
# some useful commandline commands to know
# applying the filter:
# pandoc test.md -s --metadata title="test run with tcl filter" -o test.html --filter dgtools/filter.tcl
# inspection commands:
# pandoc -s -t json test.md > test.json
# python3 -m json.tool test.json | less
# pandoc -s -t native test.md | head # easier to read
# developing:
# cat test.json | tclsh dgtools/filter.tcl
# # read the JSON AST from stdin
set jsonData {}
while {[gets stdin line] > 0} {
   append jsonData $line
}

proc debug {jsonData} {
    puts [::rl_json::json keys $jsonData]
}

# TR example from https://wiki.tcl-lang.org/page/pandoc
# which modifies the header level
proc incrHeader {jsonData} {
    for {set i 0} {$i < [llength [::rl_json::json get $jsonData blocks]]} {incr i} {
        set blockType [::rl_json::json get $jsonData blocks $i t]
        if {$blockType eq "Header"} {
            set headerLevel [::rl_json::json get $jsonData blocks $i c 0]
            set jsonData [::rl_json::json set jsonData blocks $i c 0 [expr {$headerLevel + 1}]]
        } 
    }
    return $jsonData
}

# the code which process inline Tcl code
proc evalTclCode {jsonData} {
    # the interpreter which executes the code chunks
    interp create mdi
    mdi eval {
        set res ""
        
        rename puts puts.orig
        
        proc puts {args} {
            global res
            if {[lindex $args 0] eq "-nonewline"} {
                append res "[lindex $args 1]"
            } else {
                append res "[lindex $args 0]\n"
            }
            return ""
        }
    }
    # collect all blocks and if we have a Tcl block create and append new blocks
    set blocks ""
    for {set i 0} {$i < [llength [::rl_json::json get $jsonData blocks]]} {incr i} {
        if {$i > 0} {
            append blocks "," ;# migh get trouble if echo=false results=hide ?
        }
        set blockType [::rl_json::json get $jsonData blocks $i t]
        if {$blockType eq "CodeBlock"} {
            # content has three array elements, type, attributes and the code block code 
            set type [rl_json::json get $jsonData blocks $i c 0 1] ;#type
            set attr [rl_json::json get $jsonData blocks $i c 0 2] ;# attributes
            set a [dict create echo true results show eval true]
            if {[llength $attr] > 0} {
                foreach el $attr {
                    dict set a [lindex $el 0] [lindex $el 1]
                }
                #puts [dict keys $a]
            }
            if {[dict get $a echo]} {
                append blocks "[::rl_json::json extract $jsonData blocks $i]\n"
            }
            set cont [rl_json::json get $jsonData blocks $i c 1]
            set cblock "[::rl_json::json extract $jsonData blocks $i]"
            if {$type eq "tcl"} {
                if {[dict get $a eval]} {
                    mdi eval "set res {}"
                    set eres [mdi eval $cont]
                    set eres "[mdi eval {set res}]$eres"
                    if {[dict get $a results] eq "show"} {
                        rl_json::json set cblock c 0 1 [rl_json::json array [list string tclout]]
                        rl_json::json set cblock c 1 [rl_json::json string [regsub {\{(.+)\}} $eres "\\1"]]
                        append blocks ",[::rl_json::json extract $cblock]"
                    }
                }
            }
        } elseif {$blockType eq "Para"} {
            # check for inline Tcl commands like `tcl set x 1`
            for {set j 0} {$j < [llength [::rl_json::json get $jsonData blocks $i c]]} {incr j} {
                set type [rl_json::json get $jsonData blocks $i c $j t] ;#type
                if {$type eq "Code"} {
                    set code [rl_json::json get $jsonData blocks $i c $j c]
                    set code [lindex $code 1]
                    if {[regexp {\.?tcl } $code]} {
                        set c [regsub {^[^ ]+} $code ""]
                        set res [interp eval mdi $c]
                        set jsonData [rl_json::json set jsonData blocks $i c $j c 1 [rl_json::json string $res]]
                    }
                }
            }
            append blocks "[::rl_json::json extract $jsonData blocks $i]\n"
        } else {
            append blocks "[::rl_json::json extract $jsonData blocks $i]\n"
        }
    }
    set ret "\"blocks\":\[$blocks\]"
    append ret ",\"pandoc-api-version\":[::rl_json::json extract $jsonData pandoc-api-version]"
    append ret ",\"meta\":[::rl_json::json extract $jsonData meta]"
    return "{$ret}"
}

# give the modified document back to Pandoc again:

puts -nonewline [evalTclCode $jsonData]

Markdown example (test.md)

---
## YAML START
###  Created By    : Detlef Groth
###  Created       : Sun Aug 22 14:37:16 2021
####  Last Modified : <210823.0605> Pandoc Header
title: "The pandoc-tcl-filter written in Tcl"
shorttitle: "pandoc-tcl-filter"
author: 
- Detlef Groth
date: 2021-21-22
## YAML END
---

# Header 1 - Inline code set

X is `tcl set x 1`.

## Subheader 1.1 - Inline code time

Current time is: `tcl clock format [clock seconds] -format "%Y-%m-%d %H:%M"`

# Header 2 - code blocks

Some text, first the obvious "Hello World" example.

```{.tcl echo=true results=show}
puts "Hello World!"
```

Now an example with two puts statements;

```{.tcl echo=true results=show}
puts "Hello World!"
puts "End of TclCode!"
```

More text. Then a raw code example that is not evaluated:

```
raw code line 1
raw code line 2
```

Now some more examples that just return the last line of the code chunk:

```{.tcl}
set x 1
set x
```

Now let's use results=hide to hide the results:

```{.tcl results=hide}
set x 2
set x
```

Above there is nothing shown. Next we set eval=false and results=hide.


```{.tcl eval=false results=hide}
set x 3
set x
```

Again no output and x should be still two, let's check this:

```{.tcl}
set x
```


```{.tcl}
expr { "9" == "9.0"}
```

Does inline text works? `.tcl set x`  - not yet ...! 
But should work with the github version!


```{.tcl}
expr { "9" eq "9.0"}
```
Now an example with a function:

```{.tcl}
proc sum {arg1 arg2} {
    set x [expr {$arg1 + $arg2}];
    return $x
}

puts " The sum of 2 + 3 is: [sum 2 3]\n\n"
```

## EOF

Dot filter example

Here an example code of another filter in an embedded code chunk for using a .dot filter:

```{.dot ext=pdf echo=false width=6}
digraph G {
    size="6;9";
    margin=0.1;
    node[style=filled,fillcolor=skyblue;shape="box",width=1,fontname="Alegreya"];
    { rank=same; Rmd ; R; Md ; Pandoc; }
    Rmd -> R -> Md -> Pandoc;
    Pandoc -> HTML;
    Pandoc -> Docx;
    Pandoc -> Pdf;
    R[shape=box,style=filled,fillcolor=cornsilk,];
    Pandoc[shape=box,style=filled,fillcolor=cornsilk];
    R[label="R/Knitr"];
}
```

And here the output:

pandoc-tcl-filter-dot-example

Command line use

pandoc test.md -s --metadata title="test run with tcl filter" -o test.html \ 
   -M date="`date "%B %e, %Y %H%M"`" --filter pandoc-tcl-filter.tcl

TODO's

  • code labels
  • graphics include=true|false width=px height=px etc.
  • Tcl inline commands in the Meta block - skipped use the date option for pandoc directly as can be seen in the Makefile, see above as well in command line use
  • show use cases for other packages like chesschart or gdtcl
  • other non Tcl-code filters written in Tcl for instance:
  1. .csv for loading and displaying csv files (walk over lines and create Markdown table)
  2. .dot for displaying Graphviz dot files (done)

History

  • 2021-08-21 - project started
  • 2021-08-23 - this wiki page created
  • 2021-08-25 - github folder within DGTcl
  • 2021-08-26 - adding svg creator for images using Thingy: a one-liner OO system
  • 2021-08-28 - adding Tcl filter infrastructure with examples for tsvg - the thingy svg creator and GraphViz's dot file processors as examples
  • 2021-11-15 - adding --help flag
  • 2021-11-23 - adding pikchr, pik, eqn filters, extending documentation
  • 2021-11-30 - adding rplot filter for embedding R code
  • 2021-12-04 - running as stand-alone application calling pandoc internally with itself as filter

Discussion

Please discuss here. Suggestions, ideas are welcome.

DDG 2021-08-21: Please note, that this is a work in progress.

DDG 2021-08-25: Latest code is now on github project DGTcl

DDG 2021-11-30: After the Tcl conference I added a few more filters that are included in the standalone application, filters for Pikchr, the R statistical language and for the groff tools eqn2graph and pic2graph

DDG 2021-12-04: Since version 0.4.0 the pandoc-tcl-filter.tcl file can be as well used directly as document converter as it can calls pandoc then internally itself