**NAME** ''pandoc-tcl-filter'' - filter for pandoc to embed Tcl-code within Markdown documents and show the output. Provides as well an infrastructure to create your own filters for other tools using Tcl. Code examples for embedding [GraphViz]'s dot code, [Pikchr] diagrams, PIC diagrams, EQN or LaTeX equations, https://plantuml.com/%|%PlantUML%|% diagrams, [R] plots and embedding the [svg] creator package [tsvg] (Thingy SVG creator) are given as well. Since version 0.7.0 there is as well a standalone mode for convering from Markdown to HTML only and a graphical user interface which does not need pandoc. **DESCRIPTION** The [pandoc] application is an advanced Markdown processor that can convert Markdown documents into other document formats like pdf, html or docx. As Markdown itself offers rather limited capabilities in formatting and no advanced features like references, dynamic embedding of other tools like flow chart tools or programming languages etc., the pandoc tool allows filtering of the Markdown processors before final document conversion using user written extensions. Pandoc supports per default filters using [Lua]. See here for the pandoc documentation about this: https://pandoc.org/lua-filters.html. Based on the code of [TR] on the [pandoc] Wiki page I implemented an initial version for a Tcl filter coded in Tcl that allows the embedding of Tcl code and evaluation of this code inside Markdown documents. This can be seen as an simpler alternative to the [tmdoc::tmdoc] package that is easier to embed into the [pandoc] universe. The [tmdoc::tmdoc] package in contrast can be used without using [pandoc] at all to convert such Tcl documents into HTML using just the Tcl packages [tmdoc::tmdoc] and [tcllib]'s [Markdown] package. So [pandoc-tcl-filter] is a filter in the pandoc universe whereas [tmdoc::tmdoc] is a Tcl-only solution, which is independent of pandoc. Since version 0.4.0 pandoc-tcl-filter.tcl can be as well used directly to extract Markdown documentation and process it directly. **LINKS** * Author: [Detlef Groth] * Homepage: https://github.com/mittelmark/DGTcl * Download - standalone application with all filters mentioned below: https://github.com/mittelmark/DGTcl/releases/download/latest/pandoc-tcl-filter.tapp * Download all individual files if you like to use the existing filters and like to add your own filters: https://downgit.github.io/#/home?url=https://github.com/mittelmark/DGTcl/tree/master/pandoc-tcl-filter * Manual: http://htmlpreview.github.io/?https://github.com/mittelmark/DGTcl/blob/master/pandoc-tcl-filter/pandoc-tcl-filter.html * GUI Manual: http://htmlpreview.github.io/?https://github.com/mittelmark/DGTcl/blob/master/pandoc-tcl-filter/filter-view.html * Readme: http://htmlpreview.github.io/?https://github.com/mittelmark/DGTcl/blob/master/pandoc-tcl-filter/Readme.html * Examples: https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/examples/example-dot.html * Presentation at the S & T 2021: https://github.com/mittelmark/DGTcl/blob/master/pandoc-tcl-filter/doc/Groth-S-and-T-2021.pdf * Source code: https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/pandoc-tcl-filter.tcl * Version: 0.7.0 * License: MIT **Filters** The following filters are currently available: * {.tcl} - the Tcl filter - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter-tcl.html%|%example%|% * {.abc} - filter for ABC music notation - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-abc.html%|%filter-abc.html%|% * {.cmd} - filter for Shell scripts, Scripting languages - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-cmd.html%|%filter-cmd.html%|% * {.dot} - filter for [GraphViz] tools dot and neato - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-dot.html%|%filter-dot.html%|% * {.eqn} - filter for EQN equations (groff tool) - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/examples/example-pic.html%|%example%|% * {.mmd} - filter for Mermaid diagrams - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-mmd.html%|%filter-mmd%|% * {.mtex} - filter for LaTex equations - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-mtex.html%|%filter-mtex.html%|% * {.pik} - filter for [Pikchr] diagrams - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-pik.html%|%filter-pik.html%|% * {.pic} - filter for PIC diagrams (groff tool) - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-pic.html%|%filter-pik.htmk%|% * {.pipe} - filter for R, Python, Octave (groff tool) - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-pipe.html%|%filter-pipe.html%|% * {.puml} - filter for PlantUML diagrams - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-puml.html%|%filter-puml.html%|% * {.rplot} - filter for [R] plots - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-rplot.html%|%filter-rplot.html%|% * {.sqlite} - filter for SQLite3 terminal application - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-sqlite.html%|%filter-sqlite.html%|% * {.tcrd} - filter for lyrics with music chords above - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-tcrd.html%|%filter-tcrd.html%|% * {.tdot} - filter for the [tdot] package https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-tdot.html%|%filter-tdot.html%|% * {.tsvg} - filter for the [tsvg] package - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-tsvg.html%|%filter-tsvg.html%|% For more code examples and general comments about installation and usage, see the http://htmlpreview.github.io/?https://github.com/mittelmark/DGTcl/blob/master/pandoc-tcl-filter/Readme.html%|%Readme.html%|% file of the distribution. Here an image about the infrastructure taken from the https://htmlpreview.github.io/?https://github.com/mittelmark/DGTcl/blob/master/pandoc-tcl-filter/pandoc-tcl-filter.html%|%manual page%|%: [pandoc-tcl-filter-infrastructure] **Installation** All the filters mentioned above are added to a single file Tcl script that contains as well the require *rl_json* libraries for Linux and Windows. This file can be downloaded from here https://github.com/mittelmark/DGTcl/releases/download/latest/pandoc-tcl-filter.tapp Copy this file to a folder in your PATH and make it executable. Using the following command line you can then convert, for instance, Markdown documents with embedded Tcl code or code for the other filters into an other document formats like HTML, DOCX, PDF, etc. ====== $ pandoc -s -o output.html --filter pandoc-tcl-filter.tapp input.md ====== **pandoc-tcl-filter.tcl** Below is the Tcl code for the first version; the latest code can be retrieved using the links above. ====== #!/usr/bin/env tclsh package require rl_json # some useful commandline commands to know # applying the filter: # pandoc test.md -s --metadata title="test run with tcl filter" -o test.html --filter dgtools/filter.tcl # inspection commands: # pandoc -s -t json test.md > test.json # python3 -m json.tool test.json | less # pandoc -s -t native test.md | head # easier to read # developing: # cat test.json | tclsh dgtools/filter.tcl # # read the JSON AST from stdin set jsonData {} while {[gets stdin line] > 0} { append jsonData $line } proc debug {jsonData} { puts [::rl_json::json keys $jsonData] } # TR example from https://wiki.tcl-lang.org/page/pandoc # which modifies the header level proc incrHeader {jsonData} { for {set i 0} {$i < [llength [::rl_json::json get $jsonData blocks]]} {incr i} { set blockType [::rl_json::json get $jsonData blocks $i t] if {$blockType eq "Header"} { set headerLevel [::rl_json::json get $jsonData blocks $i c 0] set jsonData [::rl_json::json set jsonData blocks $i c 0 [expr {$headerLevel + 1}]] } } return $jsonData } # the code which process inline Tcl code proc evalTclCode {jsonData} { # the interpreter which executes the code chunks interp create mdi mdi eval { set res "" rename puts puts.orig proc puts {args} { global res if {[lindex $args 0] eq "-nonewline"} { append res "[lindex $args 1]" } else { append res "[lindex $args 0]\n" } return "" } } # collect all blocks and if we have a Tcl block create and append new blocks set blocks "" for {set i 0} {$i < [llength [::rl_json::json get $jsonData blocks]]} {incr i} { if {$i > 0} { append blocks "," ;# migh get trouble if echo=false results=hide ? } set blockType [::rl_json::json get $jsonData blocks $i t] if {$blockType eq "CodeBlock"} { # content has three array elements, type, attributes and the code block code set type [rl_json::json get $jsonData blocks $i c 0 1] ;#type set attr [rl_json::json get $jsonData blocks $i c 0 2] ;# attributes set a [dict create echo true results show eval true] if {[llength $attr] > 0} { foreach el $attr { dict set a [lindex $el 0] [lindex $el 1] } #puts [dict keys $a] } if {[dict get $a echo]} { append blocks "[::rl_json::json extract $jsonData blocks $i]\n" } set cont [rl_json::json get $jsonData blocks $i c 1] set cblock "[::rl_json::json extract $jsonData blocks $i]" if {$type eq "tcl"} { if {[dict get $a eval]} { mdi eval "set res {}" set eres [mdi eval $cont] set eres "[mdi eval {set res}]$eres" if {[dict get $a results] eq "show"} { rl_json::json set cblock c 0 1 [rl_json::json array [list string tclout]] rl_json::json set cblock c 1 [rl_json::json string [regsub {\{(.+)\}} $eres "\\1"]] append blocks ",[::rl_json::json extract $cblock]" } } } } elseif {$blockType eq "Para"} { # check for inline Tcl commands like `tcl set x 1` for {set j 0} {$j < [llength [::rl_json::json get $jsonData blocks $i c]]} {incr j} { set type [rl_json::json get $jsonData blocks $i c $j t] ;#type if {$type eq "Code"} { set code [rl_json::json get $jsonData blocks $i c $j c] set code [lindex $code 1] if {[regexp {\.?tcl } $code]} { set c [regsub {^[^ ]+} $code ""] set res [interp eval mdi $c] set jsonData [rl_json::json set jsonData blocks $i c $j c 1 [rl_json::json string $res]] } } } append blocks "[::rl_json::json extract $jsonData blocks $i]\n" } else { append blocks "[::rl_json::json extract $jsonData blocks $i]\n" } } set ret "\"blocks\":\[$blocks\]" append ret ",\"pandoc-api-version\":[::rl_json::json extract $jsonData pandoc-api-version]" append ret ",\"meta\":[::rl_json::json extract $jsonData meta]" return "{$ret}" } # give the modified document back to Pandoc again: puts -nonewline [evalTclCode $jsonData] ====== **Markdown example (test.md)** ====== --- ## YAML START ### Created By : Detlef Groth ### Created : Sun Aug 22 14:37:16 2021 #### Last Modified : <210823.0605> Pandoc Header title: "The pandoc-tcl-filter written in Tcl" shorttitle: "pandoc-tcl-filter" author: - Detlef Groth date: 2021-21-22 ## YAML END --- # Header 1 - Inline code set X is `tcl set x 1`. ## Subheader 1.1 - Inline code time Current time is: `tcl clock format [clock seconds] -format "%Y-%m-%d %H:%M"` # Header 2 - code blocks Some text, first the obvious "Hello World" example. ```{.tcl echo=true results=show} puts "Hello World!" ``` Now an example with two puts statements; ```{.tcl echo=true results=show} puts "Hello World!" puts "End of TclCode!" ``` More text. Then a raw code example that is not evaluated: ``` raw code line 1 raw code line 2 ``` Now some more examples that just return the last line of the code chunk: ```{.tcl} set x 1 set x ``` Now let's use results=hide to hide the results: ```{.tcl results=hide} set x 2 set x ``` Above there is nothing shown. Next we set eval=false and results=hide. ```{.tcl eval=false results=hide} set x 3 set x ``` Again no output and x should be still two, let's check this: ```{.tcl} set x ``` ```{.tcl} expr { "9" == "9.0"} ``` Does inline text works? `.tcl set x` - not yet ...! But should work with the github version! ```{.tcl} expr { "9" eq "9.0"} ``` Now an example with a function: ```{.tcl} proc sum {arg1 arg2} { set x [expr {$arg1 + $arg2}]; return $x } puts " The sum of 2 + 3 is: [sum 2 3]\n\n" ``` ## EOF ====== **Dot filter example** Here an example code of another filter in an embedded code chunk for using a `.dot` filter: ====== ```{.dot ext=pdf echo=false width=6} digraph G { size="6;9"; margin=0.1; node[style=filled,fillcolor=skyblue;shape="box",width=1,fontname="Alegreya"]; { rank=same; Rmd ; R; Md ; Pandoc; } Rmd -> R -> Md -> Pandoc; Pandoc -> HTML; Pandoc -> Docx; Pandoc -> Pdf; R[shape=box,style=filled,fillcolor=cornsilk,]; Pandoc[shape=box,style=filled,fillcolor=cornsilk]; R[label="R/Knitr"]; } ``` ====== And here the output: [pandoc-tcl-filter-dot-example] **Command line use** ====== pandoc test.md -s --metadata title="test run with tcl filter" -o test.html \ -M date="`date "%B %e, %Y %H%M"`" --filter pandoc-tcl-filter.tcl ====== **TODO's** * code labels * graphics include=true|false width=px height=px etc. * Tcl inline commands in the Meta block - skipped use the date option for pandoc directly as can be seen in the Makefile, see above as well in command line use * show use cases for other packages like [chesschart] or [gdtcl] * other non Tcl-code filters written in Tcl for instance: 1. .csv for loading and displaying csv files (walk over lines and create Markdown table) 2. .dot for displaying Graphviz dot files (done) ---- **History** * 2021-08-21 - project started * 2021-08-23 - this wiki page created * 2021-08-25 - github folder within https://github.com/mittelmark/DGTcl%|%DGTcl%|% * 2021-08-26 - adding [svg] creator for images using [Thingy: a one-liner OO system] * 2021-08-28 - adding Tcl filter infrastructure with examples for tsvg - the thingy svg creator and GraphViz's dot file processors as examples * 2021-11-15 - adding --help flag * 2021-11-23 - adding pikchr, pik, eqn filters, extending documentation * 2021-11-30 - adding rplot filter for embedding R code * 2021-12-04 - running as stand-alone application calling pandoc internally with itself as filter * 2021-12-19 - adding include filter, ABC music, Mermaid, PlantUML, SQLite3 application, results="asis" **Discussion** Please discuss here. Suggestions, ideas are welcome. [DDG] 2021-08-21: Please note, that this is a work in progress. [DDG] 2021-08-25: Latest code is now on github project https://github.com/mittelmark/DGTcl%|%DGTcl%|% [DDG] 2021-11-30: After the Tcl conference I added a few more filters that are included in the standalone application, filters for [Pikchr], the [R] statistical language and for the groff tools eqn2graph and pic2graph [DDG] 2021-12-04: Since version 0.4.0 the pandoc-tcl-filter.tcl file can be as well used directly as document converter as it can calls pandoc then internally itself [TR] 2021-12-09: Nice stuff. How can I make the filter output the Tcl results without having it wrapped into {.tclout} code blocks? My use case is having Tcl produce a markdown table which gets inserted inline into a markdown document. So I convert from markdown to markdown and just want Tcl to produce the markdown tables. A second pandoc run would then convert the expanded markdown into PDF or other formats. Is that possible? [DDG] 2021-12-10: You mean that code block like this: ====== ```{.tcl results="asis"} set tab { | Col1 | Col2 | Col3 | Col4 | | ------------- | ------------- | ------------- | ------------- | } foreach i [list 1 2 3 4 5] { append tab "| cell $i,1 | cell $i,2 | cell $i,3 | cell $i,3 |\n" } set tab ``` ====== should be rendered as Markdown table? Probably `results="md"` should as well work. As Tcl does not know the filetype which should be the final output. With `results="tex"` You could then create LaTeX tables with Tcl as well. I have already an implementation for the code above on my machine, which I could upload to git already. [TR] 2021-12-10: Yes. The output of the Tcl script should be written to the document directly, without putting it into a markdown backticked code block with the {.tclout} class. I thought the {.tcl results="asis"} was for that purpose, leaving the Tcl output as is, but it still wraps the output into a code block. So, I also wonder what the "asis" is for, as this is not documented in the manual or readme. - In the end I would like to be able to do the same operations as in Rmarkdown where I can choose to produce the document and configure the output in various ways or I can produce a Tcl script and strip the other markdown off (with purl(), render() and knit()). Oh, but I see the complexity now. Typically, we would need to produce proper json AST code for tables which is quite hard probably. But when converting from markdown to markdown anyway, we can still let the Tcl script spit out simple markdown table syntax, but then we are looking for the '''RawBlock''' AST element. I think that will be the easiest path! [DDG] 2021-12-10: I have solved this issue by writing the Markdown code in a tempoary file and then calling pandoc with -t json option and extracting the relevant part of the json tree. Not very efficient but it seems to work. See here: * Output: http://htmlpreview.github.io/?https://github.com/mittelmark/DGTcl/blob/master/pandoc-tcl-filter/tests/table.html * Source: https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/tests/table.md To test this you should try the standalone executable probably: https://github.com/mittelmark/DGTcl/suites/4624501730/artifacts/125180612 With this standalone Tcl file you should be able to do simply: ====== pandoc-tcl-filter.tapp table.md table.html -s --css yourstyle.css ====== There is no need to convert first to Markdown and then to HTML as the pandoc call creates the AST already. Hope this helps. Oj, I see. Quite a neat 'trick'! I am testing right now, but have problems with an unmatched brace ... let's take this specific discussion to Github where it might be easier to handle and doesn't fill the Wiki with too much noise. See https://github.com/mittelmark/DGTcl/issues/3%|%this issue%|% [DDG] - 2021-12-21: With version 0.5.0 you can use `results="asis"` to create directly Markdown code using your Tcl scripts, furthermore new filters were added for https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-puml.html%|%PlantUML!!%|% (great tool, easy to install!), https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-sqlite.html%|%SQLite3%|% terminal application (embed and display SQL code and the resulting output in your documents), https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-abc.html%|%ABC music notation%|%, https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-mmd.html%|%Mermaid diagrams%|%. Adding a new filter, just takes a few minutes. Furthermore it is now possible to include other Markdown files using the `results="asis"` type of chunk option. Details are explained in the https://htmlpreview.github.io/?https://github.com/mittelmark/DGTcl/blob/master/pandoc-tcl-filter/pandoc-tcl-filter.html%|%manual%|% page. [DDG] - 2022-01-09: Adding filters for shell and scripting language scripts (Octave, R, Gnuplot, ...) - see https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-cmd.html%|%filter-cmd.html%|%, music lyrics with chords above - see https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-tcrd.html%|%filter-tcrd.html%|%, and large extension of the mtex filter with many examples for LaTeX packages, such as tikz, pgf, skak, sudoku - see https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-mtex.html%|%filter-mtex.html%|% and a filter for the [tdot] package - see https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/filter/filter-tdot.html%|%filter-tdot.html%|%. ---- <> Documentation