wiki database for offline use

2017-11-22 -- Any hope of somebody giving me access to a recent database backup?

Zip files containing all Wub, wikitcl and tcllib code you need to run a local wiki can be found here . From 30-oct-2009 on, wubwikit is working with the new tdbc::sqlite backend.

This wiki's data is stored in a single datafile, the path to which is then provided to the wiki executable code on the command line.

News

jdc 2-may-2012: Added support for HTML 5.

jdc 24-apr-2012: Added support for sqlite FTS. The copy of the wiki database available at [1 ] now also contains the required table for FTS. FTS replaces the (Google) search in titles/pages.

jdc 18-mar-2011: Fixed Windows bugs. No longer providing starkits.

jdc 17-feb-2011: Possibility to use and generate your own Wub config file, option to enable SQL queries.

jdc 13-jan-2011: Fixed possible SQL-injection bug.

jdc 03-jan-2011: Added options to disable ReCaptcha's.

jdc 10-sep-2010: Updated to latest version of Wub.

jdc 11-jun-2010: Bug fixes for utils and release based on Wub REL-2.0.1.

jdc 31-may-2010: Better support for using WubWikit as simple CMS, support for dumping images stored in the database, support for ADMIN:MOTD, ADMIN:TOC and ADMIN:Welcome pages, making it easier to restyle your Wiki.

jdc 12-mar-2010: Support for dumping a wiki as a static site.

jdc 12-dec-2009: Added doctools support.

jdc 27-nov-2009: New sqlite Wiki database with support for in-db images. New archive to run your own wiki. No longer uses command line arguments to set the wiki parameter but a configuration file. A default configuration can be generated. This release also supports in-db images and has more reports.

jdc 04-nov-2009: Rework of utils.

jdc 30-oct-2009: Archive to run your own wiki with tdbc backend. To convert your Metakit database to sqlite check [2 ]. This release no longer supports the Tk GUI.

jdc 28-oct-2009: New sqlite Wiki database. The latest Metakit database is wikit-20091026.tkd.zip, from wikit-20091028.tkd.zip on you'll get sqlite databases.

jdc 7-aug-2009: Added GUI to stop server and start browser (contributed by Gaston Rodriguez). Reworked utility to get pages from a wiki database. Files can be found at http://code.google.com/p/wubwikit/downloads/list .

jdc 5-aug-2009: Added utilities to get headers and pages from a wiki database. Files can be found at http://code.google.com/p/wubwikit/downloads/list .

jdc 23-jun-2009: Release including <<categories>> markup and adding comments. Files can be found at http://code.google.com/p/wubwikit/downloads/list or https://sourceforge.net/project/showfiles.php?group_id=211498&package_id=265649

jdc 9-jun-2009: Bugfixes issued by Gaston Rodriguez for yscroll and registry problem. Files can be found at http://code.google.com/p/wubwikit/downloads/list .

jdc 19-mar-2009: More control over url as displayed on wiki pages. Released as kit and as zip. Zip can be used without needing a tclkit. Files can be found at http://code.google.com/p/wubwikit/downloads/list .

jdc 18-feb-2009: Release based on co-routine powered Wub. The project to provide a wiki for offline use is now hosted on http://code.google.com/p/wubwikit . The database stays on http://sourceforge.net/project/showfiles.php?group_id=211498

jdc 12-dec-2008: Release with latest wiki modifications. A new kit is available for download [3 ].

jdc 7-nov-2008: Fixed problems with passing font settings to Tk mode, fixed gui problem in Tk mode where a gap appeared between TOC and data fram when resizing the main window, added options to set port and command port in wub mode. A new kit is available for download [4 ].

jdc 3-jul-2008: Fixed problem if database was placed in same directory as the wukwikit.kit, a new kit is available for download [5 ].

jdc 26-mar-2008: A new wubwikit starkit is released [6 ] adding command line arguments to set font families and sizes used when used as Tk application.

jdc 13-mar-2008: A new wubwikit starkit is released [7 ] adding new command line argument toc to set table-of-contents in the specified database.

jdc 12-mar-2008: A new wubwikit starkit is released [8 ] solving the following problems:

  • couldn't read file local.tcl (reported by MHo 3-mar-2008)
  • db.tcl: no such file or directory (reported by LV on 12-mar-2008)

jdc 3-mar-2008: I packed Wub and wikit in one Starkit. It can downloaded from here . This kit needs a 8.5 Tclkit.

jdc 4-feb-2008: From wikit-20080204.tkd on, I'm running sdx mkpack on the database before uploading it.

jdc 7-dec-2007: The wiki database can be downloaded for offline use from here .

Usage of unzippped archive:

 % tclsh wubwikit<version>/main.tcl <options>

Requirements:

  • tclsh based on Tcl/Tk 8.6
  • package tdbc, part of Tcl/Tk 8.6
  • package tdbc::sqlite3 from http://tdbc.tcl.tk , version >= 1.0b16
  • package sqlite3, version >= 3.7.3

Typical usage:

Create a new wiki database:

 % tclsh wubwikit<version>/main.tcl mkdb mywiki.tkd title "My Wiki"

This will create a new wiki database and use the specified title as wiki title.

Start a wiki:

 % tclsh wubwikit<version>/main.tcl wikidb mywiki.tkd

Start a wiki with a copy of the Tcler's wiki database:

 % tclsh wubwikit<version>/main.tcl wikidb wikit.tkd

Command line options

Basic options:

helpShow this message
wikidb <file>Set path to Wiki database. Mandatory!
mkdb <file> title <title>Create empty Wiki database.
mklocal <path>Create default Wiki local.tcl to configure your Wiki.
mkconfig <path>Create default Wiki config file to configure the Wub webserver.
mkfts <boolean>Add full-text-search table to specified wiki-database.

Configuration options:

port <port>Set port used by Wub, default is 8080.
cmdport <port>Set command port used by Wub, you can telnet to this port to interface with the webserver, default is 8082.
logfile <file>Set name of log file
local <file>Wiki config file to be used for your wiki. The output of the mklocal command is used as default.
config <file>Wub config file to be used for your wiki. The output of the mkconfig command is used as default.

Utilities:

util idsPrint one line per page with page-id, indication if ok or empty and page title to standard output. The resulting output can be used as input for the util html|markup commands.
util html|markup page <page-id> ?opath <path>? ?html_ext <html-extension>?Print html or markup for specified page to file <opath>/<page-id><html-extension>|.txt.
util html|markup pages <file> ?opath <path>? ?html_ext <html-extension>?Print html or markup for each page specified with its <page-id> in <file> to file <opath>/<page-id><html-extension>|.txt. Put each page-id as first item on a separate line in <file>. The output of other util commands can be used as input for this command.
util statsPrint some statistics about the wiki database.
util binary_idsPrint list of image text pages. Print one line per page with id and page title to standard output. The resulting output can be used as input for the util html|markup commands.
util count_empty_text_pagesCount all empty text pages (content size <= 1).
util count_image_pagesCount all image pages.
util count_image_pages_unreferenced_by_othersCount image pages not referenced by any other pages.
util count_non_empty_text_pagesCount all non empty text pages (content size > 1).
util count_non_empty_text_pages_unreferenced_by_othersCount non-empty text pages not referenced by any other pages.
util count_non_empty_text_pages_without_references_to_othersCount non-empty text pages without any references to other pages.
util count_pagesCount all pages.
util count_pages_without_contentCount without content. This are pages which have a title but haven't got content yet, not even content with size < 1.
util count_text_pagesCount all image pages.
util empty_text_pagesPrint list of empty text pages (content size <= 1). Print one line per page with id and page title to standard output. The resulting output can be used as input for the util html|markup commands.
util image_pages_unreferenced_by_othersPrint list of image pages not referenced by any other pages. Print one line per page with id and page title to standard output. The resulting output can be used as input for the util html|markup commands.
util non_empty_text_pagesPrint list of non-empty text pages (content size > 1). Print one line per page with id and page title to standard output. The resulting output can be used as input for the util html|markup commands.
util non_empty_text_pages_unreferenced_by_othersPrint list of non-empty text pages not referenced by any other pages. Print one line per page with id and page title to standard output. The resulting output can be used as input for the util html|markup commands.
util non_empty_text_pages_without_references_to_othersPrint list of non-empty text pages without any references to other pages. Print one line per page with id and page title to standard output. The resulting output can be used as input for the util html|markup commands.
util pages_without_contentPrint list of pages without content. This are pages which have a title but haven't got content yet, not even content with size < 1. Print one line per page with id and page title to standard output. The resulting output can be used as input for the util html|markup commands.
util references_from_other_pages ?page <page-id>? ?pages <file>?Print references from other pages to specified pages. Print one line per page with id and page title to standard output. The resulting output can be used as input for the util html|markup commands.
util references_to_other_pages ?page <page-id>? ?pages <file>?Print references found in specified pages to other pages. Print one line per page with id and page title to standard output. The resulting output can be used as input for the util html|markup commands.
util sql sqlstmt <sql statement>Execute the SQL statement and print the results

Questions/remarks

MHo 28-mar-2008: Sorry. Just downloaded the latest wubwikit.kit, and tried tclkit and tclkitsh. Only the first page looks correct after navigating to http://localhost:8080 . Clickering every other link leads to an nearly empty page.... Tried local mode, too: clicking on the scrollbar again gives an error... jdc 04-apr-2008 I'm also seeing empty pages when I put the database and the kit in the same directory. I need to check what's going wrong there. Can you try with db and kit in different directories? About the local mode, did you add the toc wub command line options? jdc 03-jul-2008 Fixed to file location problem.


LV So, I download the wubwikit.kit from sf.net. I run it as listed above to start it as a Tk application. What I see is the error:

couldn't read file
"./wubwikit-20080303.kit/lib/wikitcl/wikit/db.tcl": no such file or directory
   while executing
 "source ./wubwikit-20080303.kit/lib/wikitcl/wikit/db.tcl"
        ("package ifneeded" script)
        invoked from within
   "package require Wikit::Db"
        (file "gui.tcl" line 17)
        invoked from within
    "source gui.tcl"
        invoked from within
      "if { $wub } {
        lappend auto_path [file join [file dirname [info script]] lib] [file join [file dirname [info script]] lib wikitcl]
        lappend arg..."
        (file "./wubwikit-20080303.kit/main.tcl" line 28)
           invoked from within
        source "./wubwikit-20080303.kit/main.tcl"
           ("uplevel" body line 1)
                invoked from within
                "uplevel [list source [file join $self main.tcl]]"

After poking at it a bit, it looks like the problem is that the application is changing directories and no longer seeing the virtual file system. So, I invoke things as:

tclkit /tmp/.lwv/wubwikit-20080303.kit wikidb /tmp/.lwv/wikit.tkd wub 0

and now the wikit comes up.


MHo 2008-03-03: WinXP. Local mode works, but clicking on the vertical scrollbar gives an error. WebServer mode: port 8082 offers some internas. Tk-window is empty. Closing Tk-windows leavs process running. Starting with tclkitsh:

error @@View: can't rename to "_Vvfield": command already exists (-code 1 -level 0 -errorcode NONE -
errorinfo {can't rename to "_Vvfield": command already exists\n    while executing\n"rename [mk::vie
w open $db.$view] $vcmd "\n    (procedure "::View::init" line 9)\n    invoked from within\n"View ini
t v$view db.$view"} -errorline 9)
error @@View: can't rename to "_Vvview": command already exists (-code 1 -level 0 -errorcode NONE -e
rrorinfo {can't rename to "_Vvview": command already exists\n    while executing\n"rename [mk::view
open $db.$view] $vcmd   "\n    (procedure "::View::init" line 9)\n    invoked from within\n"View ini
t v$view db.$view"} -errorline 9)
log @@RESTART: Mon Mar 03 23:13:32 +0100 2008 'couldn't read file "D:/Home/Arbeit1/pgm/tcl/usr/bin/t
clkit8.5/wubwikit-20080303.kit/lib/wikitcl/wubwikit/local.tcl": no such file or directory' (-code 1
-level 0 -errorcode {POSIX ENOENT {no such file or directory}} -errorinfo {couldn't read file "D:/Ho
me/Arbeit1/pgm/tcl/usr/bin/tclkit8.5/wubwikit-20080303.kit/lib/wikitcl/wubwikit/local.tcl": no such
file or directory\n    while executing\n"source [file join [file dirname [info script]] local.tcl]"}
 -errorline 1)
log @@Listening on http://matthias:8080/ using docroot /tmp/wiki/docroot

jdc 4-mar-2008: I tested Wub and wikitcl on windows, but not this kit. When started with tclkitsh in webserver mode, it works? You can ignore the errors listed above. I'll fix the couldn't read file error, the others are related to the dub mode of Wub.

About local-mode and the scroll-bar, that's a difference in TOC handling between the webserver and local mode. To have a TOC in local mode, you need to copy the contents of the TOC file (you can find that in the kit) into page 8 of the data base.

LV 2008 March 12: jos, I'm seeing the scrollbar problem in local mode on Solaris as well. Is there a way that, in local mode, the wiki local code could just do the copy itself? Otherwise, we have to make this change each week. jdc I'll check how i can automate this. jdc 2008-mar-13 Added possibility to set TOC.


LV 2008 March 12 Just wondering - is the .tkd update process automated or manual? That web page that you point to above for wubwikit shows a March 3rd update. But there used to be a page somewhere on sf.net where the tkd file was being updated once a week. I didn't know if that weekly update was a cronjob, or just a manual process. jdc it's manual and I don't find the time every week to copy the db over to SF.


LV jos, what is the ramification of sdx mkpack for someone using the tkd file? jdc sdx mkpack removes free space from a Metakit file, a starkit or a starpack. As such, the wiki database gets 40M smaller.


CMcC is thinking about a special purpose application which updates a local wikit.db by reference to the current wiki. It would be a fairly simple modification of the Recent Pages algorithm. I don't think I'd be bothered updating History. Thoughts anyone?


LV Does local mode use of wubwikit maintain a history? Display a history? jdc Using wubwikit with Tk interface does maintain the history but can't show it. Using wubwikit as webserver has the same features as on-line one.


KJN Many thanks for this - it is very useful to have an offline copy of the Wiki as a reference when away from an internet connection. For that purpose, 95% of the value comes from having just the latest version of each page, and I would not care if the full history was not supplied.


WJP 2008-04-05: A crucial piece of information is missing from this page: how to use the data file. apw describes on the Wub pages how to use the database offline.

LV The .tkd file is read by the wiki software itself, by wikitool, and possibly some other tools.

In the old days, you would just say something like:

 wikit.kit wikit.tkd

and a desktop version of the wiki would come up on your desktop.


MHo 2008-11-07: Just downloaded the latest wubwikit and database from the above link, done a few tests on WinXP:

  • In local tk mode, clicking on the vertical scrollbar still gives an error... that's not the case when toc wub is specified
    • jdc 7-nov-2008, Wub and Tk mode handle the TOc differently. When running in Tk mode, you have to add your TOC to page 8 or use the toc wub option.
    • gasty After unwrap the kit and edit a little the proc yscroll in file wubwikit20090319.vfs\lib\wikitcl\wikit\gui.tcl, I can run the wiki without toc wub and the scroll ploblem dissapear. jdc 9-jul-2009 Fix is part of latest release. Thanks!
        proc yscroll {args} {
            variable D
            variable toc
            eval $D yview $args
            if {$toc ne ""} {
              eval $toc yview $args
            }
        }
  • tk load lasts several (!) seconds, main window does not come to foreground (I use focus -force . one windows regulary)
    • jdc 7-nov-2008, also when you just start-n-wait?
  • seems that font_ options are ignored
    • jdc 7-nov-2008, could trace this bug, will be solved in next release.
      • MHo 7-nov-2008: solved with new version
  • window resizing leads to ugly results, a purple block appears between toc and content
    • jdc 7-nov-2008, could trace this bug, will be solved in next release.
      • MHo 7-nov-2008: happens no more with new version
  • window width is flipping depending on the page
    • jdc 7-nov-2008, can you give some examples?
  • Webserver mode: seems I had to request the starting page twice before it is served
    • jdc 7-nov-2008, couldn't see this problem ,but I've only been testing on Linux yet
      • MHo 7-nov-2008: tested with new version, again had to reload (only the first page)
        • jdc 7-nov-2008 odd; I'll try to test on XP.
  • Q: Where can I specify the server-port?
    • jdc 7-nov-2008, you can't with the current version; I'll add this option.
  • Q: How can I gracefully terminate the server?
    • jdc 7-nov-2008, use these commands (I will also make the telnet port configurable):
      % telnet localhost 8082
      ::exit
  • Webserver says on console:
error @@View: can't rename to "_Vvfield": command already exists (-code 1 -level 0 -errorcode NONE -
errorinfo {can't rename to "_Vvfield": command already exists\n    while executing\n"rename [mk::vie
w open $db.$view] $vcmd "\n    (procedure "::View::init" line 9)\n    invoked from within\n"View ini
t v$view db.$view"} -errorline 9)
error @@View: can't rename to "_Vvview": command already exists (-code 1 -level 0 -errorcode NONE -e
rrorinfo {can't rename to "_Vvview": command already exists\n    while executing\n"rename [mk::view
open $db.$view] $vcmd   "\n    (procedure "::View::init" line 9)\n    invoked from within\n"View ini
t v$view db.$view"} -errorline 9)
log @@RESTART: Fri Nov 07 01:27:33 +0100 2008 '' (-code 0 -level 0)
log @@Listening on http://matthias:8080/ using docroot /tmp/wiki/docroot
log @@Site LOCAL: '' (-code 0 -level 0)
    • jdc 7-nov-2008, you can safely ignore these error messages.

(jdc 7-nov-2008 I'll look into these issue in the next couple of days, thanks for reporting)

MHo 7-nov-2008: Another question: the whole *.tkd is copied to the temp directory where wub additionally creates some other files (*.css, etc.). What *.tkd is altered afterwords - the original one, or the copy? Why is the whole file copied?

    • jdc 7-nov-2008, The copy will be modified, I'll check if it's possible to work with the specified file.
    • jdc 19-feb-2009 The latest kit no longer makes copies and the database must be specified at the command line.

RZ Is there a reason why there are now two different locations, one for the database and one for the kit? Having both pieces in one place is at least for me more convenient. jdc 19-feb-2009 I find the repository at google code more convenient than the one at source forge. I'll provide the kits at both places. This first release on google code needs more testing and needs Tcl 8.6 (including co-routines and TclOO) which is not widespread yet at this moment.


gasty - 2009-09-29 00:04:58

Hi. In my work we start using wubwikit to make an internal wiki. It's amazing! We wanted a way to reference images and other files stored in the server's filesystem within wiki pages.

After read some wub documentation and make a little of reverse engineering of wikit, I add the following (little) patches to a couple of files and works very well:

  • In "wubwikit20090806.vfs/lib/wikitcl/wubwikit/wikit.nub" I add this line to create a new domain:
 domain /_mydomain/ {File mydomain} root <path-to-files> expires "next week"
  • In "wubwikit20090806.vfs/lib/wikitcl/wikit/format.tcl" I add some code starting from line 1590:
  # link of the form [file:*] are interpreted as local files
  lassign [split $text ":"] type newText
  if {$type eq "file"} {
    set text _mydomain/$newText
  }

between

x {

and

  lassign [split_url_link_text $text] link text

Now we can put links to files using [file:<filename>] . If the file has extension gif, png, jpeg or jpg, then the image are showed inlined.

The directory <path-to-files> can be a "shared folder", so we can copy files to it without the need to access physically to the server.

Files within sub-directories of <path-to-files> can be accessed as [file:subdir/<filename>] , wich are useful to organize the <path-to-files> content.

I hope this can be useful to other wubwikit users!


zz When I use kit file, searching a word will error, and I use wub2012.vfs/main.tcl, it is normal when I search. Why?

f:\wikit>tclkit sdx.kit wrap wub2012.kit
f:\wikit>tclkit wub2012.kit wikidb 999.tkd
....
....
....
....
Error from thread tid0000012C
can't find package Http
    while executing
"package require Http"
1342704805530-179637mS log @@{Bogus watchdog over -1}

jdc Must be something wrong with the auto_path to look up packages in the kit version. I always run without wrapping up in a kit.


How to create and clean up a Wikit .tkd file