Wikit's performance

January 11, 2003 - The BiggestWiki page [L1 ] was updated, but how can one get an updated count of the number of pages in this wiki?

Not easily... I've looked at it with MK database cmds: 6253 page entries, of which 2595 are empty (never filled in) -jcw

June 20, 2002 - obsolete and confusing info from this page has been removed -jcw

Performance of wiki markup rendering used to be pretty bad (see comment further down). The kiwi project includes a new renderer which is probably 10x faster (but only HTML so far, no Tk version).

It's all largely irrelevant for CGI use, now that wiki uses a static page cache. Things are more than fast enough.

Having just converted to new hyperlink tracking code in wikit, I decided to collect a bit of stats. First the output (updated Nov 15, 2002):

 4605 pages
 1476 pages have never been filled in
 9794 archive entries
 16487 hyperlinks
 2782 pages have hyperlinks to others
 2838 pages are being hyperlinked from others
 260 leaves (pages not referencing any others wiki pages):

12 33 45 80 94 98 110 112 114 115 117 119 120 124 126 130 133 134 135 142 143 157 158 160 169 186 195 205 287 288 289 292 293 306 307 310 311 315 316 318 319 326 330 331 352 368 370 372 374 381 408 416 422 424 426 429 431 433 474 475 476 558 568 570 574 575 589 616 629 636 651 670 673 675 678 679 680 681 710 716 730 762 767 785 790 811 816 832 841 842 858 865 896 905 928 939 1087 1095 1098 1135 1142 1144 1172 1173 1174 1190 1192 1201 1210 1219 1240 1244 1265 1285 1289 1342 1350 1357 1359 1521 1528 1541 1610 1629 1634 1641 1649 1654 1747 1804 1826 1828 1833 1885 1891 2040 2059 2063 2111 2125 2127 2128 2132 2134 2137 2152 2156 2168 2194 2247 2250 2346 2377 2386 2387 2407 2416 2458 2497 2515 2535 2539 2558 2606 2661 2701 2754 2815 2861 2939 2940 2975 3001 3019 3033 3050 3064 3066 3103 3115 3122 3182 3288 3289 3294 3317 3405 3406 3435 3438 3442 3449 3472 3473 3476 3491 3536 3543 3545 3585 3664 3671 3686 3703 3762 3763 3828 3843 3872 3877 3880 3881 3898 3965 3975 3983 4018 4031 4034 4040 4041 4045 4065 4069 4076 4096 4134 4140 4148 4192 4208 4238 4277 4301 4310 4311 4314 4315 4316 4318 4320 4322 4341 4342 4347 4368 4371 4382 4392 4431 4450 4469 4480 4481 4482 4499 4500 4501 4502 4540

 281 orphans (pages not referenced from any others):

23 65 151 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 328 329 384 385 386 387 388 389 390 394 404 412 415 419 432 438 439 449 452 455 456 464 465 466 467 520 521 522 536 538 545 554 572 577 578 579 580 581 582 597 598 599 601 605 641 655 658 664 665 695 700 705 725 726 742 745 746 753 754 789 798 802 806 807 810 817 818 845 854 862 867 868 892 893 899 914 925 944 965 1005 1068 1076 1131 1136 1138 1139 1160 1169 1209 1216 1221 1226 1229 1231 1238 1250 1257 1271 1282 1284 1288 1311 1319 1320 1338 1341 1349 1352 1370 1371 1376 1377 1382 1523 1524 1676 1706 1768 1774 1795 1839 1843 1858 1876 1907 1920 1929 1930 2121 2182 2184 2209 2213 2225 2388 2410 2412 2427 2438 2454 2474 2481 2505 2511 2521 2522 2526 2534 2568 2646 2654 2669 2700 2733 2742 2764 2800 2840 2847 2873 2880 2934 2938 2972 2998 3013 3017 3029 3045 3046 3049 3059 3072 3073 3088 3101 3136 3147 3160 3165 3168 3185 3196 3198 3207 3213 3214 3230 3231 3246 3247 3262 3273 3285 3297 3310 3314 3328 3329 3335 3339 3357 3391 3398 3400 3402 3434 3436 3437 3469 3470 3481 3492 3504 3587 3590 3642 3658 3693 3695 3696 3718 3719 3725 3727 3775 3799 3806 3816 3820 3995 3999 4008 4033 4092 4160 4598

The code that generated the above:

 #! /usr/bin/env tclkit

 mk::file open db wikit.tkd -readonly

 puts " [mk::view size db.pages] pages"
 puts " [llength [mk::select db.pages date 0]] pages have never been filled in"
 puts " [mk::view size db.archive] archive entries"
 puts " [mk::view size db.refs] hyperlinks"

 mk::loop c db.refs {
   lassign [mk::get $c from to] f t
   if {$f < 10 || $t < 10 || [mk::get db.pages!$t date] == 0} continue
   if {![info exists froms($f)]} {
     set froms($f) 0
   }
   incr froms($f)
   if {![info exists tos($t)]} {
     set tos($t) 0
   }
   incr tos($t)
 }

 puts " [array size froms] pages have hyperlinks to others"
 puts " [array size tos] pages are being hyperlinked from others"

 set leaves {}
 foreach {k v} [array get tos] {
   if {![info exists froms($k)]} {
     lappend leaves $k
   }
 }

 puts " [llength $leaves] leaves (pages not referencing any others wiki pages):"
 puts [join [lsort -integer $leaves] "\t "]

 set orphans {}
 mk::loop c db.pages 10 {
   set i [mk::cursor pos c]
   if {[mk::get $c date] != 0 && ![info exists tos($i)]} {
     lappend orphans $i
   }
 }

 puts " [llength $orphans] orphans (pages not referenced from any others):"
 puts [join [lsort -integer $orphans] "\t "]

06oct02 jcw - Wikit has been modified to enhance its performance when rendering pages with lots of embedded page references, including pages such as Recent Changes . This has a more-than-order-of-magnitude effect. This is especially noticeable in local mode, but CGI also benefits.

escargo - 19 Nov 2002: Are there any built-in functions for trimming the unfilled pages? To me it seems like many of them must have been created by accident, probably by new people who created pages with typographical errors in their names. I would think that a mark/sweep or generational approach, marking pages as dead in one phase, and then reaping them in a later phase, would be both straight forward and useful.

A function to generate a list of all [links] that point to completely blank pages might be nice as well.

Sure ... <insert sound of flexing fingers> ... here ya go!

 mk::file open db wikit.tkd -readonly
 puts [mk::select db.pages page ""]

Remember, though, they aren't necessarily candidates for immediate deletion - the refs view has to be sure not to contain any links to those pages first.

jdc 28-apr-2008

Running the above script (archive lines remove) on a recent wikit.tkd results in the following:

21023 pages

10806 pages have never been filled in

83414 hyperlinks

9549 pages have hyperlinks to others

9522 pages are being hyperlinked from others

258 leaves (pages not referencing any others wiki pages):

363 orphans (pages not referenced from any others):

  • Programming Oracle stored-procedure cursors [L260 ]
  • Help! My Tcl application just crashed! [L261 ]
  • COMPANY: The Tolis Group [L262 ]
  • COMPANY: TiVo Inc. [L263 ]
  • BIG SCIENCE: Jet Propulsion Laboratory [L264 ]
  • COMPANY: Software AG [L265 ]
  • GOVERNMENT: USPS [L266 ]
  • Bignums Fast and Dirty [L267 ]
  • tkbusy [L268 ]
  • XPS [L269 ]
  • TesteDesMotsFran�ais [L270 ]
  • BOOK: Tcl/Tk: A Developer's Guide [L271 ]
  • obfusc [L272 ]
  • fileinput [L273 ]
  • TclVSForm [L274 ]
  • أَهْﻻً وَ سَهْﻻً [L275 ]
  • <u>I really shouldn't be able to do this.</u> [L276 ]
  • COMPANY: GISTEK ( [L277 ]
  • color [L278 ]
  • TL Wonder [L279 ]
  • CFlexText [L280 ]
  • COMPANY: Muonics [L281 ]
  • Moving Canvas Window Items - Can someone tell me what's going on here? [L282 ]
  • Comments inside Exec [L283 ]
  • Timple SQL [L284 ]
  • Scripting or how to write a RAD tool in a couple of hours [L285 ]
  • The TclLib functions [L286 ]
  • Vapourware [L287 ]
  • Mimers brunn - tools for translators [L288 ]
  • conv [L289 ]
  • reading win32 exectuable version information [L290 ]
  • TerraVision [L291 ]
  • Determining precision of numbers [L292 ]
  • new page SQLServlet [L293 ]
  • SQLServlet [L294 ]
  • Learn programming to children [L295 ]
  • A needlmeter widget: type 1 [L296 ]
  • MyFavoritePage [L297 ]
  • Anjunta [L298 ]
  • Virtual Hosts in Tclhttpd [L299 ]
  • SMIL [L300 ]
  • Widget catalog [L301 ]
  • interval arithmic [L302 ]
  • tabula rasa [L303 ]
  • XML Query [L304 ]
  • Malboge [L305 ]
  • Notebook App - User Code Snippets [L306 ]
  • Open Source Weanie [L307 ]
  • digfe - a free GUI front end for the DNS client program DIG [L308 ]
  • TestJe [L309 ]
  • sticky note [L310 ]
  • stippling [L311 ]
  • Inline::C for Tcl [L312 ]
  • Alistair Grant [L313 ]
  • Widget Destruction [L314 ]
  • TWINE [L315 ]
  • tclnettle [L316 ]
  • Blurring [L317 ]
  • CAT [L318 ]
  • Command syntax definitions [L319 ]
  • Roaring Penguin [L320 ]
  • punycode [L321 ]
  • dali [L322 ]
  • Skinable interfaces [L323 ]
  • GO startup [L324 ]
  • parse_cmd_line [L325 ]
  • divider [L326 ]
  • get_width [L327 ]
  • parse_command_line [L328 ]
  • getpass [L329 ]
  • question [L330 ]
  • putc [L331 ]
  • Specials [L332 ]
  • XLosung [L333 ]
  • Flower Buttons in a canvas [L334 ]
  • Joysticks on Windows [L335 ]
  • duh [L336 ]
  • curses text screen saver [L337 ]
  • Favorite menu [L338 ]
  • doctool [L339 ]
  • Cisco IOS Scripting with Tcl [L340 ]
  • Starkits Repository [L341 ]
  • Adolf [L342 ]
  • cfk [L343 ]
  • mktclapps [L344 ]
  • Appropriate Tcl Chatroom Discussions [L345 ]
  • Dynamic procs as performance tools [L346 ]
  • WikiLinks [L347 ]
  • José E. Marchesi [L348 ]
  • testing umlauts: äöüßÄÖÜ [L349 ]
  • unperson Problem [L350 ]
  • George Peter Staplin Problem [L351 ]
  • People and community [L352 ]
  • Rules and rights for this site [L353 ]
  • SetProcTitle [L354 ]
  • Marcus O'Malley [L355 ]
  • Julian Marcel Noble Problem History [L356 ]
  • Here is the situation JCW [L357 ]
  • Jamshed [L358 ]
  • Basic TCL Programs [L359 ]
  • Using multiple images in a single canvas [L360 ]
  • Pretty checkbuttons [L361 ]
  • How to compile into an exe program [L362 ]
  • Showcasing TCL Programs [L363 ]
  • XLST [L364 ]
  • Man Tcl по-русски [L365 ]
  • Detecting Hybernation [L366 ]
  • Matthias Hoffmann - Tclhttpd [L367 ]
  • עברית [L368 ]
  • A wiki running locally with a browser and a server [L369 ]
  • Remainder rings [L370 ]
  • TreeCL [L371 ]
  • Vector3d [L372 ]
  • Overlap polygons [L373 ]
  • WikiNode [L374 ]
  • a chroot interpreter [L375 ]
  • menubar and menubutton [L376 ]
  • Tracing a script's execution [L377 ]
  • the 'Freiburg' project [L378 ]
  • Tcl Command Evaluation: Layer 0: Tcl_ObjCmdProc call [L379 ]
  • Xadsen [L380 ]
  • XOTclIdeWishesh [L381 ]
  • Ebay with Tcl [L382 ]
  • Tcl as a markup language [L383 ]
  • the 'Freiburg' project port to windows [L384 ]
  • Falun Gong [L385 ]
  • wanyozxy [L386 ]
  • Fidessa [L387 ]
  • Jonathon Merz [L388 ]
  • Tcl-based scientific and engineering applications [L389 ]
  • Another Tcl Logger Proc [L390 ]
  • tclFreiburg [L391 ]
  • Touchpad Binding for Horizontal Scrolling [L392 ]
  • net-snmp [L393 ]
  • DiaNa [L394 ]
  • Work on the "Answered questions on ..." pages [L395 ]
  • Answered Questions to be classified [L396 ]
  • Answered Questions on: Template [L397 ]
  • Tcl trivia [L398 ]
  • FAQ: HTTP/FTP etc [L399 ]
  • Mc Cormack-unperson exchange re: Answered questions on pages [L400 ]
  • McCormack's objections to the Answered Questions on system [L401 ]
  • McCormack's proposal for a new "Answered Questions on" System [L402 ]
  • Proof of concept: Detecting the sort of files in an Oracle Console [L403 ]
  • A possible system for Ask and you shall be given questions [L404 ]
  • Windows XP SP2 Breakage [L405 ]
  • Ask and it shall be given Survey [L406 ]
  • The Wit and Wisdom of unperson [L407 ]
  • The conclusions of the vote and the next step [L408 ]
  • Thread Logger [L409 ]
  • Widgets Q&A [L410 ]
  • Zach Peters [L411 ]
  • zach [L412 ]
  • Real Time control of a sound program [L413 ]
  • pkgconfig [L414 ]
  • wm toplevel [L415 ]
  • Restored Tcl/TkAqua FAQ [L416 ]
  • Telemaco [L417 ]
  • The Garage with the Pipe Wrench [L418 ]
  • Programmer of the Month [L419 ]
  • Tcl vs POSIX - what functionality is missing [L420 ]
  • A method of classification for Require for help items [L421 ]
  • Tk resource pickers [L422 ]
  • Groovy [L423 ]
  • menu accelator utility seed code [L424 ]
  • klik [L425 ]
  • Smartbar [L426 ]
  • MajaMaja [L427 ]
  • Web Publishing [L428 ]
  • SCOOP [L429 ]
  • Free Credit Report - Credit Score [L430 ]
  • Univert [L431 ]
  • I am already 84 years old [L432 ]
  • Useful Basic TCL Formulas [L433 ]
  • Sixth European Tc/Tk Users Meeting [L434 ]
  • 100 useful basic TCL/TK procedures [L435 ]
  • 100 useful basic TCL/TK commands 1-30 [L436 ]
  • Diffs algorithm [L437 ]
  • The TCL/TK Games CD Project [L438 ]
  • 100 useful basic TCL/TK commands for simple editors 1-30 [L439 ]
  • The legal and ethical aspects of TCL/TK [L440 ]
  • TCL/TK Games CD Project: Marketing feasibility Study [L441 ]
  • TCL/TK Games CD Project [L442 ]
  • The TCL/TK Games CD Project: Marketing feasibility Study [L443 ]
  • TCL/TK Coders' Coop [L444 ]
  • The great TCL/TK Sabotage Operation [L445 ]
  • BWidget::ScrollableFrame [L446 ]
  • dxcClips [L447 ]
  • dxcChanges [L448 ]
  • tcl-dbi [L449 ]
  • Why Tcl Still Matter [L450 ]
  • Returning localized string to interpreter from Tcl C extension [L451 ]
  • SQLite and Tcl [L452 ]
  • How Tcl differs between Windows and Unix [L453 ]
  • Rules for the names of variables [L454 ]
  • When is a good time to use eval? [L455 ]
  • Course Forum [L456 ]
  • Modern alternatives to "big tclsh" and "big wish" [L457 ]
  • Keith W Johnson [L458 ]
  • ICFP contest [L459 ]
  • database design [L460 ]
  • TkOGL OpenGL widget [L461 ]
  • uDraw(Graph) [L462 ]
  • Replicated and distrubated slave interpreters [L463 ]
  • ActiveSlide [L464 ]
  • TkKasse [L465 ]
  • glossator [L466 ]
  • Peer to peer [L467 ]
  • MaxOS X shorcuts [L468 ]
  • MacOS X shorcuts [L469 ]
  • Duro a relational database library [L470 ]
  • ESSAY: Ten digit algorithms [L471 ]
  • txt2regex [L472 ]
  • Integer (Computer Science) [L473 ]
  • BWidget::ComboBox [L474 ]
  • Scale widget [L475 ]
  • GHL [L476 ]
  • Subtext [L477 ]
  • string map {"%" "%%"} $msg [L478 ]
  • Of spreads and quadrances [L479 ]
  • WDDX [L480 ]
  • Tcl-Coach [L481 ]
  • Polygon Puzzle [L482 ]
  • min's text (mtext) : Improved text widget [L483 ]
  • TeaTime [L484 ]
  • notepad++ [L485 ]
  • Restart Windows OS [L486 ]
  • Uptime in Windows [L487 ]
  • The Boustrophedon Text Reader [L488 ]
  • Frag [L489 ]
  • I8N [L490 ]
  • Heap Pirority Queue [L491 ]
  • Dil-new [L492 ]
  • Minimizing file names [L493 ]
  • testrgomez [L494 ]
  • Hot Drinks [L495 ]
  • How do I author a new widget in C? [L496 ]
  • MAS-UAC Fallback CWP [L497 ]
  • Tux Racer [L498 ]
  • tkfichier [L499 ]
  • iTickle [L500 ]
  • HP-15 Simulation [L501 ]
  • Get the color of the pixel under the cursor [L502 ]
  • Tcl Wiki Server [L503 ]
  • canonical way of site configuration [L504 ]
  • Adrian Davis [L505 ]
  • Libes' [L506 ]
  • Don Libes' [L507 ]
  • Works with any Web Server [L508 ]
  • Runs on most platforms [L509 ]
  • State and Session Management [L510 ]
  • URL Manager [L511 ]
  • Ready-to-use Components. TabNotebooks, Vertical & Horizontal Menus, etc. [L512 ]
  • [email protected] [L513 ]
  • Form widget [L514 ]
  • Self Documented Online Library & Help [L515 ]
  • Multi-Language [L516 ]
  • Remote Editing [L517 ]
  • Persistent Data Storage, including states [L518 ]
  • Toolkit Architecture [L519 ]
  • and much much more... [L520 ]
  • Reading COM & stdin [L521 ]
  • link? [L522 ]
  • Wolf-Dieter Busch [L523 ]
  • Essay on Scheme [L524 ]
  • Essay on Tcl/Tk [L525 ]
  • amk [L526 ]
  • Package feature map [L527 ]
  • careck [L528 ]
  • vhe [L529 ]
  • TTS [L530 ]
  • dinbrief [L531 ]
  • Slate megawidget [L532 ]
  • The refactoring of the Ask and it shall be given pages [L533 ]
  • fullscreener [L534 ]
  • Yet again dll caler [L535 ]
  • xtt: XML <--> Text Tag translator [L536 ]
  • Why is a homemade database the best database? [L537 ]
  • TotalView [L538 ]
  • XRC [L539 ]
  • CrowAMP [L540 ]
  • Wiki-Errors [L541 ]
  • Don Libes' cgi.tcl run in a Slave Interpreter with TclHttpd [L542 ]
  • TorXviz [L543 ]
  • disable windows error report [L544 ]
  • finomation [L545 ]
  • Phylographer [L546 ]
  • WyattERP [L547 ]
  • Tk9 [L548 ]
  • zorro [L549 ]
  • zipvfs shows empty dir with special crafted archives [L550 ]
  • Tcl 8.5 To-do List [L551 ]
  • Zorgan [L552 ]
  • determinante [L553 ]
  • VCRI [L554 ]
  • Seeker: A Case Tool Based on OO Sequence Diagrams [L555 ]
  • Xinu [L556 ]
  • EasyNumerics [L557 ]
  • Nam: Network Animator [L558 ]
  • Zebra [L559 ]
  • Anastasia [L560 ]
  • MultiMarkdown [L561 ]
  • ReLaTe [L562 ]
  • Find all words [L563 ]
  • SLIDE: Scene Language for Interactive Dynamic Environments [L564 ]
  • XMI2TXT [L565 ]
  • Does my hardware controller need to use threads? [L566 ]
  • failsh [L567 ]
  • TACOS [L568 ]
  • Open Instrumentation Project [L569 ]
  • Dan Eyre [L570 ]
  • Ask, and it shall be given -Index- [L571 ]
  • Spam by 57.87.2.189 [L572 ]
  • SplitIt [L573 ]
  • Humor as seen by Wikit contributors [L574 ]
  • waydoctor [L575 ]
  • Learning through examples [L576 ]
  • BOOK: Reliable Distributed Systems: Technologies, Web Services, And Applications [L577 ]
  • mathfunc [L578 ]
  • SQLite Optimization [L579 ]
  • Applications in Tcl and Tcl/Tk -Text Processing- [L580 ]
  • Cell Messenger [L581 ]
  • Minpair [L582 ]
  • Word Generator [L583 ]
  • Oklin PAM [L584 ]
  • Tcl Ticker [L585 ]
  • Axis [L586 ]
  • Blueline [L587 ]
  • TkRat (Ratatosk) [L588 ]
  • Arclabel [L589 ]
  • Alana [L590 ]
  • Audela [L591 ]
  • gpsfeed+ [L592 ]
  • tkxmllint [L593 ]
  • ProjMan [L594 ]
  • Multivariate Linear Regression [L595 ]
  • WebApp [L596 ]
  • Spreadheets in Tcl-Tk [L597 ]
  • exe [L598 ]
  • Markup test page [L599 ]
  • Open letter to unperson about reorganizing the Tcler's wiki [L600 ]
  • René Zaumseil [L601 ]
  • Tao Data Encapsulation [L602 ]
  • wolfw [L603 ]
  • here-document [L604 ]
  • Citation needed [L605 ]
  • Getting help [L606 ]
  • Learning Tcl [L607 ]
  • The Tcl Dev Xchange [L608 ]
  • Tcl roadmap [L609 ]
  • Applications [L610 ]
  • The Tcl Core [L611 ]
  • Development Tools [L612 ]
  • Toys and Games [L613 ]
  • Major applications [L614 ]
  • Businesses [L615 ]
  • Education [L616 ]
  • Engineering [L617 ]
  • Government [L618 ]
  • Other Industries [L619 ]
  • Tcl in Industry [L620 ]
  • Medical [L621 ]
Category WikitCategory Tcler's WikiCategory Performance
Updated 2014-04-10 19:16:05