Error processing request

Parameters

Body

Error

Unknow state transition: LINE -> END

-code

1

-level

0

-errorstack

INNER {returnImm {Unknow state transition: LINE -> END} {}} CALL {my render_wikit {suspend and resume} \[Lars\ H\],\ 2008-08-27:\nThe\ following\ was\ first\ posted\ to\ the\ tcl-core\ mailing\ list,\ as\ a\ kind\ of\ ''well\ the\ \[NRE\]\ \[coroutine\]s\ may\ be\ nice\ but\ it\ might\ actually\ be\ possible\ to\ do\ something\ even\ more\ powerful…''\n\nThe\ main\ reason\ I'm\ putting\ it\ here\ now\ is\ that\ I\ actually\ managed\ to\ produce\ a\ very\ elementary\ \[proof\ of\ concept:\ suspend\ and\ resume\]\ in\ pure\ Tcl.\n\n\n**The\ specification**\n\nJust\ the\ plain\ text,\ for\ now.\ Should\ wikify\ it,\ though.\n\nBASIC\ PREMISE\n\nPrimarily\ I\ wanted\ a\ non-nesting\ \[vwait\]\ --\ one\ which\ would\ not\ hang\ in\n\nafter\ 1000\ \{\n\ \ \ after\ 1000\ \{set\ ::a\ 1\;\ vwait\ ::b\;\ puts\ \"Finished\"\}\n\ \ \ vwait\ ::a\n\ \ \ set\ ::b\ 1\n\}\n\nThis\ meant\ the\ \[vwait\]\ would\ somehow\ have\ to\ save\ away\ what\ was\ currently\ in\ the\ C\ call\ stack,\ drop\ out\ into\ the\ outer\ event\ loop\ and\ process\ events\ there\ for\ a\ while,\ until\ the\ variable\ has\ been\ set,\ at\ which\ point\ everything\ would\ have\ to\ be\ put\ back\ so\ that\ processing\ could\ continue.\n\nAlso,\ this\ would\ have\ to\ be\ accomplished\ \"within\ the\ box\"\ --\ no\ fancy\ C\ compiler\ or\ OS\ features\ could\ be\ relied\ on,\ and\ recursive\ calls\ should\ still\ be\ supported.\n\n\nSOLUTION\n\nSince\ there's\ no\ safe\ way\ to\ operate\ on\ the\ C\ stack\ from\ outside,\ the\ only\ way\ to\ get\ it\ out\ of\ the\ way\ is\ to\ explicitly\ unwind\ it\ --\ make\ every\ C\ function\ in\ there\ return\ to\ its\ caller.\ That's\ not\ too\ different\ from\ having\ an\ error\ propagate\ down\ the\ call\ stack,\ so:\n\nYielding\ is\ done\ by\ returning\ with\ a\ new\ \[return\]\ code,\nTCL_SUSPEND,\ say.\n\nWhen\ a\ yielding\ unwind\ is\ in\ progress,\ the\ interpreter\ result\nis\ a\ list.\ Each\ entity\ in\ a\ call\ chain\ receiving\ a\ TCL_SUSPEND\nresult\ code\ from\ a\ call\ it\ made\ must\ do\ one\ of\ the\ following:\n\n1.\ Append\ to\ the\ interp\ result\ a\ list\ element\ with\ sufficient\ information\ that\ it\ can\ later\ reconstruct\ the\ exact\ state\ it\ was\ in\ immediately\ before\ receiving\ the\ TCL_SUSPEND,\n\ \ \ \ \ and\ thus\ resume\ processing\ from\ where\ it\ was\ suspended.\ Then\ return\ to\ caller\ with\ TCL_SUSPEND\ code.\n\n2.\ Throw\ an\ error\ \"Cannot\ be\ suspended\".\n\n3.\ Catch\ the\ yield,\ if\ the\ entity\ is\ prepared\ to\ handle\ it.\n\nNormally,\ a\ yield\ would\ proceed\ all\ the\ way\ down\ to\ the\ main\ event\ loop,\ where\ it\ gets\ handed\ off\ to\ \[bgerror\]\ (or\ the\ like),\ and\ that\ command\ is\ then\ responsible\ for\ arranging\ so\ that\ the\ suspended\ calls\ are\ resumed\ at\ a\ suitable\ point\ in\ the\ future.\n\nA\ tricky\ detail\ is\ that\ there\ would\ now\ have\ to\ be\ two\ entry\ points\ for\ the\ C\ implementation\ of\ each\ Tcl\ command:\ one\ which\ is\ used\ when\ calling\ it\ normally,\ and\ one\ which\ is\ used\ when\ resuming\ it.\ I'll\ return\ to\ that\ matter\ below,\ but\ for\ now,\ let's\ look\ at\ what\ the\ script\ level\ behaviour\ is\ supposed\ to\ be.\n\n\nTCL\ API\n\nI\ imagined\ the\ Tcl\ side\ of\ this\ could\ be\ handled\ with\ only\ two\ commands\ (although\ \[catch\]\ and\ \[return\]\ would\ probably\ figure\ as\ prominently\ as\ with\ any\ new\ control\ structure):\n\n\ \ suspend\ ?\$arg\ ...?\n\ \ resume\ \$unwinding\ ?-code\ \$code?\ ?-level\ \$level?\ ?...?\ ?\$result?\n\n\[suspend\]\ stops\ execution\ at\ some\ point\ and\ causes\ the\ call\ stack\ to\ unwind,\ whereas\ \[resume\]\ resumes\ it.\ If\ you\ do\n\n\ \ proc\ suspending\ \{args\}\ \{list\ \[suspend\ \{*\}\$args\]\ \"in\ suspending\"\}\n\ \ catch\ \{suspending\ foo\ bar\}\ res\n\nthen\ the\ result\ will\ be\ TCL_SUSPEND\ and\ \$res\ will\ be\ an\ unwinding\ that\ contains\ the\ arguments\ of\ the\ \[suspend\]\ in\ its\ 0th\ element,\ i.e.,\ it\ looks\ like\n\n\ \ \{foo\ bar\}\ \{...\n\nwhere\ the\ ...\ denotes\ the\ beginning\ of\ the\ serialization\ of\ the\ internal\ state\ of\ the\ call\ stack\ entity\ corresponding\ to\ the\ \[suspending\]\ procedure\ (yes,\ that\ is\ potentially\ quite\ a\ mouthful).\n\nThe\ \[resume\]\ arguments\ following\ the\ \$unwinding\ are\ supposed\ to\ be\ the\ same\ as\ for\ \[return\],\ but\ control\ how\ the\ original\ \[suspend\]\ behaves\ when\ resumed.\ Thus\ if\ one\ then\ does\n\n\ \ resume\ \$res\ \"Return\ from\ with\"\n\nthen\ the\ result\ will\ be\ the\ list\n\n\ \ \{Return\ from\ with\}\ \{in\ suspending\}\n\nbecause\ \[resume\]\ put\ back\ the\ \[suspending\]\ proc\ and\ the\ \[suspend\]\ command,\ the\ latter\ returns\ with\ TCL_OK\ (since\ there\ was\ no\ other\ -code\ specified),\ the\ \[list\]\ in\ suspended\ is\ called\ as\n\n\ \ list\ \{Return\ from\ with\}\ \"in\ suspending\"\n\nand\ the\ result\ of\ that\ becomes\ the\ result\ of\ \[suspending\]\ as\ a\ whole,\ which\ is\ then\ also\ the\ result\ of\ the\ \[resume\].\n\n\nC\ API\n\n(Not\ my\ strong\ side,\ but\ one\ that\ has\ to\ be\ addressed.)\ Since\ the\ idea\ is\ mostly\ to\ give\ new\ meaning\ to\ things\ that\ can\ already\ happen,\ there\ isn't\ /that/\ much\ that\ has\ to\ be\ added\ on\ the\ C\ side,\ but\ one\ major\ thing\ is\ the\ prototype\ for\ the\ function\ that\ is\ called\ when\ a\ command\ is\ resumed\;\ let's\ call\ this\ a\ resumeProc:\n\ntypedef\ int\ Tcl_ObjResumeProc(\n\tClientData\ \ \ \ \ \ clientData,\n\tTcl_Interp\ \ \ \ \ *interp,\n\tint\t\ \ \ \ \ objc,\n\tTcl_Obj\ *const\ \ objv\[\],\n\tint\t\ \ \ \ \ resumeIdx,\n\tTcl_Obj\ *const\ \ resumeData\[\]\ \ )\;\n\nThe\ idea\ here\ is\ that\ the\ resumeData\ should\ be\ the\ objv\ from\ applying\ Tcl_ListObjGetElements\ to\ the\ result\ caught\ with\ the\ TCL_SUSPEND,\ and\ resumeIdx\ is\ the\ index\ into\ this\ array\ of\ the\ element\ which\ is\ relevant\ for\ this\ command\;\ the\ resumeProc\ will\ eventually\ call\ the\ resumeProc\ of\ the\ command\ it\ received\ TCL_SUSPEND\ from\ with\ the\ same\ resumeData\ but\ resumeIdx-1.\n\nFor\ practical\ convenience,\ it\ is\ probably\ also\ useful\ to\ have\ a\ library\ function\ for\ calling\ the\ resumeProc\ of\ a\ particular\ command,\ so\ Tcl_EvalObjEx\ (or\ whichever\ is\ the\ core\ one\ these\ days)\ would\ get\ the\ cousin\n\nint\ Tcl_ResumeObjEx(interp,\ objPtr,\ flags,\ resumeIdx,\ resumeData)\n\n\nThe\ tricky\ part\ is\ of\ course\ that\ all\ these\ resumeProcs\ must\ be\ provided\ when\ the\ command\ is\ created.\ For\ a\ long\ time\ I\ thought\ that\ this\ meant\ nothing\ of\ this\ could\ be\ implemented\ before\ Tcl\ 9\ anyway\ (hence\ there\ would\ be\ no\ rush\ to\ generate\ a\ string\ representation\ :-\]\ of\ the\ suspend/resume\ idea),\ but\ now\ I'm\ not\ so\ sure\ --\ it's\ perfectly\ reasonable\ to\ have\ e.g.\ Tcl_CreateObjCommand\ create\ commands\ without\ a\ resumeProc\ (resumeProc==NULL),\ and\ instead\ introduce\ a\ new\ function\n\nTcl_Command\ Tcl_CreateSuspendableCommand(\n\ \ \ \ interp,\ cmdName,\ proc,\ clientData,\ deleteProc,\ resumeProc\n)\n\nfor\ creating\ commands\ that\ do\ have\ it.\ Calling\ Tcl_ResumeObjEx\ for\ a\ command\ that\ doesn't\ have\ a\ resumeProc\ won't\ work,\ but\ we\ can\ simply\ report\ that\ as\ an\ error,\ so\ it's\ no\ big\ deal\ (at\ the\ C\ level).\n\nHowever,\ rather\ than\ reporting\ such\ errors\ at\ the\ resuming\ stage,\ it\ would\ be\ better\ to\ detect\ them\ at\ the\ suspend\ stage.\ In\ principle,\ that\ could\ be\ done\ by\ having\ Tcl_Eval*\ verify\ that\ any\ command\ returning\ a\ TCL_SUSPEND\ also\ has\ a\ resumeProc,\ or\ else\ convert\ that\ TCL_SUSPEND\ into\ an\ appropriate\ error.\ As\ far\ as\ I\ can\ tell,\ that\ would\ be\ the\ only\ ***potential\ incompatibility***\ of\ this\ scheme\ against\ current\ Tcl.\n\n\nCOMPARISON\ OF\ \[coroutine\]/\[yield\]\ AND\ \[suspend\]/\[resume\]\n\nRoughly,\ \[suspend\]\ corresponds\ to\ \[yield\],\ whereas\ \[resume\]\ is\ more\ like\ the\ coroutine-cmds\;\ there\ is\ no\ explicit\ \[coroutine\]-like\ command\ needed\ in\ the\ \[suspend\]/\[resume\]\ approach,\ since\ one\ is\ implicitly\ provided\ by\ the\ main\ event\ loop\ (when\ that\ is\ running).\n\n*Existence.*\ Obviously,\ a\ big\ difference\ is\ that\ an\ implementation\ of\ \[coroutine\]/\[yield\]\ exists,\ whereas\ \[suspend\]/\[resume\]\ is\ at\ best\ a\ sketch.\n\n*Model.*\ \[coroutine\]/\[yield\]\ is,\ as\ far\ as\ I\ can\ tell,\ modelled\ after\ the\ concept\ of\ a\ generator:\ a\ subroutine\ than\ can\ return\ several\ times.\ \[suspend\]/\[resume\]\ rather\ follows\ the\ continuation\ model:\ the\ rough\ idea\ of\ \"the\ program\ from\ this\ point\ on\"\ is\ given\ tangible\ existence.\ It\ is\ different\ from\ the\ LISPish\ (call-cc)\ in\ that\ the\ continuation\ is\ not\ given\ the\ form\ of\ an\ anonymous\ function,\ but\ that's\ natural\ given\ the\ more\ imperative\ nature\ of\ Tcl,\ and\ it\ is\ also\ different\ in\ that\ it\ is\ not\ the\ part\ of\ the\ program\ that\ calls\ \[suspend\]\ which\ gets\ to\ handle\ the\ continuation,\ but\ rather\ some\ generic\ handler\ at\ the\ top\ level.\n\n*Storage.*\ Since\ \[suspend\]\ starts\ putting\ data\ in\ the\ interpreter\ result,\ everything\ has\ to\ be\ put\ under\ a\ Tcl_Obj\ wrapper.\ This\ is\ potentially\ a\ slow\ operation\ (but\ probably\ not\ so\ slow\ that\ it\ becomes\ a\ major\ issue\ when\ suspending\ the\ handler\ for\ an\ event).\ By\ contrast,\ \[coroutine\]/\[yield\]\ hides\ everything\ under\ the\ opaque\ handle\ of\ a\ command.\n\nGiving\ continuations\ the\ form\ of\ a\ Tcl_Obj\ brings\ all\ the\ usual\ EIAS\ advantages:\ can\ pass\ by\ value\ and\ store\ in\ data\ structures,\ can\ save\ on\ file\ or\ hand\ over\ to\ another\ thread,\ one\ can\ even\ resume\ the\ same\ continuation\ several\ times\ (thus\ admitting\ a\ native\ implementation\ of\ oracles\ for\ nondeterministic\ automata)!\ The\ cost\ is\ that\ one\ actually\ has\ to\ implement\ (for\ every\ resumable\ command!)\ a\ relevant\ freeIntRepProc,\ dupIntRepProc,\ updateStringProc,\ and\ setFromAnyProc.\ This\ _is_\ just\ a\ Simple\ Matter\ Of\ Programming\ compared\ to\ \[coroutine\]/\[yield\],\ since\ the\ potentially\ nontrivial\ task\ of\ isolating\ the\ internal\ state\ of\ the\ generator\ from\ the\ rest\ of\ the\ interpreter\ has\ already\ been\ done\ and\ all\ that\ remains\ is\ to\ serialize/deserialize\ this\ state.\ Designing\ a\ serialization\ format\ capable\ of\ expressing\ the\ necessary\ data\ structures\ is\ obviously\ possible.\ Designing\ it\ so\ that\ it\ can\ _only_\ express\ sensible\ instances\ of\ the\ necessary\ data\ structures\ is\ perhaps\ less\ easy,\ but\ this\ is\ not\ technically\ required\ for\ EIAS.\n\nThere\ is\ OTOH\ no\ principal\ problem\ in\ introducing\ explicit\ commands\ that\ serialize\ and\ deserialize\ a\ coroutine-command\ in\ the\ \[coroutine\]/\[yield\]\ approach\ (it\ just\ hasn't\ been\ done\ yet),\ and\ this\ could\ then\ be\ used\ to\ transport\ coroutines\ across\ thread\ boundaries,\ but\ there\ is\ probably\ a\ practical\ problem\ in\ that\ cooperation\ (serialization/deserialisation\ of\ clientData)\ would\ be\ required\ from\ anything\ that\ creates\ an\ NRE\ callback.\n\n*Emulation.*\ \[coroutine\]\ and\ \[yield\]\ can\ be\ emulated\ using\ \[suspend\]/\[resume\],\ as\ follows\ (rough\ implementation):\n\ninterp\ alias\ \{\}\ yield\ \{\}\ suspend\ yield\n\nproc\ coroutine\ \{cmd\ args\}\ \{\n\ \ \ if\ \{\[catch\ \{\{*\}\$args\}\ res\ opt\]\ !=\ \$::TCL_SUSPEND\}\ then\ \{\n\ \ \ \ \ \ return\ -options\ \$opt\ \$res\n\ \ \ \}\n\ \ \ switch\ --\ \[lindex\ \$res\ 0\ 0\]\ \"yield\"\ \{\n\ \ \ \ \ \ interp\ alias\ \{\}\ \$cmd\ \{\}\ call-again\ \$cmd\ \$res\n\ \ \ \ \ \ return\ \[lindex\ \$res\ 0\ 1\]\n\ \ \ \}\n\}\n\nproc\ call-again\ \{cmd\ continuation\ value\}\ \{\n\ \ \ rename\ \$cmd\ \"\"\n\ \ \ coroutine\ \$cmd\ resume\ \$continuation\ \$value\n\}\n\nI\ don't\ see\ how\ the\ converse\ would\ be\ possible\ (but\ since\ NRE\ is\ more\ than\ just\ \[coroutine\],\ that\ needn't\ be\ a\ problem).\n\n*Intervention.*\ Since\ a\ TCL_SUSPEND\ is\ a\ return\ code\ and\ can\ be\ caught,\ it\ allows\ surrounding\ control\ structures\ to\ intervene\ and/or\ react\ to\ being\ suspended.\ As\ I\ understand\ it,\ NRE\ provides\ nothing\ in\ this\ area.\n\nAs\ an\ example\ of\ intervention,\ consider\ control\ structures\ that\ exist\ not\ to\ do\ flow\ control,\ but\ to\ guard\ the\ utilization\ of\ some\ limited\ resource.\ One\ such\ case\ is\ the\ transaction\ subcommand\ of\ a\ TDBC\ handle:\n\n\ proc\ foo\ \{db\ expr\}\ \{\n\ \ \ ...\n\ \ \ \$db\ transaction\ \{\n\ \ \ \ \ \ ...\n\ \ \ \ \ \ bar\ \$expr\ \$value\n\ \ \ \ \ \ ...\n\ \ \ \}\n\ \ \ ...\n\ \}\n\ proc\ bar\ \{expr\ value\}\ \{\n\ \ \ ...\n\ \ \ if\ \$expr\ then\ \{set\ value\ \[yield\ \$value\]\}\n\ \ \ ...\n\ \}\n\ coroutine\ baz\ foo\ \$db\ \$expr\n\nChances\ are\ that\ the\ \[\[\$db\ transaction\]\]\ above\ will\ count\ as\ recursion\ with\ respet\ to\ the\ NRE\ and\ thus\ cause\ \[yield\]\ to\ error\ out,\ with\ a\ \[\$db\ rollback\]\ as\ consequence,\ but\ if\ it\ does\ not\ then\ the\ database\ could\ stay\ locked\ for\ a\ rather\ long\ time,\ without\ a\ clear\ offender.\ By\ contrast,\ if\ the\ \[yield\]\ had\ been\ a\ \[suspend\]\ instead\ then\ \[\$db\ transaction\]\ would\ be\ explicitly\ informed\ about\ what\ is\ going\ on,\ and\ thus\ given\ a\ chance\ to\ react\ sensibly.\ Doing\ an\ immediate\ \[\$db\ rollback\]\ and\ changing\ the\ TCL_SUSPEND\ into\ a\ TCL_ERROR\ is\ the\ most\ elementary\ reaction\ (and\ perhaps\ the\ only\ one\ directly\ supported\ by\ the\ TDBC\ interfaces),\ but\ it\ might\ often\ be\ more\ practical\ to\ capture\ the\ operations\ made\ so\ far\ in\ some\ kind\ of\ diff,\ and\ attempt\ to\ reapply\ them\ when\ the\ transaction\ is\ resumed.\ In\ that\ case,\ the\ TCL_SUSPEND\ should\ be\ passed\ on.\n\nA\ complication\ for\ control\ structures\ implemented\ in\ Tcl\ is\ that\ they\ need\ to\ create\ some\ representation\ of\ their\ own\ internal\ state\ when\ passing\ on\ a\ TCL_SUSPEND.\ If\ a\ third\ command\ besides\ \[suspend\]\ and\ \[resume\]\ is\ needed,\ then\ this\ will\ probably\ be\ why.\ On\ the\ other\ hand,\ if\ the\ serialization\ format\ used\ for\ the\ internal\ state\ of\ a\ proc\ is\ documented,\ then\ all\ sorts\ of\ programming\ tricks\ (e.g.\ disconnecting\ an\ \[upvar\]ed\ variable)\ can\ be\ implemented\ as\ \[suspend\]ing\ execution\ and\ having\ some\ detail\ in\ the\ state\ of\ some\ surrounding\ procedure\ modified...\n\n\n\nmiguel\ sofer\ skrev:\n>\ Lars\ Hellström\ wrote:\n>>\ miguel\ skrev:\n>>>\ I\ managed\ to\ commit\ an\ experimental\ implementation\ of\ coroutines\ in\ time\ for\ 8.6a2.\ This\ provides\ two\ new\ commands\n>>>\ \ \ \ ::tcl::unsupported::coroutine\n>>>\ \ \ \ ::tcl::unsupported::yield\n>>>\n>>>\ A\ brief\ description\ can\ be\ found\ at\ http://msofer.com:8080/wiki?name=Coroutines,\ otherwise\ the\ code\ in\ tests/unsupported.test\ is\ the\ only\ documentation.\n>>>\n>>>\ Test!\ Enjoy!\ Help\ make\ it\ better\n>>\n>>\ This\ might\ fall\ in\ the\ \"help\ make\ better\"\ category,\ although\ it's\ really\ some\ loose\ sketches\ for\ an\ alternative\ realization\ of\ the\ same\ concept\ that\ I've\ been\ contemplating\ on\ and\ off\ during\ the\ last\ couple\ of\ years\ (judging\ from\ http://wiki.tcl.tk/13232,\ since\ April\ 2005).\ I'm\ posting\ this\ mostly\ to\ provide\ an\ alternative\ point\ of\ view\ --\ maybe\ some\ details\ of\ the\ Tcl\ interface\ would\ be\ better\ if\ done\ differently?\ --\ although\ it\ seems\ this\ approach\ could\ actually\ do\ some\ things\ mentioned\ as\ unsolved\ in\ the\ NRE\ implementation\ (moving\ coroutines\ between\ threads,\ handle\ C-coded\ commands\ calling\ Tcl_Eval).\n>\n>\ Thanks.\n>\n>\ There\ is\ one\ problem\ with\ this\ approach:\ it\ breaks\ every\ extension\ that\ calls\ for\ an\ evaluation,\ as\ it\ doesn't\ know\ about\ TCL_SUSPEND\ nor\ what\ to\ do\ with\ it.\ Some\ of\ those\ are\ under\ our\ control\ (Tk\ for\ example),\ but\ unknown\ quantities\ are\ not.\n>\n>\ A\ second\ (or\ is\ it\ the\ same?)\ problem\ is\ that\ extensions\ may\ well\ be\ using\ their\ own\ return\ codes\ ...\ a\ new\ return\ code\ is\ not\ something\ that\ we\ can\ do\ until\ Tcl9.\n\nHadn't\ thought\ of\ that\ one,\ but\ OTOH\ I\ was\ mostly\ thinking\ Tcl9\ anyway\ (until\ recently).\n\nRealistically\ though,\ if\ some\ TCT\ member\ had\ come\ up\ with\ an\ idea\ that\ required\ usurping\ another\ return\ code,\ then\ I\ doubt\ there\ would\ have\ been\ much\ opposition\ against\ doing\ so\ within\ the\ 8.x\ series.\ Also,\ that\ noone\ else\ reported\ bug#647307\ is\ perhaps\ an\ indication\ that\ nonstandard\ return\ codes\ aren't\ that\ heavily\ used.\ \;-)\n\n>\ A\ third\ problem\ with\ the\ special\ return\ code\ is\ that\ \[catch\]\ would\ block\ it\;\ the\ current\ code\ can\ yield\ in\ arbitrarily\ nested\ catch\ ranges.\n\n*That*\ is\ not\ a\ bug\;\ it\ is\ a\ feature\ (as\ explained\ above)!\n\nLate\ addition:\ One\ could\ improve\ interoperability\ between\ \[suspend\]/\[resume\]\ and\ old\ control\ structures\ implemented\ in\ Tcl\ by\ making\ it\ so\ that\ \[catch\]\ only\ catches\ TCL_SUSPEND\ if\ it\ has\ an\ extra\ argument\ --\ thus\ extending\ the\ syntax\ to\n\n\ \ catch\ \$script\ ?resvar?\ ?optionsvar?\ ?resumeitemvar?\n\n--\ where\ the\ resumeitemvar\ is\ filled\ in\ with\ the\ resumeData\ item\ that\ would\ resume\ processing\ in\ this\ proc\ from\ within\ that\ \[catch\].\ Then\ the\ coroutine\ emulator\ above\ could\ let\ non-yield\ TCL_SUSPENDs\ pass\ through\ if\ changed\ to\n\n\ proc\ coroutine\ \{cmd\ args\}\ \{\n\ \ \ if\ \{\[catch\ \{\{*\}\$args\}\ res\ opt\ item\]\ !=\ \$::TCL_SUSPEND\}\ then\ \{\n\ \ \ \ \ \ return\ -options\ \$opt\ \$res\n\ \ \ \}\n\ \ \ switch\ --\ \[lindex\ \$res\ 0\ 0\]\ \"yield\"\ \{\n\ \ \ \ \ \ #\ This\ kind\ of\ \[suspend\]\ is\ handled\ here.\n\ \ \ \ \ \ interp\ alias\ \{\}\ \$cmd\ \{\}\ call-again\ \$cmd\ \$res\n\ \ \ \ \ \ return\ \[lindex\ \$res\ 0\ 1\]\n\ \ \ \}\ default\ \{\n\ \ \ \ \ \ #\ Other\ kinds\ get\ to\ suspend\ also\ \[coroutine\],\n\ \ \ \ \ \ #\ backing\ up\ to\ the\ \[catch\]\ above.\n\ \ \ \ \ \ lappend\ res\ \$item\n\ \ \ \ \ \ return\ -code\ suspend\ \$res\n\ \ \ \}\n\ \}\n\n\n>\ The\ thing\ I\ implemented\ is\ 100%\ compatible,\ extensions\ that\ do\ not\ adapt\ keep\ on\ working\ but\ may\ block\ suspending.\ There\ is\ an\ api\ for\ extenders\ to\ adapt.\n\nNo\ big\ difference\ there,\ I'd\ say.\n\n>\ This\ is\ also\ one\ of\ the\ reasons\ why\ you\ have\ to\ \"declare\"\ a\ coroutine:\ you\ have\ to\ prepare\ the\ infrastucture\ to\ be\ able\ to\ suspend\ a\ \"thread\"\ (?),\ which\ may\ or\ may\ not\ mean\ returning\ to\ the\ top\ level.\n>\n>\ Your\ ideas\ would\ still\ fail\ to\ produce\ coroutines\ that\ can\ migrate\ accross\ threads,\ at\ least\ without\ big\ changes\ in\ the\ core:\ the\ problem\ is\ that\ the\ same\ interp\ cannot\ be\ run\ in\ different\ threads.\ The\ same\ problem\ as\ the\ ones\ in\ head.\n\nThread\ migration\ would\ be\ done\ via\ serialization,\ as\ explained\ above.\ Reconstructing\ the\ state\ of\ a\ proc\ from\ string\ rep\ in\ another\ thread\ may\ seem\ like\ a\ hazardous\ task,\ but\ probably\ no\ more\ so\ than\ in\ the\ original\ interp.\ In\ particular,\ I\ suspect\ one\ must\ be\ able\ to\ handle\ changes\ in\ the\ bytecode\ for\ the\ procedure\ one\ is\ resuming,\ since\ redefenitions\ of\ other\ command\ may\ well\ have\ triggered\ a\ recompilation,\ can\ they\ not?\n\nAre\ you\ otherwise\ saying\ that\ NRE\ is\ employing\ threads\ to\ do\ its\ work?\ Or\ did\ you\ assume\ \[suspend\]\ would?\ (It\ doesn't.)\n\n>\ What\ I\ implemented\ relies\ on\ the\ NRE\ execution\ model:\ (newer,\ adapted)\ C\ code\ never\ calls\ an\ eval,\ it\ rather\ queues\ a\ callback\ with\ enough\ info\ do\ any\ post-processing\ or\ cleanup\ it\ needs\ to\ do\ after\ the\ evaluation,\ schedules\ the\ new\ evaluation\ and\ returns\ TCL_OK.\ There\ is\ a\ trampoline\ that\ runs\ the\ evaluation\ and\ then\ the\ callbacks.\ If\ a\ C\ function\ actually\ does\ call\ an\ eval\ using\ the\ regular\ API,\ trying\ to\ suspend\ before\ it\ returns\ will\ raise\ an\ error.\n>\n>\ That\ means\ that\ no\ C\ function\ is\ ever\ suspended:\ it\ was\ implemented\ as\ a\ two-part\ function\ to\ begin\ with.\ Even\ though\ your\ email\ got\ truncated,\ I\ imagine\ this\ is\ not\ too\ different\ in\ spirit\ to\ the\ two-entry-point\ system\ you\ were\ about\ to\ describe?\n\nSlightly\ different\ in\ spirit,\ but\ perhaps\ not\ in\ nature.\ In\ order\ to\ make\ a\ typical\ resumeProc\ for\ Tcl_CreateSuspendableCommand,\ one\ should\ be\ able\ to\ just\ take\ the\ corresponding\ proc\ and\ add\ \"spaghetti\"\ for\ jumping\ to\ the\ place\ where\ resuming\ requires\ one\ to\ be\ --\ provided\ that\ one\ has\ the\ \"stomach\"\ for\ such\ violations\ of\ Structured\ Programming\ dogma.\ By\ contrast,\ the\ NRE\ API\ of\ callbacks\ one\ can\ add\ rather\ suggests\ that\ the\ proc\ should\ be\ transformed\ into\ an\ automaton.\ It\ is\ possible\ that\ any\ resumeProc\ obtained\ through\ adding\ spaghetti\ will\ also\ be\ possible\ to\ operate\ as\ an\ automaton\ through\ the\ Tcl_NRAddCallback\ interface,\ in\ which\ case\ the\ two\ would\ indeed\ not\ be\ different\ in\ nature,\ but\ I\ haven't\ explored\ the\ design\ in\ that\ much\ detail.\n\n======\n\n\n\n----\n!!!!!!\n%|\ \[Category\ Suggestions\]\ |\ \[Category\ Control\ Structure\]\ |%\n!!!!!! regexp2} CALL {my render {suspend and resume} \[Lars\ H\],\ 2008-08-27:\nThe\ following\ was\ first\ posted\ to\ the\ tcl-core\ mailing\ list,\ as\ a\ kind\ of\ ''well\ the\ \[NRE\]\ \[coroutine\]s\ may\ be\ nice\ but\ it\ might\ actually\ be\ possible\ to\ do\ something\ even\ more\ powerful…''\n\nThe\ main\ reason\ I'm\ putting\ it\ here\ now\ is\ that\ I\ actually\ managed\ to\ produce\ a\ very\ elementary\ \[proof\ of\ concept:\ suspend\ and\ resume\]\ in\ pure\ Tcl.\n\n\n**The\ specification**\n\nJust\ the\ plain\ text,\ for\ now.\ Should\ wikify\ it,\ though.\n\nBASIC\ PREMISE\n\nPrimarily\ I\ wanted\ a\ non-nesting\ \[vwait\]\ --\ one\ which\ would\ not\ hang\ in\n\nafter\ 1000\ \{\n\ \ \ after\ 1000\ \{set\ ::a\ 1\;\ vwait\ ::b\;\ puts\ \"Finished\"\}\n\ \ \ vwait\ ::a\n\ \ \ set\ ::b\ 1\n\}\n\nThis\ meant\ the\ \[vwait\]\ would\ somehow\ have\ to\ save\ away\ what\ was\ currently\ in\ the\ C\ call\ stack,\ drop\ out\ into\ the\ outer\ event\ loop\ and\ process\ events\ there\ for\ a\ while,\ until\ the\ variable\ has\ been\ set,\ at\ which\ point\ everything\ would\ have\ to\ be\ put\ back\ so\ that\ processing\ could\ continue.\n\nAlso,\ this\ would\ have\ to\ be\ accomplished\ \"within\ the\ box\"\ --\ no\ fancy\ C\ compiler\ or\ OS\ features\ could\ be\ relied\ on,\ and\ recursive\ calls\ should\ still\ be\ supported.\n\n\nSOLUTION\n\nSince\ there's\ no\ safe\ way\ to\ operate\ on\ the\ C\ stack\ from\ outside,\ the\ only\ way\ to\ get\ it\ out\ of\ the\ way\ is\ to\ explicitly\ unwind\ it\ --\ make\ every\ C\ function\ in\ there\ return\ to\ its\ caller.\ That's\ not\ too\ different\ from\ having\ an\ error\ propagate\ down\ the\ call\ stack,\ so:\n\nYielding\ is\ done\ by\ returning\ with\ a\ new\ \[return\]\ code,\nTCL_SUSPEND,\ say.\n\nWhen\ a\ yielding\ unwind\ is\ in\ progress,\ the\ interpreter\ result\nis\ a\ list.\ Each\ entity\ in\ a\ call\ chain\ receiving\ a\ TCL_SUSPEND\nresult\ code\ from\ a\ call\ it\ made\ must\ do\ one\ of\ the\ following:\n\n1.\ Append\ to\ the\ interp\ result\ a\ list\ element\ with\ sufficient\ information\ that\ it\ can\ later\ reconstruct\ the\ exact\ state\ it\ was\ in\ immediately\ before\ receiving\ the\ TCL_SUSPEND,\n\ \ \ \ \ and\ thus\ resume\ processing\ from\ where\ it\ was\ suspended.\ Then\ return\ to\ caller\ with\ TCL_SUSPEND\ code.\n\n2.\ Throw\ an\ error\ \"Cannot\ be\ suspended\".\n\n3.\ Catch\ the\ yield,\ if\ the\ entity\ is\ prepared\ to\ handle\ it.\n\nNormally,\ a\ yield\ would\ proceed\ all\ the\ way\ down\ to\ the\ main\ event\ loop,\ where\ it\ gets\ handed\ off\ to\ \[bgerror\]\ (or\ the\ like),\ and\ that\ command\ is\ then\ responsible\ for\ arranging\ so\ that\ the\ suspended\ calls\ are\ resumed\ at\ a\ suitable\ point\ in\ the\ future.\n\nA\ tricky\ detail\ is\ that\ there\ would\ now\ have\ to\ be\ two\ entry\ points\ for\ the\ C\ implementation\ of\ each\ Tcl\ command:\ one\ which\ is\ used\ when\ calling\ it\ normally,\ and\ one\ which\ is\ used\ when\ resuming\ it.\ I'll\ return\ to\ that\ matter\ below,\ but\ for\ now,\ let's\ look\ at\ what\ the\ script\ level\ behaviour\ is\ supposed\ to\ be.\n\n\nTCL\ API\n\nI\ imagined\ the\ Tcl\ side\ of\ this\ could\ be\ handled\ with\ only\ two\ commands\ (although\ \[catch\]\ and\ \[return\]\ would\ probably\ figure\ as\ prominently\ as\ with\ any\ new\ control\ structure):\n\n\ \ suspend\ ?\$arg\ ...?\n\ \ resume\ \$unwinding\ ?-code\ \$code?\ ?-level\ \$level?\ ?...?\ ?\$result?\n\n\[suspend\]\ stops\ execution\ at\ some\ point\ and\ causes\ the\ call\ stack\ to\ unwind,\ whereas\ \[resume\]\ resumes\ it.\ If\ you\ do\n\n\ \ proc\ suspending\ \{args\}\ \{list\ \[suspend\ \{*\}\$args\]\ \"in\ suspending\"\}\n\ \ catch\ \{suspending\ foo\ bar\}\ res\n\nthen\ the\ result\ will\ be\ TCL_SUSPEND\ and\ \$res\ will\ be\ an\ unwinding\ that\ contains\ the\ arguments\ of\ the\ \[suspend\]\ in\ its\ 0th\ element,\ i.e.,\ it\ looks\ like\n\n\ \ \{foo\ bar\}\ \{...\n\nwhere\ the\ ...\ denotes\ the\ beginning\ of\ the\ serialization\ of\ the\ internal\ state\ of\ the\ call\ stack\ entity\ corresponding\ to\ the\ \[suspending\]\ procedure\ (yes,\ that\ is\ potentially\ quite\ a\ mouthful).\n\nThe\ \[resume\]\ arguments\ following\ the\ \$unwinding\ are\ supposed\ to\ be\ the\ same\ as\ for\ \[return\],\ but\ control\ how\ the\ original\ \[suspend\]\ behaves\ when\ resumed.\ Thus\ if\ one\ then\ does\n\n\ \ resume\ \$res\ \"Return\ from\ with\"\n\nthen\ the\ result\ will\ be\ the\ list\n\n\ \ \{Return\ from\ with\}\ \{in\ suspending\}\n\nbecause\ \[resume\]\ put\ back\ the\ \[suspending\]\ proc\ and\ the\ \[suspend\]\ command,\ the\ latter\ returns\ with\ TCL_OK\ (since\ there\ was\ no\ other\ -code\ specified),\ the\ \[list\]\ in\ suspended\ is\ called\ as\n\n\ \ list\ \{Return\ from\ with\}\ \"in\ suspending\"\n\nand\ the\ result\ of\ that\ becomes\ the\ result\ of\ \[suspending\]\ as\ a\ whole,\ which\ is\ then\ also\ the\ result\ of\ the\ \[resume\].\n\n\nC\ API\n\n(Not\ my\ strong\ side,\ but\ one\ that\ has\ to\ be\ addressed.)\ Since\ the\ idea\ is\ mostly\ to\ give\ new\ meaning\ to\ things\ that\ can\ already\ happen,\ there\ isn't\ /that/\ much\ that\ has\ to\ be\ added\ on\ the\ C\ side,\ but\ one\ major\ thing\ is\ the\ prototype\ for\ the\ function\ that\ is\ called\ when\ a\ command\ is\ resumed\;\ let's\ call\ this\ a\ resumeProc:\n\ntypedef\ int\ Tcl_ObjResumeProc(\n\tClientData\ \ \ \ \ \ clientData,\n\tTcl_Interp\ \ \ \ \ *interp,\n\tint\t\ \ \ \ \ objc,\n\tTcl_Obj\ *const\ \ objv\[\],\n\tint\t\ \ \ \ \ resumeIdx,\n\tTcl_Obj\ *const\ \ resumeData\[\]\ \ )\;\n\nThe\ idea\ here\ is\ that\ the\ resumeData\ should\ be\ the\ objv\ from\ applying\ Tcl_ListObjGetElements\ to\ the\ result\ caught\ with\ the\ TCL_SUSPEND,\ and\ resumeIdx\ is\ the\ index\ into\ this\ array\ of\ the\ element\ which\ is\ relevant\ for\ this\ command\;\ the\ resumeProc\ will\ eventually\ call\ the\ resumeProc\ of\ the\ command\ it\ received\ TCL_SUSPEND\ from\ with\ the\ same\ resumeData\ but\ resumeIdx-1.\n\nFor\ practical\ convenience,\ it\ is\ probably\ also\ useful\ to\ have\ a\ library\ function\ for\ calling\ the\ resumeProc\ of\ a\ particular\ command,\ so\ Tcl_EvalObjEx\ (or\ whichever\ is\ the\ core\ one\ these\ days)\ would\ get\ the\ cousin\n\nint\ Tcl_ResumeObjEx(interp,\ objPtr,\ flags,\ resumeIdx,\ resumeData)\n\n\nThe\ tricky\ part\ is\ of\ course\ that\ all\ these\ resumeProcs\ must\ be\ provided\ when\ the\ command\ is\ created.\ For\ a\ long\ time\ I\ thought\ that\ this\ meant\ nothing\ of\ this\ could\ be\ implemented\ before\ Tcl\ 9\ anyway\ (hence\ there\ would\ be\ no\ rush\ to\ generate\ a\ string\ representation\ :-\]\ of\ the\ suspend/resume\ idea),\ but\ now\ I'm\ not\ so\ sure\ --\ it's\ perfectly\ reasonable\ to\ have\ e.g.\ Tcl_CreateObjCommand\ create\ commands\ without\ a\ resumeProc\ (resumeProc==NULL),\ and\ instead\ introduce\ a\ new\ function\n\nTcl_Command\ Tcl_CreateSuspendableCommand(\n\ \ \ \ interp,\ cmdName,\ proc,\ clientData,\ deleteProc,\ resumeProc\n)\n\nfor\ creating\ commands\ that\ do\ have\ it.\ Calling\ Tcl_ResumeObjEx\ for\ a\ command\ that\ doesn't\ have\ a\ resumeProc\ won't\ work,\ but\ we\ can\ simply\ report\ that\ as\ an\ error,\ so\ it's\ no\ big\ deal\ (at\ the\ C\ level).\n\nHowever,\ rather\ than\ reporting\ such\ errors\ at\ the\ resuming\ stage,\ it\ would\ be\ better\ to\ detect\ them\ at\ the\ suspend\ stage.\ In\ principle,\ that\ could\ be\ done\ by\ having\ Tcl_Eval*\ verify\ that\ any\ command\ returning\ a\ TCL_SUSPEND\ also\ has\ a\ resumeProc,\ or\ else\ convert\ that\ TCL_SUSPEND\ into\ an\ appropriate\ error.\ As\ far\ as\ I\ can\ tell,\ that\ would\ be\ the\ only\ ***potential\ incompatibility***\ of\ this\ scheme\ against\ current\ Tcl.\n\n\nCOMPARISON\ OF\ \[coroutine\]/\[yield\]\ AND\ \[suspend\]/\[resume\]\n\nRoughly,\ \[suspend\]\ corresponds\ to\ \[yield\],\ whereas\ \[resume\]\ is\ more\ like\ the\ coroutine-cmds\;\ there\ is\ no\ explicit\ \[coroutine\]-like\ command\ needed\ in\ the\ \[suspend\]/\[resume\]\ approach,\ since\ one\ is\ implicitly\ provided\ by\ the\ main\ event\ loop\ (when\ that\ is\ running).\n\n*Existence.*\ Obviously,\ a\ big\ difference\ is\ that\ an\ implementation\ of\ \[coroutine\]/\[yield\]\ exists,\ whereas\ \[suspend\]/\[resume\]\ is\ at\ best\ a\ sketch.\n\n*Model.*\ \[coroutine\]/\[yield\]\ is,\ as\ far\ as\ I\ can\ tell,\ modelled\ after\ the\ concept\ of\ a\ generator:\ a\ subroutine\ than\ can\ return\ several\ times.\ \[suspend\]/\[resume\]\ rather\ follows\ the\ continuation\ model:\ the\ rough\ idea\ of\ \"the\ program\ from\ this\ point\ on\"\ is\ given\ tangible\ existence.\ It\ is\ different\ from\ the\ LISPish\ (call-cc)\ in\ that\ the\ continuation\ is\ not\ given\ the\ form\ of\ an\ anonymous\ function,\ but\ that's\ natural\ given\ the\ more\ imperative\ nature\ of\ Tcl,\ and\ it\ is\ also\ different\ in\ that\ it\ is\ not\ the\ part\ of\ the\ program\ that\ calls\ \[suspend\]\ which\ gets\ to\ handle\ the\ continuation,\ but\ rather\ some\ generic\ handler\ at\ the\ top\ level.\n\n*Storage.*\ Since\ \[suspend\]\ starts\ putting\ data\ in\ the\ interpreter\ result,\ everything\ has\ to\ be\ put\ under\ a\ Tcl_Obj\ wrapper.\ This\ is\ potentially\ a\ slow\ operation\ (but\ probably\ not\ so\ slow\ that\ it\ becomes\ a\ major\ issue\ when\ suspending\ the\ handler\ for\ an\ event).\ By\ contrast,\ \[coroutine\]/\[yield\]\ hides\ everything\ under\ the\ opaque\ handle\ of\ a\ command.\n\nGiving\ continuations\ the\ form\ of\ a\ Tcl_Obj\ brings\ all\ the\ usual\ EIAS\ advantages:\ can\ pass\ by\ value\ and\ store\ in\ data\ structures,\ can\ save\ on\ file\ or\ hand\ over\ to\ another\ thread,\ one\ can\ even\ resume\ the\ same\ continuation\ several\ times\ (thus\ admitting\ a\ native\ implementation\ of\ oracles\ for\ nondeterministic\ automata)!\ The\ cost\ is\ that\ one\ actually\ has\ to\ implement\ (for\ every\ resumable\ command!)\ a\ relevant\ freeIntRepProc,\ dupIntRepProc,\ updateStringProc,\ and\ setFromAnyProc.\ This\ _is_\ just\ a\ Simple\ Matter\ Of\ Programming\ compared\ to\ \[coroutine\]/\[yield\],\ since\ the\ potentially\ nontrivial\ task\ of\ isolating\ the\ internal\ state\ of\ the\ generator\ from\ the\ rest\ of\ the\ interpreter\ has\ already\ been\ done\ and\ all\ that\ remains\ is\ to\ serialize/deserialize\ this\ state.\ Designing\ a\ serialization\ format\ capable\ of\ expressing\ the\ necessary\ data\ structures\ is\ obviously\ possible.\ Designing\ it\ so\ that\ it\ can\ _only_\ express\ sensible\ instances\ of\ the\ necessary\ data\ structures\ is\ perhaps\ less\ easy,\ but\ this\ is\ not\ technically\ required\ for\ EIAS.\n\nThere\ is\ OTOH\ no\ principal\ problem\ in\ introducing\ explicit\ commands\ that\ serialize\ and\ deserialize\ a\ coroutine-command\ in\ the\ \[coroutine\]/\[yield\]\ approach\ (it\ just\ hasn't\ been\ done\ yet),\ and\ this\ could\ then\ be\ used\ to\ transport\ coroutines\ across\ thread\ boundaries,\ but\ there\ is\ probably\ a\ practical\ problem\ in\ that\ cooperation\ (serialization/deserialisation\ of\ clientData)\ would\ be\ required\ from\ anything\ that\ creates\ an\ NRE\ callback.\n\n*Emulation.*\ \[coroutine\]\ and\ \[yield\]\ can\ be\ emulated\ using\ \[suspend\]/\[resume\],\ as\ follows\ (rough\ implementation):\n\ninterp\ alias\ \{\}\ yield\ \{\}\ suspend\ yield\n\nproc\ coroutine\ \{cmd\ args\}\ \{\n\ \ \ if\ \{\[catch\ \{\{*\}\$args\}\ res\ opt\]\ !=\ \$::TCL_SUSPEND\}\ then\ \{\n\ \ \ \ \ \ return\ -options\ \$opt\ \$res\n\ \ \ \}\n\ \ \ switch\ --\ \[lindex\ \$res\ 0\ 0\]\ \"yield\"\ \{\n\ \ \ \ \ \ interp\ alias\ \{\}\ \$cmd\ \{\}\ call-again\ \$cmd\ \$res\n\ \ \ \ \ \ return\ \[lindex\ \$res\ 0\ 1\]\n\ \ \ \}\n\}\n\nproc\ call-again\ \{cmd\ continuation\ value\}\ \{\n\ \ \ rename\ \$cmd\ \"\"\n\ \ \ coroutine\ \$cmd\ resume\ \$continuation\ \$value\n\}\n\nI\ don't\ see\ how\ the\ converse\ would\ be\ possible\ (but\ since\ NRE\ is\ more\ than\ just\ \[coroutine\],\ that\ needn't\ be\ a\ problem).\n\n*Intervention.*\ Since\ a\ TCL_SUSPEND\ is\ a\ return\ code\ and\ can\ be\ caught,\ it\ allows\ surrounding\ control\ structures\ to\ intervene\ and/or\ react\ to\ being\ suspended.\ As\ I\ understand\ it,\ NRE\ provides\ nothing\ in\ this\ area.\n\nAs\ an\ example\ of\ intervention,\ consider\ control\ structures\ that\ exist\ not\ to\ do\ flow\ control,\ but\ to\ guard\ the\ utilization\ of\ some\ limited\ resource.\ One\ such\ case\ is\ the\ transaction\ subcommand\ of\ a\ TDBC\ handle:\n\n\ proc\ foo\ \{db\ expr\}\ \{\n\ \ \ ...\n\ \ \ \$db\ transaction\ \{\n\ \ \ \ \ \ ...\n\ \ \ \ \ \ bar\ \$expr\ \$value\n\ \ \ \ \ \ ...\n\ \ \ \}\n\ \ \ ...\n\ \}\n\ proc\ bar\ \{expr\ value\}\ \{\n\ \ \ ...\n\ \ \ if\ \$expr\ then\ \{set\ value\ \[yield\ \$value\]\}\n\ \ \ ...\n\ \}\n\ coroutine\ baz\ foo\ \$db\ \$expr\n\nChances\ are\ that\ the\ \[\[\$db\ transaction\]\]\ above\ will\ count\ as\ recursion\ with\ respet\ to\ the\ NRE\ and\ thus\ cause\ \[yield\]\ to\ error\ out,\ with\ a\ \[\$db\ rollback\]\ as\ consequence,\ but\ if\ it\ does\ not\ then\ the\ database\ could\ stay\ locked\ for\ a\ rather\ long\ time,\ without\ a\ clear\ offender.\ By\ contrast,\ if\ the\ \[yield\]\ had\ been\ a\ \[suspend\]\ instead\ then\ \[\$db\ transaction\]\ would\ be\ explicitly\ informed\ about\ what\ is\ going\ on,\ and\ thus\ given\ a\ chance\ to\ react\ sensibly.\ Doing\ an\ immediate\ \[\$db\ rollback\]\ and\ changing\ the\ TCL_SUSPEND\ into\ a\ TCL_ERROR\ is\ the\ most\ elementary\ reaction\ (and\ perhaps\ the\ only\ one\ directly\ supported\ by\ the\ TDBC\ interfaces),\ but\ it\ might\ often\ be\ more\ practical\ to\ capture\ the\ operations\ made\ so\ far\ in\ some\ kind\ of\ diff,\ and\ attempt\ to\ reapply\ them\ when\ the\ transaction\ is\ resumed.\ In\ that\ case,\ the\ TCL_SUSPEND\ should\ be\ passed\ on.\n\nA\ complication\ for\ control\ structures\ implemented\ in\ Tcl\ is\ that\ they\ need\ to\ create\ some\ representation\ of\ their\ own\ internal\ state\ when\ passing\ on\ a\ TCL_SUSPEND.\ If\ a\ third\ command\ besides\ \[suspend\]\ and\ \[resume\]\ is\ needed,\ then\ this\ will\ probably\ be\ why.\ On\ the\ other\ hand,\ if\ the\ serialization\ format\ used\ for\ the\ internal\ state\ of\ a\ proc\ is\ documented,\ then\ all\ sorts\ of\ programming\ tricks\ (e.g.\ disconnecting\ an\ \[upvar\]ed\ variable)\ can\ be\ implemented\ as\ \[suspend\]ing\ execution\ and\ having\ some\ detail\ in\ the\ state\ of\ some\ surrounding\ procedure\ modified...\n\n\n\nmiguel\ sofer\ skrev:\n>\ Lars\ Hellström\ wrote:\n>>\ miguel\ skrev:\n>>>\ I\ managed\ to\ commit\ an\ experimental\ implementation\ of\ coroutines\ in\ time\ for\ 8.6a2.\ This\ provides\ two\ new\ commands\n>>>\ \ \ \ ::tcl::unsupported::coroutine\n>>>\ \ \ \ ::tcl::unsupported::yield\n>>>\n>>>\ A\ brief\ description\ can\ be\ found\ at\ http://msofer.com:8080/wiki?name=Coroutines,\ otherwise\ the\ code\ in\ tests/unsupported.test\ is\ the\ only\ documentation.\n>>>\n>>>\ Test!\ Enjoy!\ Help\ make\ it\ better\n>>\n>>\ This\ might\ fall\ in\ the\ \"help\ make\ better\"\ category,\ although\ it's\ really\ some\ loose\ sketches\ for\ an\ alternative\ realization\ of\ the\ same\ concept\ that\ I've\ been\ contemplating\ on\ and\ off\ during\ the\ last\ couple\ of\ years\ (judging\ from\ http://wiki.tcl.tk/13232,\ since\ April\ 2005).\ I'm\ posting\ this\ mostly\ to\ provide\ an\ alternative\ point\ of\ view\ --\ maybe\ some\ details\ of\ the\ Tcl\ interface\ would\ be\ better\ if\ done\ differently?\ --\ although\ it\ seems\ this\ approach\ could\ actually\ do\ some\ things\ mentioned\ as\ unsolved\ in\ the\ NRE\ implementation\ (moving\ coroutines\ between\ threads,\ handle\ C-coded\ commands\ calling\ Tcl_Eval).\n>\n>\ Thanks.\n>\n>\ There\ is\ one\ problem\ with\ this\ approach:\ it\ breaks\ every\ extension\ that\ calls\ for\ an\ evaluation,\ as\ it\ doesn't\ know\ about\ TCL_SUSPEND\ nor\ what\ to\ do\ with\ it.\ Some\ of\ those\ are\ under\ our\ control\ (Tk\ for\ example),\ but\ unknown\ quantities\ are\ not.\n>\n>\ A\ second\ (or\ is\ it\ the\ same?)\ problem\ is\ that\ extensions\ may\ well\ be\ using\ their\ own\ return\ codes\ ...\ a\ new\ return\ code\ is\ not\ something\ that\ we\ can\ do\ until\ Tcl9.\n\nHadn't\ thought\ of\ that\ one,\ but\ OTOH\ I\ was\ mostly\ thinking\ Tcl9\ anyway\ (until\ recently).\n\nRealistically\ though,\ if\ some\ TCT\ member\ had\ come\ up\ with\ an\ idea\ that\ required\ usurping\ another\ return\ code,\ then\ I\ doubt\ there\ would\ have\ been\ much\ opposition\ against\ doing\ so\ within\ the\ 8.x\ series.\ Also,\ that\ noone\ else\ reported\ bug#647307\ is\ perhaps\ an\ indication\ that\ nonstandard\ return\ codes\ aren't\ that\ heavily\ used.\ \;-)\n\n>\ A\ third\ problem\ with\ the\ special\ return\ code\ is\ that\ \[catch\]\ would\ block\ it\;\ the\ current\ code\ can\ yield\ in\ arbitrarily\ nested\ catch\ ranges.\n\n*That*\ is\ not\ a\ bug\;\ it\ is\ a\ feature\ (as\ explained\ above)!\n\nLate\ addition:\ One\ could\ improve\ interoperability\ between\ \[suspend\]/\[resume\]\ and\ old\ control\ structures\ implemented\ in\ Tcl\ by\ making\ it\ so\ that\ \[catch\]\ only\ catches\ TCL_SUSPEND\ if\ it\ has\ an\ extra\ argument\ --\ thus\ extending\ the\ syntax\ to\n\n\ \ catch\ \$script\ ?resvar?\ ?optionsvar?\ ?resumeitemvar?\n\n--\ where\ the\ resumeitemvar\ is\ filled\ in\ with\ the\ resumeData\ item\ that\ would\ resume\ processing\ in\ this\ proc\ from\ within\ that\ \[catch\].\ Then\ the\ coroutine\ emulator\ above\ could\ let\ non-yield\ TCL_SUSPENDs\ pass\ through\ if\ changed\ to\n\n\ proc\ coroutine\ \{cmd\ args\}\ \{\n\ \ \ if\ \{\[catch\ \{\{*\}\$args\}\ res\ opt\ item\]\ !=\ \$::TCL_SUSPEND\}\ then\ \{\n\ \ \ \ \ \ return\ -options\ \$opt\ \$res\n\ \ \ \}\n\ \ \ switch\ --\ \[lindex\ \$res\ 0\ 0\]\ \"yield\"\ \{\n\ \ \ \ \ \ #\ This\ kind\ of\ \[suspend\]\ is\ handled\ here.\n\ \ \ \ \ \ interp\ alias\ \{\}\ \$cmd\ \{\}\ call-again\ \$cmd\ \$res\n\ \ \ \ \ \ return\ \[lindex\ \$res\ 0\ 1\]\n\ \ \ \}\ default\ \{\n\ \ \ \ \ \ #\ Other\ kinds\ get\ to\ suspend\ also\ \[coroutine\],\n\ \ \ \ \ \ #\ backing\ up\ to\ the\ \[catch\]\ above.\n\ \ \ \ \ \ lappend\ res\ \$item\n\ \ \ \ \ \ return\ -code\ suspend\ \$res\n\ \ \ \}\n\ \}\n\n\n>\ The\ thing\ I\ implemented\ is\ 100%\ compatible,\ extensions\ that\ do\ not\ adapt\ keep\ on\ working\ but\ may\ block\ suspending.\ There\ is\ an\ api\ for\ extenders\ to\ adapt.\n\nNo\ big\ difference\ there,\ I'd\ say.\n\n>\ This\ is\ also\ one\ of\ the\ reasons\ why\ you\ have\ to\ \"declare\"\ a\ coroutine:\ you\ have\ to\ prepare\ the\ infrastucture\ to\ be\ able\ to\ suspend\ a\ \"thread\"\ (?),\ which\ may\ or\ may\ not\ mean\ returning\ to\ the\ top\ level.\n>\n>\ Your\ ideas\ would\ still\ fail\ to\ produce\ coroutines\ that\ can\ migrate\ accross\ threads,\ at\ least\ without\ big\ changes\ in\ the\ core:\ the\ problem\ is\ that\ the\ same\ interp\ cannot\ be\ run\ in\ different\ threads.\ The\ same\ problem\ as\ the\ ones\ in\ head.\n\nThread\ migration\ would\ be\ done\ via\ serialization,\ as\ explained\ above.\ Reconstructing\ the\ state\ of\ a\ proc\ from\ string\ rep\ in\ another\ thread\ may\ seem\ like\ a\ hazardous\ task,\ but\ probably\ no\ more\ so\ than\ in\ the\ original\ interp.\ In\ particular,\ I\ suspect\ one\ must\ be\ able\ to\ handle\ changes\ in\ the\ bytecode\ for\ the\ procedure\ one\ is\ resuming,\ since\ redefenitions\ of\ other\ command\ may\ well\ have\ triggered\ a\ recompilation,\ can\ they\ not?\n\nAre\ you\ otherwise\ saying\ that\ NRE\ is\ employing\ threads\ to\ do\ its\ work?\ Or\ did\ you\ assume\ \[suspend\]\ would?\ (It\ doesn't.)\n\n>\ What\ I\ implemented\ relies\ on\ the\ NRE\ execution\ model:\ (newer,\ adapted)\ C\ code\ never\ calls\ an\ eval,\ it\ rather\ queues\ a\ callback\ with\ enough\ info\ do\ any\ post-processing\ or\ cleanup\ it\ needs\ to\ do\ after\ the\ evaluation,\ schedules\ the\ new\ evaluation\ and\ returns\ TCL_OK.\ There\ is\ a\ trampoline\ that\ runs\ the\ evaluation\ and\ then\ the\ callbacks.\ If\ a\ C\ function\ actually\ does\ call\ an\ eval\ using\ the\ regular\ API,\ trying\ to\ suspend\ before\ it\ returns\ will\ raise\ an\ error.\n>\n>\ That\ means\ that\ no\ C\ function\ is\ ever\ suspended:\ it\ was\ implemented\ as\ a\ two-part\ function\ to\ begin\ with.\ Even\ though\ your\ email\ got\ truncated,\ I\ imagine\ this\ is\ not\ too\ different\ in\ spirit\ to\ the\ two-entry-point\ system\ you\ were\ about\ to\ describe?\n\nSlightly\ different\ in\ spirit,\ but\ perhaps\ not\ in\ nature.\ In\ order\ to\ make\ a\ typical\ resumeProc\ for\ Tcl_CreateSuspendableCommand,\ one\ should\ be\ able\ to\ just\ take\ the\ corresponding\ proc\ and\ add\ \"spaghetti\"\ for\ jumping\ to\ the\ place\ where\ resuming\ requires\ one\ to\ be\ --\ provided\ that\ one\ has\ the\ \"stomach\"\ for\ such\ violations\ of\ Structured\ Programming\ dogma.\ By\ contrast,\ the\ NRE\ API\ of\ callbacks\ one\ can\ add\ rather\ suggests\ that\ the\ proc\ should\ be\ transformed\ into\ an\ automaton.\ It\ is\ possible\ that\ any\ resumeProc\ obtained\ through\ adding\ spaghetti\ will\ also\ be\ possible\ to\ operate\ as\ an\ automaton\ through\ the\ Tcl_NRAddCallback\ interface,\ in\ which\ case\ the\ two\ would\ indeed\ not\ be\ different\ in\ nature,\ but\ I\ haven't\ explored\ the\ design\ in\ that\ much\ detail.\n\n======\n\n\n\n----\n!!!!!!\n%|\ \[Category\ Suggestions\]\ |\ \[Category\ Control\ Structure\]\ |%\n!!!!!!} CALL {my revision {suspend and resume}} CALL {::oo::Obj1875490 process revision/suspend+and+resume} CALL {::oo::Obj1875488 process}

-errorcode

NONE

-errorinfo

Unknow state transition: LINE -> END
    while executing
"error $msg"
    (class "::Wiki" method "render_wikit" line 6)
    invoked from within
"my render_$default_markup $N $C $mkup_rendering_engine"
    (class "::Wiki" method "render" line 8)
    invoked from within
"my render $name $C"
    (class "::Wiki" method "revision" line 31)
    invoked from within
"my revision $page"
    (class "::Wiki" method "process" line 56)
    invoked from within
"$server process [string trim $uri /]"

-errorline

4