''[APW] 2007-06-06'' Some time ago I started efforts to reimplement [incr Tcl] on top of [tclOO] [http://tip.tcl.tk/257]. I started based on [tclOO], using only the [API] functions of [tclOO] and a lot of additonal code in [C]. With that approach I had about 90% of the test suite of [incr Tcl] running successfully, but I saw, after all that, I was only using a small part of [tclOO] code. My second approach was implementing a [TMOS]-like [http://www.tcl.tk/cgi-bin/tct/tip/279.html] extension using a lot of code from the [XOTcl] reference implementation and again achieved about 90% of the test suite of [incr Tcl] running successfully. I felt that the code was too big and too complicated, partially because of having all these XOTcl features running. Additionally there were some features of [incr Tcl] which would have needed a lot of redesign of that code to pass the [incr Tcl] test suite. So I decided to come back to the [tclOO] implementation, thinking that it is, after all, the best starting point. This time I changed/extended the [tclOO] code directly so as to be able to use as much as possible of that code, because I had the feeling it has a lot of good implemented features, which I did like to use. I also did like the implementation of the dispatch and [class]/[object] creation mechanism much more than the one in the TMOS implementation. I started implementing [extension]s to tclOO (directly in [tclOO], as the [API] is not enough) to get a lot of features of [incr Tcl] to run. This includes a lot of code implemented in [pure-Tcl] on top of [tclOO]. Doing that, I learned a lot more details and "specialities" about [incr Tcl]. Currently I have an implementation which is successfully running about 70% of the [incr Tcl] test suite. Having now a better feeling about what is needed for [incr Tcl], I will start to write down the missing tclOO features and my solutions for them, in the hope that there will be some comments (especially from [dkf]). These comments could also include suggestions on how to solve the problem in a different way, without having to modify tclOO. In the following, ''solution'' means that is my current (not always optimal) solution for the problem. In the code examples I am using a meta syntax, not the [incr Tcl] or [tclOO] syntax, to make the code shorter! Here starts the list; I will add topics as they come up: ---- what: '''[Namespace] names for [incr Tcl]''' problem: [incr Tcl] expects that the namespace of a class is the same as the class name solution: Add an option -withnamespace or a [method] createwithnamespace [DKF]: This is actually a serious problem, since the current design of the system specifically decouples the namespace and the class (i.e., object) name. ''[APW] 2007-06-11'': I saw that [DKF] is working on that. ---- what: '''calling of [method]s without''' ''my'' '''in front of the method name''' problem: [incr Tcl] resolves command calls first within the class hierarchy before using the normal Tcl rules solution: Add a [namespace unknown] command as a dispatcher for [incr Tcl] methods [DKF]: Good solution. [MJ]: Won't this lead to problems if you define commands with the same names as [incr Tcl] commands in the global namespace? In that case [namespace unknown] will not trigger. [DKF]: Good point. In that case, better to use [namespace path] and [interp alias]. ''[APW] 2007-06-10'': [namespace path] alone will not work, as the methods are "hidden" using [tclOO], if I understand that correctly, but at the moment I am using a "namepace command resolver" and that works (maybe even faster as [interp alias] if there are a lot of methods?) ---- what: '''access to [class] [variable]s''' problem: [incr Tcl] has 3 protection levels which guard the access to class variables in the class hierachy. solution: I use the [apply] command for constructing the body parts for the class methods (and [[constructor]]s and [[destructor]]s) after the complete class has been parsed and all classes in the hierarchy are known. There is an additional parameter, ''self,'' added automatically to the arguments of the method, and the variables of all the classes in the hierarchy are mapped using namespace upvar. Variables in [incr Tcl] have to live in the namespace where they are defined as there might be the same variable name in different classes in the hierarchy even with different protection (private, protected and public). In generating the namespace upvar commands the private variables of inherited classes are skipped. [DKF]: implement the ''self'' variable by prepending each method body with: set self [self] ''[APW] 2007-06-10'': looks ok, I will try that. ---- what: '''[incr Tcl] rules for calling/traversing the constructors and destructors''' problem: The rules for traversing the constructors/destructors in [incr Tcl] are very "special" and cannot be mapped to the model in tclOO. Example: class foo { constructor {args} { puts foo } } class bar { constructor {args} { puts bar } } class foobar { superclass foo bar constructor {args} { puts foobar } } class geek { constructor {args} { puts geek } } class mongrel { superclass foobar geek constructor {args} { puts mongrel } } output: geek bar foo foobar mongrel In other words [incr Tcl] walks through the list of superclasses starting at the end based on the definition order. It looks for inherited classes up to the root class and unrolls the built stack. Then for the next class in inheritance, the same is done. solution: New command: "invokeitclconstructors" similar to the [[next]] command in [tclOO] (to be done: add details here) [DKF]: The [[next]] command, if put ''first'', causes the constructors to be walked in the desired order. % package req TclOO; namespace path oo % class create foo { constructor {} {next; puts foo} } ::foo % class create bar { constructor {} {next; puts bar} } ::bar % class create foobar { superclass foo bar; constructor {} {next; puts foobar}} ::foobar % class create geek {constructor {} {next; puts geek}} ::geek % class create mongrel {superclass foobar geek; constructor {} {next; puts mongrel}} ::mongrel % [mongrel new] destroy geek bar foo foobar mongrel [DKF]: Discussion seems to indicate that there is a deeper problem here, but not one I understand yet. [APW]: I need ''info class superclasses ...'' on a lot of places for [incr Tcl] and it returns the class names not in the order the have been defined, so I am calling class create mongrel {superclass geek foobar; constructor {} {next; puts mongrel}} to get the order as expected in [incr Tcl], for the [incr Tcl] input: itcl::class mongrel { inherit foobar geek; constructor {} {puts mongrel}} [DKF]: Ah, I don't provide any code to walk the whole superclass hierarchy and describe it. Perhaps I should; it could leverage information stored at the C level very nicely, and so be a lot more efficient than a Tcl-implemented version. ---- what: '''[incr Tcl] special init code script in constructors''' problem: In [incr Tcl] constructors it is possible to define an init script, which is invoked directly before the code in the constructor, but after the object is instantiated. Example: The script after the arguments and before the body is the "constructor init code" class foo { constructor {args} { puts foo } } class bar { constructor {args} { puts bar } } class foobar { superclass foo bar constructor {args} { foo::constructor ; # this is the init code which calls the constructor of class foo } { puts foobar } } class geek { constructor {args} { puts geek } } class mongrel { superclass foobar geek constructor {args} { foobar::constructor } { puts mongrel } } output: foo bar foobar geek mongrel solution: Add the init code script in building the apply code for the constructors [DKF]: Definitely sounds like you need a custom '''constructor''' command. It might even be easier to not use TclOO constructors at all and instead define your own scheme using multiple methods (this is what you'd have to do if doing [XOTcl] on top of TclOO). ''[APW] 2007-06-10'': Will have to live with that. ---- what: '''[incr Tcl] command [itcl::body]''' problem: In [incr Tcl], there are several ways to partially define a method: protection + name, protection + name + params, protection + name + params + body. That method can be redefined later on, but the params list for the method - if defined in the class definition - has to be compatible. solution: Add additional information about method params and status to each method definition in the tclOO internal structures. There is an additional problem in that tclOO cannot handle partially defined methods, the solution to which is adding this functionality. [DKF]: I thought about having abstract methods - early drafts had it in - but in the end I took it out. I reasoned that it would be easier to implement such things on top of a basic core than it would be to figure out how to do it directly. ''[APW] 2007-06-10'': Have a solution for that using dicts (see also "info" problem) ---- what: '''[incr Tcl] info command''' problem: Getting the original body of a method when using apply internally solution: Save the original body in the tclOO internal structures of a method. In general I am generating a method '''info''' within each class. That is like a class proc in [incr Tcl] syntax. It is something like a singleton method and it can be called without an object being instantiated. '''info''' first trys the [incr Tcl] specific commands, and if that does not work, it forwards to the ::info command. [DKF]: I think this sort of thing should be fairly simple to hang off the metadata mechanism. ''[APW] 2007-06-10'': In parsing the [incr Tcl] classes I put all the info in a class specific dict, so that the "info" command can use that information (solves also the [::itcl::body] problem.) ---- what: '''[incr Tcl] singelton methods''' problem: In [incr Tcl] it is possible to have class methods which are like singletons. They can be called uisng the class or an instantiated object. Nevertheless, if they have protection level ''protected'' they can be only called from within the class namespace. That will not work with the current tclOOCall.c implementation, as it just calls unknown and also if that is a user defined method, there is not enough information available for specific diagnostics. solution: Add some code in AddMethodToCallChain, to do some additional checks and use a class specific call back to report the appropriate error. [DKF]: This is like [Java]'s class methods? If so, try this: * Put them as procedures in the class's namespace (you won't be treating the class as an object from the [incr Tcl] view) * Use [namespace path] to splice in (but be aware that there is already an existing path) ''[APW] 2007-06-10'': See my suggestion concerning a "callback" below. ---- what: '''[incr Tcl] method calls and protection violation errors''' problem: In [incr Tcl] it is necessary to generate specific error messages when a protection violation is detected. That will not work with the current tclOOCall.c implementation. solution: Add some code in AddMethodToCallChain, to use a class-specific call back in that case, which can be set via a setprotectioncallback command, to report the appropriate error. [DKF]: I'm thinking that this would perhaps be better better done through a custom method type. One of the key things I did with TclOO is to provide a general method implementation system, allowing many different implementations to be used within a single framework. As far as I'm aware, this is quite different to how Tcl-based OO systems have worked in the past. Maybe I'm wrong (it depends on whether the protection applies to the method implementation or to the interface) but in that case I'd need to understand exactly what is going on. As a side note, it has been my policy in TclOO to make non-callable methods invisible (except through [my]) as opposed to throwing errors when you try to call them. ''[APW] 2007-06-10'' - Throwing errors where calling non-callable methods is exactly what [incr Tcl] expects. :;) [DKF]: Then the methods are "exported". They just do something (based on [self caller]?) to decide after being called whether to throw an error. [APW]: yes in that direction. [incr Tcl] wants to make a difference: if a non-visible method is called the caller gets an error message like "this is a protected/private method ..." instead of getting and "unknown ..." error. ''[APW] 2007-06-10'': see also "callbacks" below. ---- what: '''Calling [incr Tcl] methods''' problem: In [incr Tcl] when calling a method it is expected that the method runs in the class namespace with the variables of the class and all inherited (and accessible) variables to be mapped into that namespace temporarly for the call. solution: In creating classes allow a parameter namespace for the AllocObject method, which is NULL for the "normal" [tclOO] system and is != NULL for generating classes for [incr Tcl] (that needs also changes for "class create"). This namespace is used later on in InvokeProcedureMethod if != NULL, which fits for [incr Tcl] calling methods and also to allow the class to have the same namespace as the class name (see also what: ''[Namespace] names for [incr Tcl]'') if (tobjPtr->selfClsPtr->thisPtr->namespacePtr != NULL) { /* use the classes namespace if it exists for example for [incr Tcl] */ namespacePtr = tobjPtr->selfClsPtr->thisPtr->namespacePtr; } else { namespacePtr = tobjPtr->namespacePtr; } The variables of the different classes are mapped (if accessable) using the "namespace upvar" command (that code is inserted in the body of a method at the very beginning), when the class is completely parsed. The path for that "namespace upvar" command is handled using the self variable in the method parameters (see above). This allows the methods to be compiled once only for performance on the very first call to a class method. [DKF]: This feels very much like something best achieved through custom method types. All this stuff can be controlled easily at that level. ''[APW] 2007-06-10'' - But then I will also need an own InvokeProcedureMethod function, so most of the method handling is the done without [tclOO] :-( . [DKF]: I'm open to actually hosting support code so that you don't need to worm yourself in deeply to the guts of Tcl (the IPM implementation is non-trivial since it splices itself into the heart of [proc] at a point that didn't exist before 8.5). ''[APW] 2007-06-10': I have seen the differences in IPM , which are necessary for [info frame] command, I am a little bit familar with the part, as I have played around and debugged quite some time in that function and the called functions in the past to get my code working, so problem with IPM seems to be not so big for me. I especially had problems with the old (before [info frame]) error handling, as I had invented some new flags for [incr Tcl] which made the old implementation core dump. RenderDeclarerName etc. look much more readable for me, and you directly see what happens here. Had also to look into TclProcCompileProc and TclObjInterpProcCore for the same reason, so I have an approximate idea what's going on there. ---- ---- ''[APW] 2007-06-09'' - @[DKF]: after spending some thoughts on your answers I have the feeling there will not much I can use directly from [tclOO]. Using custom method types, I will need my own InvokeProcedureMethod and then I can directly pass all the needed data via clientData to my InvokeProcedureMethod. So there will be no further need to use the metadata mechanism, as that would be more overhead to call it. For the info stuff I will nevertheless need my own code, as it is so different to [tclOO] info. For constructors and destructors - as you suggested - I will need my own code. To handle protection violation - as you suggested - I will need my own code. So what I can use from [tclOO] would mostly be creation, administration and deletion of classes and objects. I cannot use tclOOCall.c, because I need additional information (and the special code to provide in there the checking of protection violation), I will not need tclOODefineCmds.c, as I directly call the [tclOO] API functions, I will not need tclOOInfo.c (see above). That was not my intention! I would like to use much more of it! I like the implementation of tclOOCall.c, but at least for [incr Tcl] (maybe also for others) it is too limited at the moment. Do you have any better ideas? [DKF]: Right now, I don't understand enough to know exactly what additional APIs are needed. This is a large part of why we (myself and others) decided that TclOO had to go for a round of usage as an extension rather than going straight in as part of [Tcl]. I want to understand how to fix these problems; I've just not yet understood. ''[APW] 2007-06-10'': In the meantime I have looked for a different solution for some of the problems I have. I have put all the stuff needed for "info" into some dicts (in Tcl), so that part (if there are not big problems) is no longer a problem. For the [incr Tcl] calling problems (without my in front) I have today implemented a namespace command resolver, which also avoids the problems I had with namespace unknown. I am building dicts for method access per class including all the superclasses and have access to that information in the C-part (really cool what is able with dicts). What I am not sure is if dicts are about equally fast as using the class and superclasses with the hash tables. I am using the dicts as the hashtables for methods and the lists for superclasses are not in the API. So after all, I think the "big" problems are the names of the namespaces and the callback mechanism below and the a possibility for using the class namespace in InvokeProcedureMethod instead of the object namespace. ---- ''[APW] 2007-06-10'' - @[DKF]: "callbacks" what about the following: you call the following code in tclOOCall.c in function AddMethodToCallChain only if there is no other callback function in the object (oPtr) of that function. /* * Enforce real private method handling here. We will skip adding this * method IF * 1) we are not allowing private methods, AND * 2) this is a private method, AND * 3) this is a class method, AND * 4) this method was not declared by the class of the current object. * * This does mean that only classes really handle private methods. This * should be sufficient for [incr Tcl] support though. */ if (!(contextPtr->flags & PRIVATE_METHOD) && (mPtr->flags & PRIVATE_METHOD) && (mPtr->declaringClassPtr != NULL) && (mPtr->declaringClassPtr != contextPtr->oPtr->selfCls)) { return; } and you add a functionality to register a callback on a per object basis which either returns NULL, which is the case of "return;" above or you get back a tmPtr which is either the same as before calling the callback function or a method which could for example - as in [incr Tcl] needed - handle protection violation as needed. That method in tmPtr must be a registered method for the object/the classes of the object. ---- ''[APW] 2007-06-10'': have removed the following part, which included some suggestions for the layout, as that is no longer necessary. ---- [[ [Category Discussion] | [Category Itcl] | [Category Object Orientation] ]]