tclOO missing features for Itcl

APW 2007-06-06 Some time ago I started efforts to reimplement incr Tcl on top of tclOO [L1 ]. I started based on tclOO, using only the API functions of tclOO and a lot of additonal code in C. With that approach I had about 90% of the test suite of incr Tcl running successfully, but I saw, after all that, I was only using a small part of tclOO code.

My second approach was implementing a TMOS-like [L2 ] extension using a lot of code from the XOTcl reference implementation and again achieved about 90% of the test suite of incr Tcl running successfully. I felt that the code was too big and too complicated, partially because of having all these XOTcl features running. Additionally there were some features of incr Tcl which would have needed a lot of redesign of that code to pass the incr Tcl test suite.

So I decided to come back to the tclOO implementation, thinking that it is, after all, the best starting point. This time I changed/extended the tclOO code directly so as to be able to use as much as possible of that code, because I had the feeling it has a lot of good implemented features, which I did like to use. I also did like the implementation of the dispatch and class/object creation mechanism much more than the one in the TMOS implementation.

I started implementing extensions to tclOO (directly in tclOO, as the API is not enough) to get a lot of features of incr Tcl to run. This includes a lot of code implemented in pure-Tcl on top of tclOO. Doing that, I learned a lot more details and "specialities" about incr Tcl.

Currently I have an implementation which is successfully running about 70% of the incr Tcl test suite.

Having now a better feeling about what is needed for incr Tcl, I will start to write down the missing tclOO features and my solutions for them, in the hope that there will be some comments (especially from dkf). These comments could also include suggestions on how to solve the problem in a different way, without having to modify tclOO.

In the following, solution means that is my current (not always optimal) solution for the problem.

In the code examples I am using a meta syntax, not the incr Tcl or tclOO syntax, to make the code shorter!

Here starts the list; I will add topics as they come up:

what: Namespace names for incr Tcl
problem: incr Tcl expects that the namespace of a class is the same as the class name
solution: Add an option -withnamespace or a method createwithnamespace

DKF: This is actually a serious problem, since the current design of the system specifically decouples the namespace and the class (i.e., object) name.

APW 2007-06-11: I saw that DKF is working on that.

DKF: Yeah. I've done it at the C level, but I'm still considering how to expose it to scripts.

DKF: Exposed to scripts as a hidden method of class: createWithNamespace.

APW 2007-06-11: That forces me to start over with your new version, and build my stuff on top of it, as there seems to be a solution for almost all problems, I had before. I will report, if that is done. The missing points are desribed at the end (nothing new right now).

what: calling of methods without my in front of the method name
problem: incr Tcl resolves command calls first within the class hierarchy before using the normal Tcl rules
solution: Add a namespace unknown command as a dispatcher for incr Tcl methods

DKF: Good solution.

MJ: Won't this lead to problems if you define commands with the same names as incr Tcl commands in the global namespace? In that case namespace unknown will not trigger.

DKF: Good point. In that case, better to use namespace path and interp alias.

APW 2007-06-10: namespace path alone will not work, as the methods are not in the namespace command table using tclOO, but at the moment I am using a "namepace command resolver" (written in C using Tcl_SetNamespaceResolver) and that works (maybe even faster as interp alias if there are a lot of methods?). Would be helpfull, if I could get then an API access to PublicObjectCmd and PrivateObjectCmd.

what: access to class variables
problem: incr Tcl has 3 protection levels which guard the access to class variables in the class hierachy.
solution: I use the apply command for constructing the body parts for the class methods (and [constructor]s and [destructor]s) after the complete class has been parsed and all classes in the hierarchy are known. There is an additional parameter, self, added automatically to the arguments of the method, and the variables of all the classes in the hierarchy are mapped using namespace upvar. Variables in incr Tcl have to live in the namespace where they are defined as there might be the same variable name in different classes in the hierarchy even with different protection (private, protected and public). In generating the namespace upvar commands the private variables of inherited classes are skipped.

DKF: implement the self variable by prepending each method body with:

 set self [self]

APW 2007-06-10: looks ok, I will try that.

what: incr Tcl rules for calling/traversing the constructors and destructors
problem: The rules for traversing the constructors/destructors in incr Tcl are very "special" and cannot be mapped to the model in tclOO. Example:

  class foo {
      constructor {args} { puts foo }
  }
  class bar {
      constructor {args} { puts bar }
  }
  class foobar {
      superclass foo bar
      constructor {args} { puts foobar }
  }
  class geek {
      constructor {args} { puts geek }
  }
  class mongrel {
      superclass foobar geek
      constructor {args} { puts mongrel }
  }

  output: geek bar foo foobar mongrel

In other words incr Tcl walks through the list of superclasses starting at the end based on the definition order. It looks for inherited classes up to the root class and unrolls the built stack. Then for the next class in inheritance, the same is done.

solution: New command: "invokeitclconstructors" similar to the [next] command in tclOO (to be done: add details here)

DKF: The [next] command, if put first, causes the constructors to be walked in the desired order.

 % package req TclOO; namespace path oo
 % class create foo { constructor {} {next; puts foo} }
 ::foo
 % class create bar { constructor {} {next; puts bar} }
 ::bar
 %  class create foobar { superclass foo bar; constructor {} {next; puts foobar}}
 ::foobar
 % class create geek {constructor {} {next; puts geek}}
 ::geek
 % class create mongrel {superclass foobar geek; constructor {} {next; puts mongrel}}
 ::mongrel
 % [mongrel new] destroy
 geek
 bar
 foo
 foobar
 mongrel

DKF: Discussion seems to indicate that there is a deeper problem here, but not one I understand yet.

APW: I need info class superclasses ... on a lot of places for incr Tcl and it returns the class names not in the order the have been defined, so I am calling

  class create mongrel {superclass geek foobar; constructor {} {next; puts mongrel}}

to get the order as expected in incr Tcl, for the incr Tcl input:

  itcl::class mongrel { inherit foobar geek; constructor {} {puts mongrel}}

DKF: Ah, I don't provide any code to walk the whole superclass hierarchy and describe it. Perhaps I should; it could leverage information stored at the C level very nicely, and so be a lot more efficient than a Tcl-implemented version.

what: incr Tcl special init code script in constructors
problem: In incr Tcl constructors it is possible to define an init script, which is invoked directly before the code in the constructor, but after the object is instantiated. Example:

  The script after the arguments and before the body is the "constructor init code"

  class foo {
      constructor {args} { puts foo }
  }
  class bar {
      constructor {args} { puts bar }
  }
  class foobar {
      superclass foo bar
      constructor {args} {
         foo::constructor ; # this is the init code which calls the constructor of class foo
      } { puts foobar }
  }
  class geek {
      constructor {args} { puts geek }
  }
  class mongrel {
      superclass foobar geek
      constructor {args} {
        foobar::constructor
      } { puts mongrel }
  }

  output: foo bar foobar geek mongrel

solution: Add the init code script in building the apply code for the constructors

DKF: Definitely sounds like you need a custom constructor command. It might even be easier to not use TclOO constructors at all and instead define your own scheme using multiple methods (this is what you'd have to do if doing XOTcl on top of TclOO).

APW 2007-06-10: Will have to live with what DKF suggests.

what: incr Tcl command itcl::body
problem: In incr Tcl, there are several ways to partially define a method: protection + name, protection + name + params, protection + name + params + body. That method can be redefined later on, but the params list for the method - if defined in the class definition - has to be compatible.
solution: Add additional information about method params and status to each method definition in the tclOO internal structures. There is an additional problem in that tclOO cannot handle partially defined methods, the solution to which is adding this functionality.

DKF: I thought about having abstract methods - early drafts had it in - but in the end I took it out. I reasoned that it would be easier to implement such things on top of a basic core than it would be to figure out how to do it directly.

APW 2007-06-10: Have a solution for that using dicts (see also "info" problem)

what: incr Tcl info command
problem: Getting the original body of a method when using apply internally
solution: Save the original body in the tclOO internal structures of a method. In general I am generating a method info within each class. That is like a class proc in incr Tcl syntax. It is something like a singleton method and it can be called without an object being instantiated. info first trys the incr Tcl specific commands, and if that does not work, it forwards to the ::info command.

DKF: I think this sort of thing should be fairly simple to hang off the metadata mechanism.

APW 2007-06-10: In parsing the incr Tcl classes I put all the info in a class specific dict, so that the "info" command can use that information (solves also the ::itcl::body problem.)

what: incr Tcl singelton methods
problem: In incr Tcl it is possible to have class methods which are like singletons. They can be called uisng the class or an instantiated object. Nevertheless, if they have protection level protected they can be only called from within the class namespace. That will not work with the current tclOOCall.c implementation, as it just calls unknown and also if that is a user defined method, there is not enough information available for specific diagnostics.
solution: Add some code in AddMethodToCallChain, to do some additional checks and use a class specific call back to report the appropriate error.

DKF: This is like Java's class methods? If so, try this:

Put them as procedures in the class's namespace (you won't be treating the class as an object from the incr Tcl view)
Use namespace path to splice in (but be aware that there is already an existing path)

APW 2007-06-10: See my suggestion concerning a "callback" below.

what: incr Tcl method calls and protection violation errors
problem: In incr Tcl it is necessary to generate specific error messages when a protection violation is detected. That will not work with the current tclOOCall.c implementation.
solution: Add some code in AddMethodToCallChain, to use a class-specific call back in that case, which can be set via a setprotectioncallback command, to report the appropriate error.

DKF: I'm thinking that this would perhaps be better better done through a custom method type. One of the key things I did with TclOO is to provide a general method implementation system, allowing many different implementations to be used within a single framework. As far as I'm aware, this is quite different to how Tcl-based OO systems have worked in the past. Maybe I'm wrong (it depends on whether the protection applies to the method implementation or to the interface) but in that case I'd need to understand exactly what is going on.

As a side note, it has been my policy in TclOO to make non-callable methods invisible (except through my) as opposed to throwing errors when you try to call them.

APW 2007-06-10 - Throwing errors where calling non-callable methods is exactly what incr Tcl expects. :;)

DKF: Then the methods are "exported". They just do something (based on self caller?) to decide after being called whether to throw an error.

APW: yes in that direction. incr Tcl wants to make a difference: if a non-visible method is called the caller gets an error message like "this is a protected/private method ..." instead of getting and "unknown ..." error.

APW 2007-06-10: see also "callbacks" below.

what: Calling incr Tcl methods
problem: In incr Tcl when calling a method it is expected that the method runs in the class namespace with the variables of the class and all inherited (and accessible) variables to be mapped into that namespace temporarly for the call.
solution: In creating classes allow a parameter namespace for the AllocObject method, which is NULL for the "normal" tclOO system and is != NULL for generating classes for incr Tcl (that needs also changes for "class create"). This namespace is used later on in InvokeProcedureMethod if != NULL, which fits for incr Tcl calling methods and also to allow the class to have the same namespace as the class name (see also what: Namespace names for incr Tcl)

  if (tobjPtr->selfClsPtr->thisPtr->namespacePtr != NULL) {
     /* use the classes namespace if it exists for example for [incr Tcl] */
     namespacePtr = tobjPtr->selfClsPtr->thisPtr->namespacePtr;
  } else {
     namespacePtr = tobjPtr->namespacePtr;
  }

The variables of the different classes are mapped (if accessible) using the "namespace upvar" command (that code is inserted in the body of a method at the very beginning), when the class is completely parsed. The path for that "namespace upvar" command is handled using the self variable in the method parameters (see above). This allows the methods to be compiled once only for performance on the very first call to a class method.

DKF: This feels very much like something best achieved through custom method types. All this stuff can be controlled easily at that level.

APW 2007-06-10 - But then I will also need an own InvokeProcedureMethod function, so most of the method handling is the done without tclOO :-( .

DKF: I'm open to actually hosting support code so that you don't need to worm yourself in deeply to the guts of Tcl (the IPM implementation is non-trivial since it splices itself into the heart of proc at a point that didn't exist before 8.5).

APW 2007-06-10: I have seen the differences in IPM , which are necessary for info frame command, I am a little bit familar with the part, as I have played around and debugged quite some time in that function and the called functions in the past to get my code working, so problem with IPM seems to be not so big for me. I especially had problems with the old (before info frame) error handling, as I had invented some new flags for incr Tcl which made the old implementation core dump. RenderDeclarerName etc. look much more readable for me, and you directly see what happens here. Had also to look into TclProcCompileProc and TclObjInterpProcCore for the same reason, so I have an approximate idea what's going on there.

APW 2007-06-11 have reorganized the "callback" topic for better understanding.

what: detecting calls to non-accessible method
problem: In incr Tcl it is necessary to give a specific error message, if a non-accessible method is called
solution: enhancing the calling mechanism of tclOO to allow installing a callback function on an per object base which is called instead of the current accessible checking code:

/*

Enforce real private method handling here. We will skip adding this
method IF
1) we are not allowing private methods, AND
2) this is a private method, AND
3) this is a class method, AND
4) this method was not declared by the class of the current object.

This does mean that only classes really handle private methods. This
should be sufficient for incr Tcl support though.

     */

    if (!(contextPtr->flags & PRIVATE_METHOD)
            && (mPtr->flags & PRIVATE_METHOD)
            && (mPtr->declaringClassPtr != NULL)
            && (mPtr->declaringClassPtr != contextPtr->oPtr->selfCls)) {
        return;
    }

This code is currently found in tclOOCall.c in function AddMethodToCallChain. I would like to have a functionality which is calling an installed (from the application user) callback instead of that code (if it exists). The logic would be: in the above code there is a return if the check for accessibility fails, the callback should return a NULL if the accessibility fails, in that case AddMethodToCallChain would also return as in the above code. Otherwise it returns a mPtr. mPtr is either the parameter which has been given to the callback as parameter if the accessibility is ok or (if desired) another mPtr, which must be the structure for a method also defined in that object/class. In that case the rest of the code is executed as before. The new method can then handle the generation of an appropriate error message if the execution comes to that point, or do whatever is wanted in that case, for example do some forwarding.

APW 2007-06-09 - @DKF: after spending some thoughts on your answers I have the feeling there will not much I can use directly from tclOO. Using custom method types, I will need my own InvokeProcedureMethod and then I can directly pass all the needed data via clientData to my InvokeProcedureMethod. So there will be no further need to use the metadata mechanism, as that would be more overhead to call it. For the info stuff I will nevertheless need my own code, as it is so different to tclOO info. For constructors and destructors - as you suggested - I will need my own code. To handle protection violation - as you suggested - I will need my own code. So what I can use from tclOO would mostly be creation, administration and deletion of classes and objects. I cannot use tclOOCall.c, because I need additional information (and the special code to provide in there the checking of protection violation), I will not need tclOODefineCmds.c, as I directly call the tclOO API functions, I will not need tclOOInfo.c (see above). That was not my intention! I would like to use much more of it! I like the implementation of tclOOCall.c, but at least for incr Tcl (maybe also for others) it is too limited at the moment. Do you have any better ideas? -- APW 2007-06-11 -- no longer relevant, there have been enhancments for tclOO and other ideas from me see below.

DKF: Right now, I don't understand enough to know exactly what additional APIs are needed. This is a large part of why we (myself and others) decided that TclOO had to go for a round of usage as an extension rather than going straight in as part of Tcl. I want to understand how to fix these problems; I've just not yet understood.

APW 2007-06-10: In the meantime I have looked for a different solution for some of the problems I have. I have put all the stuff needed for "info" into some dicts (in Tcl), so that part (if there are not big problems) is no longer a problem. For the incr Tcl calling problems (without my in front) I have today implemented a namespace command resolver, which also avoids the problems I had with namespace unknown. I am building dicts for method access per class including all the superclasses and have access to that information in the C-part (really cool what is able with dicts). What I am not sure is if dicts are about equally fast as using the class and superclasses with the hash tables. I am using the dicts as the hashtables for methods and the lists for superclasses are not in the API.

APW 2007-06-11: So after all, I think the "big" problems still remaining are the callback mechanism described above and a possibility for using the class namespace in InvokeProcedureMethod instead of the object namespace.

APW 2007-06-18: After the implementation of functionality for using the class namespace in InvokeProcedureMethod (thanks to DKF), the remaining callback problem seems to be doable by a small additional API interface, which I have suggested to DKF. I will now continue to make the implemention more complete, to see what is left over as a problem (nothing at the moment).

APW 2007-06-10: have removed the following part, which included some suggestions for the layout, as that is no longer necessary.

Category Discussion

Category Itcl

Category Object Orientation