Object Type Control Discussion

Log of chat from Mon 4 Aug 2003

 jyl        OK its 1:45 my time, I'll give it till 2:45 my time, afterwards I have to do donuts
 dkf        OK
 jyl        OK so... lets first agree on the problem: object internal reps disappear when 
        an object it used as part of some string and no other ref to the object exists, right?
 jyl        s/it used/is used/
 jyl        is that a correct description or is the problem also composed of other sub-problems
 dkf        You mean as in: set str "foo [expr {1+2}] bar"
 dkf        where the numeric rep is lost
 jyl        yes dkf thats one example
 jyl        or set obj [makeamagicobject]
 jyl        set str "$magicobject"
 jyl        x-ed
 jyl        set str "$obj foo foo foo"
 jyl        set obj blah
 jyl        now, the object in $obj is inaccessible and irretrievable even though it should be retrievable?
 jyl        dkf?
 dkf         In the following case, the rep is *not* lost:
           set magicobj [makemagicobj]
           set str "$magicobj"
           unset magicobj
        as Tcl is smart enough to just discard those quotes, but combining the object with 
        anything via string concatenation will lose the rep.
 jyl        thats why I said "$obj foo foo foo"
 dkf        In the [set str "$magicobj foo foo foo"] case, the rep is currently *definitely* lost 
        (excluding the UNICODE string internal rep, which we can ignore for this discussion as 
        it is a transparent rep anyway.)
 dkf        Now, if we use [list] instead, we can retain the rep.
 jyl        OK.. sorry but cannot resist: I think where I want to go is to replace the string built
        in object type with a special kind of list that just "looks like a string"
 dkf        The essence of TIP#126 is that the internal rep would be retrievable (with a suitable 
        [string range] command, for example) from $str
 jyl        of course this has problems such as what happens if someone modifies the part of the 
        "string" whose component is an object... mind boggling 
 <davidw>        lost where? magicobj or str?
 jyl        I have read #126 briefly but I need to understand better
 dkf        Which in turn means, contrary to the current situation, that the internal rep would need 
        to be stored in $str in the first place 
 jyl        yes, dkf
 jyl        sorry davidw not ignoring you, but u'll need to provide better context....
 jyl        dkf sounds like we're agreeing about a kind of segmented string-list combination
 dkf        Something like that.
 jyl        itd be a humungous job, certainly not for 8.5
 dkf        My idea is that a string rep should have an ordered list of non-overlapping ranges 
        that are associated with objects.
 jyl        in order to get started we need a 9.0 tag in the CVS so we can hack
 dkf        The TIP never claims to be for 8.5
 <davidw>        jyl: when you say the internal rep gets lost, are you talking about in the 
        original 'magicobj' variable, or lost in the 'str' variable?
 jyl        would this solve shimmering *entirely*?
 jyl        davidw the original magicobj
 dkf        davidw: Try this instead: set str "[makemagicobj] foo foo foo"
 <davidw>        ah, ok, that's what I thought, just wanted to be positive
 jyl        davidw TEA seems to say the internal rep should be reconstructable from the stringrep 
        but noone is doing that in practice
 dkf        It won't solve shimmering entirely, but it will make it much rarer indeed.
 jyl        dkf thx for the better example
 jyl        dkf OK lets turn to the parts of shimmering that arent covered... I'd like a *complete* 
        approach if possible
 dkf        All core objects are reconstructable right now.
 jyl        dkf I meant extensions
 dkf        (well, given the right context; many of them can't be directly converted to.)
 jyl        what parts of shimmering (with example, if possible, please) arent addressed by 
        the segmented string-list approach?
 dkf        The Tcl_Obj structure should have been called Tcl_Value, except that was already taken 
 jyl        Tcl_ObjValue
 dkf        The main part of shimmering that isn't addressed is this: 
        set foo [makeMagicObj]; llength $foo
 <davidw>        err... what's the way to printf something as an unsigned value?
 jyl        OK step through what happens there
 jyl        %ud
 <davidw>        do you have to cast it or ... what's the flag
 dkf        If something fails that, then it's not generally a valid object name.
 dkf        s/valid/nice to use/
 jyl        or more succintly the example should be
 jyl        llength [makemagicobj]
 dkf        And we should never have called the things objects.  Everyone has (misplaced) 
        expectations about objects, and only the core itself views them that way.
 <davidw>        oh right...thanks
 dkf        jyl: But that would discard the object anyway.
 jyl        dkf yep OK so I dont understand the way in which llength loses
 dkf        The main other thing needed is an "inhibit type conversion" flag in Tcl_ObjType
 dkf        The way in which llength loses is that it does an internal-rep smash on the object.
 jyl        so llength is an example of the general problem of wanting to have two or more 
        valid objtypes that are also diffrent from string
 jyl        in this case magic-object and list
 jyl        lindex would do a similar job on this value, correct?
 dkf        yes
 dkf        and yes again
 jyl        or [expr .. ] and so on... there are many examples
 jyl        all of this is based on the (often incorrect) assumption that the original intrep 
        can be reconstructed from the stringrep which can be constructed (and will be the same) 
        from any special objtype
 dkf        Most examples involve list (and theoretically dict, though that doesn't like singletons)
 jyl        s/and will be the same/& - also often an incorrect assumption/
 dkf        Most reasonable object formats are probably like this: [a-z]+\d+
 jyl        well expr (convert to number), using as an array, and ... hmm.. vaguely gesturing 
        at space, many others (  ) could do similar things
 dkf        which is distinct from the other sources of trouble: ints and floats 
        (strings are dealt with by #126 )
 jyl        well... ints and floats --> strings are handled by #126 but is the original intrep 
        saved if we do <magicalobj> -> int -> string?
 dkf        It'd be nuts to use an object rep like "0" 
 dkf        Note that intrep smash only happens when the object is successfully converted to another type.
 dkf        (Anywhere where that is not true in the core is a bug.)
 jyl        OK I getcha... people have to be perverse and make their stringrep for the object be like "17"
 dkf        exactly
 jyl        we still have to worry about construction... what happens if a string looks like the 
        stringrep of "magicalobject23", is it a real reference?
 dkf        I've no intention of babysitting someone 
 jyl        s/is it/does it become/
 dkf        "magicalobject23" isn't a reference unless it came from [makemagicalobj]
 jyl        how would you even know to go look for the real reference.. I say no to that... 
        yes I agree with what you wrote about having to come originally from [makemagicobject]
 jyl        so:
 jyl        concat magic alob ject17 ==> (string) "magicalobject17" does not make a 
        reference to the 17th magical object
 jyl        and for {set i 0} {$i < $lastmagicalobject} {incr i} {set str "magicalobject$i"} 
        does not make references to all magicalobjects created so far
 dkf        Depends.  It is possible to build a weakhashmap-alike to go from names to objects, 
        but not required. That's up to the extension.
 dkf        (by weakhashmap, I mean a hashtable that doesn't retain references to the objects, 
        just pointers, and which loses the entry for an object when the object is destroyed)
 jyl        dkf if the weakhashmap is intended,  the extension would have to provide an EXPLICIT 
        operation to take a stringrep and convert it to a ref
 jyl        just by consing the name you dont automatically get a reference, right?
 dkf        The main complexity is what to do if an entry is deleted when the weakhashmap is being searched 
 jenglish        You should.
 dkf        The only safe way to convert a pure string back to a reference is by a table search, of course.
 jyl        dkf that is up to the extension what to do about consistency issues like that
 jenglish        [join [list "magica" "lobje" "ct17"] ""] should be utterly 
                indistinguishable from "magicalobject17".
 miguel         jyl: is this a counterexample to your claim/expectation?
        [mig@mini mig]$ tclsh
        % open /tmp/m
        % string length [read [join [list fi le 3] {}]]
 jyl        jenglish dkf and I are going in a different direction...
 dkf        The main thing is if the reference ever really goes, the table won't preserve it.
 jenglish        I see that... not sure if I like that direction though.
 jyl        miguel and jenglish thats because I wrote the lookup code that goes from the name 
        to the object and it does the lookup inside "read" if it gets a string object 
        instead of a channel object
 dkf        The weakhashmap idea is how to add Tcl-style introspection. It's just quite a 
        bit more complex to implement.
 dkf        It's quite possible that if #126 was implemented, another TIP would be done to 
        provide canned weakhashmap support...
 jyl        the porblem with what jenglish and miguel are asking for is that currently its 
        up to each objtype implementation... do we want to provide a general facility that 
        will do it for all objtypes? something worth considering, at the least
 jyl        dkf again we're thinking alike
 jyl        Id rather solve the whole issue, dkf, i.e. enhance TIP#126 to do it all
 jyl        makes it much more likely to be accepted in the TCT imho
 dkf        I find I think even more like kbk 
 jyl        dkf explain? 
 dkf        You've not seen us talking together at the conference 
 jyl        completing each others' sentences and such? 
 jyl        ...anyways...
 dkf        I think I'd like to leave #126 (and related stuff) for the moment.  
        It feels like something to work on in a BK fork...
 dkf        jyl: Yes, we were doing that.
 jyl        I think a new objtype operation to ask an objtype to take a string and 
        convert it to a valid tcl_obj with this objtype is what we're looking for
 jenglish        Hrm... I think I disagree with the entire rationale behind TIP 126.  
                I _don't_ think we should encourage the kind of thing that TclBlend does.
 jyl        objtypes would then have to age objects much more carefully since they can 
        "come back from the dead" anytime... memory mgmt issues loom
 jyl        jenglish and e4graph ...
 dkf        the memory issues are definitely why I would want to try this in a branch.
 jyl        jenglish very soon I'm going to jump the gun and make it easy to write such 
        extensions, by releasing a library called genobject that abstracts the guts 
        of what these extensions do and makes it easy
 dkf        that and the fact that it stamps its merry way through a large fraction of the core.
 jyl        I think by forcing the shimmering issue out in the open like that is the 
        only way to get it resolved
 <davidw>        yay!
 <davidw>        go jyl!
 jenglish        You want to encourage MORE buggy extensions?
 jyl        jenglish define buggy
 dkf        At least it would be standardizing the bugginess! 
 jyl        dkf I will never again be bitten by releases such as 8.4.3 
 jenglish        Handle semantics are a well-defined, time-honored way of representing 
                stateful objects in Tcl.  If you do things that way, intrep preservation 
                becomes an efficiency issue only, not a corectness issue.
 dkf        Hmph.  Since I reckon that the intrep of an object is the exclusive preserve 
        of that object's type, I think that the hack that broke with 8.4.3 was a 
        poor hack anyway.
 dkf        OTOH, I strongly suspect that the 8.4.3 problems could be prevented with the 
        "don't convert me" flag.
 jyl        jenglish I agree with you that hanging stuff off of ptr2 is only one possible 
        approach.. another would be to map the address of the Tcl_Obj --> intrep... 
        but currently there's no way to get a callback when a Tcl_Obj reaches refcount 0... 
        if that were possible then yes, Id agree completely with you... 
        however the current state of things with Tcl_Obj makes such uses of ptr2 necessary
 jyl        dkf are you saying the extensions which exposed the 8.4.3 problems are
 jyl        "a poor hack"? or that the change in 8.4.3 was "a poor hack" 
        (dangerous smile curling around the corners of my mouth  )
 dkf        I'm saying that they are doing something deeply non-kosher
 jyl        .. but necessary?
 dkf        I think I'll reserve judgement on the necessity of that kind of thing.
 jyl        dkf tell me how to make an extension that wants to recycle its own intrep 
        when there are no more Tcl_Obj references
 dkf        It's definitely not a good idea on the core not using ptr2 for anything; 
        several types do in-fact use it (including the list, dict and subcommand types)
 dkf        In theory, it's easy.
 dkf        Obvious.
 jyl        Oh and also wants to use the object as a functor
 dkf        Only problem is that it is very susceptible to intrep loss 
        (frex, it can't be used as a command name)
 jyl        dkf thats exactly the problem solved by genobj and feather
 jyl        and independently by tclblend
 dkf        Hardly.  AFAICR, feather had the intrep smash problem.
 jyl        dkf yes, if you stick a feather object in a list, its gone
 jyl        s/stick in a list/treat as a list/
 dkf        The easiest way to fix this is a flag in Tcl_ObjType, but that can't be done in 8.*
 jyl        same with tclblend, e4graph and everyone else
 dkf        (structure and field-positions are fixed by both the core and extension code.)
 jyl        why/how would that fix things? lets say we have a flag set on the 
        objtype saying dont convert me... OK then what?
 jyl        what happens when you *do* try?
 jyl        forget 8.* we're only discussig 9.* here
 dkf        If the flag was set, the core would notice and refuse to convert the type, 
        instead of converting it to command-name type.
 jyl        whoa... so refuse to convert to type foo for whatever target foo, 
        instead do the conversion to command type? why?!
 jyl        anyways this was fun, its 2:45 my time... time's up
 dkf        Eh?  That's backward
 dkf        It's already type foo.  It just won't be converted to any core type instead.
 jyl        OK so I misunderstood... so you want to convert a type foo --> list, 
        it refuses and thats that, or it refuses and converts to command-name?
        patthoyts wanders in.
 dkf        It refuses and that's that.
 jyl        OK agreed... gets a Tcl error, I suppose
 dkf        (won't go to list *or* command-name.)
 dkf        Depends on the situation. In some cases, there might be a sensible fallback instead.
 jyl        Instead of a flag, Id like to allow conversion to command names but disallow list... possible?
 dkf        (for command-name, you just don't cache the command info. It's not a big deal. 
        For lists, you can build a singleton list around it.)
 dkf        In theory, could have several flags to describe what to do.
 jyl        that is, instead of only asking the target type to convert an object TO it 
        I'd first like to ask the source type to convert *itself* to the target type. 
        This could return one of three results: a converted object, a flat-out refusal, 
        or a defer-to-target-type
 jyl        If we get a defer, then only call the target type's convert-to method
 dkf        There's obviously things still to resolve in this area.    Now you get back to donuts!
 jyl        yeah thx Donal... that was mucho fun 
 jyl        pls record all this if you can, cut-n-paste to the wiki or something

Category Suggestions