The [comp.lang.tcl%|%clt%|%] thread, "[http://groups.google.com/group/comp.lang.tcl/browse_frm/thread/6a6853a37b1d95a6%|%Tcl_ObjSetVar2..., Martin Lembug, 2005-10-12]", and [https://sourceforge.net/tracker/?func=detail&atid=110894&aid=1334947&group_id=10894%|%Bug #1334947%|%] revived the discussion of how to properly manage the reference count of [Tcl_Obj%|%Tcl_Objs]. This is an attempt at clarifying the issue, and a roadmap to improving both the management of reference counts in the [The Tcl core%|%core implementation] as well as the documentation of the relevant API functions. ** See Also ** [routines safe for zero-ref objs]: Enumerates routines that have this quality. [Tcl_Obj]: The structure that holds Tcl values. [Managing the reference count of Tcl objects]: When to and when not to increment and decrement reference counts. ** Description ** Some functions return a new `Tcl_Obj` with a reference count of `0`, which indicates that the caller now owns the Tcl_Obj, and should either store it or decrement its reference count to free it. A function that receives a Tcl_Obj with a reference count of `0` now owns that object, and should either increment the reference count and store it, or decrement the reference count to free it. `Tcl_ObjSetVar2()` is an example of such a function. If an internal representation for a Tcl_Obj contains a reference-counted pointer to itself, the reference count never drops to `0`, which is a memory leak. To avoid this, the manager of an internal representation must make sure that the following is true for each Tcl_Obj it stores a points to: The referenced object is not the object that is storing it.: The referenced object is unshared.: This avoids subsequent generation of an internal representation that might contain pointer to the storing object. The internal representation of the refeenced object does not contain a reference-counted pointer to containing object.: One easy way to do this is to free the internal representation of the referencing object before storing a pointer to it (ensure that there is a string representation first). ** Categorization ** There are five classes of library routines regarding Tcl_Obj reference counts: Constructor: Returns a fresh Tcl_Obj with a reference count of `0`. Reader: Accesses the Tcl_Obj without modifying the reference count, but may modify the internal representation. Consumer: Stores a new reference to an existing Tcl_Obj and increments the reference count, which is decremented later when the storage is cleared. An example is `newValuePtr` of `Tcl_ObjSetVar2`. Bouncer: Might temporarily increment reference count of an object and then decrement it again before returning. If a Tcl_Obj with a reference count of `0` is passed to such a routine, it might get freed. Examples are `part1Ptr` and `part2Ptr` of `Tcl_ObjSetVar2()`. A caller can protect the Tcl_Obj if needed by incrementing its reference count before making the call, and decrementing it afterwards (which makes the caller a bouncer too). Mutator: Changes the Tcl_Obj and can '''only''' be used with unshared Tcl_Objs (reference count == 0 or 1). '''As a general rule, all Tcl commands should be considered to be bouncers of the objects in the objv array.''' In other words, when constructing an objv array to pass to such commands, the objects in objv must have non-zero reference counts; in particular, newly allocated Tcl_Obj that are passed through objv must have had Tcl_IncrRef called on them, with a matching Tcl_DecrRef after the call returns. We hope to improve the documentation of the categorization of the different functions, and also to significantly reduce the population of bouncers. Constructors and Mutators are already documented as such. ** Rules for dealing safely with the different categories ** Note first that '''Constructors''' are not an issue: there is no Tcl_Obj to manage before you call them. The '''always-safe rules''' are: * '''Mutators''': pass an unshared object (refCount is 0 or 1). In order to respect ''copy-on-write'' semantics, make a copy for your use if you need to modify a shared object, and modify the copy. * '''Readers''', '''Consumers''' and '''Bouncers''': Tcl_IncrRefCount(objPtr) before calling the library function, Tcl_DecrRefCount(objPtr) on return. This means: assume that every function in the Tcl library that is not a Mutator is a Bouncer. The '''optimal rules''' in terms of performance and code simplicity (but risky in light of incomplete documentation) are: * '''Mutators''': pass an unshared object (refCount is 0 or 1). In order to respect ''copy-on-write'' semantics, make a copy for your use if you need to modify a shared object, and modify the copy. Use ''Tcl_IsShared'' to determine whether you may modify, and ''Tcl_DuplicateObj'' to get the copy. * '''Readers''': if you pass an object with refCount==0, make sure to Tcl_DecrRefCount(objPtr) on return in order not to leak the object. * '''Consumers''': do not worry about reference counts as the consumer takes care of it, including the freeing of unneeded objects. This is ''fire-and-forget''. Passing a fresh Tcl_Obj* to a consumer means you're through using it. * '''Bouncers''': Tcl_IncrRefCount(objPtr) before calling the library function, Tcl_DecrRefCount(objPtr) on return. [PYK] 2017-12-30: Another category is '''Accessors''', which return a reference to an existing `Tcl_Obj`. The reference count of such an object should be incremented before passing the object to a function that might decrement it. In [https://core.tcl.tk/itcl/artifact?ln=362&name=7e544408dac0c412%|%the following example] from [incr Tcl], the result of `Tcl_GetObjResult` is passed directly to `Tcl_GetObjectFromObj`, which, in case of an error, passes the object to `Tcl_SetObjResult`, which in turn frees it, but then accesses `bytes` member of the argument it was passed, which happens to be the same object it just freed: ======c Tcl_GetObjectFromObj(interp, Tcl_GetObjResult(interp)); ====== This is a form of [https://en.wikipedia.org/wiki/Aliasing_(computing)%|%aliasing], where a routine ends up with two different handles to the same `Tcl_Obj`. To avoid errors in this case, increment the reference count before passing the `Tcl_Obj` to another routine: ======c resPtr = Tcl_GetObjResult(interp); Tcl_IncrRefCount(resPtr); Tcl_GetObjectFromObj(interp,resPtr); Tcl_DecrRefCount(resPtr); ====== In [https://core.tcl.tk/itcl/artifact?ln=727-730&name=b13d401e71090091%|%this example] from [itcl], a new `Tcl_Obj` with a reference count of 0 is passed to `Tcl_DictObjPut`, which may or may not increment the reference count: ======c objPtr = Tcl_NewStringObj("vars", -1); Tcl_DictObjRemove(interp, mapDict, objPtr); Tcl_DictObjPut(interp, mapDict, objPtr, infoPtr->infoVars4Ptr); Tcl_DecrRefCount(objPtr); ====== Instead, the reference count should first be incremented: ======c objPtr = Tcl_NewStringObj("vars", -1); Tcl_IncrRefCount(objPtr); Tcl_DictObjRemove(interp, mapDict, objPtr); Tcl_DictObjPut(interp, mapDict, objPtr, infoPtr->infoVars4Ptr); Tcl_DecrRefCount(objPtr); ====== ** Tcl Does Not Garbage Collect ! ** I have been trying to understand clearly the rules for Tcl reference counting of objects, and how to properly use the increment and decrement ref count operations. I finally came across a question and answer posting elsewhere, in which Donal K. Fellows clearly explains a very, very, VERY important concept: '''Tcl Does Not Garbage Collect!''' What does this mean for the Tcl extension writer? If you create an object, and you never pass it back to Tcl as part of another object (eg a list object), or as the result object, it will not get freed. You MUST call `Tcl_DecrRefCount()` because this is where the memory deallocator gets called... and nowhere else! It goes without saying that for a very experienced programmer such as myself to have to hunt around for this morsel of information means that the documentation for Tcl Objects in the Tcl C reference is ''not explicit enough'' in making this fact crystal clear. It should also be pointed out that it is perfectly OK to call `Tcl_DecrRefCount()` to free an allocated object without first having called `Tcl_IncrRefCount()`. The deallocation will happen if the reference count is zero ''or negative''. This can be the case if you must create a new object for the sole purpose of passing it as an argument to another Tcl API, but then have no further use for it. If that bit of code is called over and over, you will end up with many cats and hats wandering in the woods outside Mr. Tesla's mountain laboratory, homeless. I hope someone finds this clarification useful. [MS]: disagrees strongly with ''is perfectly OK'' - at least in the unqualified version above! If you created an object without calling `Tcl_IncrRefCount()`, and passed it somewhere, calling `Tcl_DecrRefCount()` on it is possibly disastrous: If some part of Tcl kept a reference to it you will be removing it and freeing the object, which will cause memory corruption further down the line when the reference count is decremented by the rightful owner of the reference! Similarly, if some part of Tcl did an incr/decr of the refCount, the object will already be free when you call Tcl_DecrRefCount() so that your call causes memory corruption. That advise is '''ONLY''' correct if you never ever pass the Tcl_Obj anywhere else. I also disagree (less emphatically) about it being difficult to find that Tcl does not garbage collect. It is never implied that it does; if you have to hunt around for such a morsel of information in the C documentation you will also spend a lot of time. It is even harder to find the morsel of information that Tcl will not wash your dishes. Then again, neither will C, C++, Java or C# - and the docs keep absolute silence about '''that'''! ** Page Authors ** Joe English: Wrote up one of the original messages categorizing library routines by effect on reference counting. [dkf%|%Donal Fellows]: Added the category '''hairy monsters''', which later became '''bouncers'''. [pyk%|%Poor Yorick]: Renamed '''hairy monster''' to '''bouncer'''. <> Internals