Tcl_Obj refCount HOWTO

The clt thread, Tcl_ObjSetVar2..., Martin Lembug, 2005-10-12 , and Bug #1334947 revived the discussion of how to properly manage the reference count of Tcl_Objs.

This is an attempt at clarifying the issue, and a roadmap to improving both the management of reference counts in the core implementation as well as the documentation of the relevant API functions.

Description

Some functions return a new Tcl_Obj with a reference count of 0, which indicates that the caller now owns the Tcl_Obj, and should either store it or decrement its reference count to free it. A function that receives a Tcl_Obj with a reference count of 0 now owns that object, and should either increment the reference count and store it, or decrement the reference count to free it. Tcl_ObjSetVar2() is an example of such a function.

If an internal representation for a Tcl_Obj contains a reference-counted pointer to itself, the reference count never drops to 0, which is a memory leak. To avoid this, the manager of an internal representation must make sure that the following is true for each Tcl_Obj it stores a points to:

The referenced object is not the object that is storing it.

The referenced object is unshared.: This avoids subsequent generation of an internal representation that might contain pointer to the storing object.

The internal representation of the refeenced object does not contain a reference-counted pointer to containing object.: One easy way to do this is to free the internal representation of the referencing object before storing a pointer to it (ensure that there is a string representation first).

Categorization

There are five categories of library routines regarding the reference count of a Tcl_Obj returned by the routine:

Accessor: Returns a pointer to an existing Tcl_Obj. Normally the reference count has not been incremented to account for the pointer the caller now holds. The caller should increment the reference count when storing a pointer to the Tcl_Obj. To protect the Tcl_Obj when passing it to another function, the caller should increment its reference count before passing it, and decrement it later. Often, there does not change the reference count of the returned Tcl_Obj at all, simply reading its value and doing nothing more with it.

Constructor: Returns a fresh Tcl_Obj with a reference count of 0. The caller must arrange to ultimately decrement the reference count at an appropriate time, or pass the Tcl_Obj along to somethiing else that will make the proper arrangements.

There are five categories of library routines regarding the reference count of a Tcl_Obj passed to the routine:

Bouncer: Might temporarily increment reference count of an object and then decrement it again before returning. If a Tcl_Obj with a reference count of 0 is passed to such a routine, it might get freed. Examples are part1Ptr and part2Ptr of Tcl_ObjSetVar2(). To protect a Tcl_Obj passed to a bouncer, increment its reference count prior to passing it, and decrement it later.

Deleter: May decrement the reference count of a Tcl_Obj after removing a pointer to it from structure. To protect a Tcl_Obj passed to a deleter, increment its it before passing it, and decrement it later.

Mutator: Changes the Tcl_Obj value. Can only be used with an unshared Tcl_Obj (reference count == 0 or 1). When writing a mutator, prior to modifyig the value of a Tcl_Obj, conform to copy-on-write semantics by using Tcl_DuplicateObj() to make an unshared copy of the Tcl_Obj if it is shared.

Reader: Accesses the Tcl_Obj without modifying the reference count, but may modify the internal representation. There is no need to do anything with the reference count of a Tcl_Obj prior to passing it to a reader.

Setter: Stores a new reference to an existing Tcl_Obj and increments the reference count, which is decremented later when the storage is cleared. An example is newValuePtr of Tcl_ObjSetVar2. To protect a Tcl_Obj passed to a setter, increment its reference count prior to passing it, and decrement it later. Do not worry about reference counts as presumably thre is a corresponding deleter that will decrement the refeence count at the right time.This is fire-and-forget. Passing a Tcl_Obj pointer with a reference count of 0 to a setter means do nothing more with that Tcl_Obj.

As a general rule, all Tcl commands should be considered to be bouncers of the objects in objv. When constructing an objv array to pass to such routines, protect each Tcl_Obj in objv that is needed after the call by incrementing its reference count prior to the call, and decrementing it later when finished with the Tcl_Obj.

We hope to improve the documentation of the categorization of the different functions, and also to significantly reduce the population of bouncers. Constructors and Mutators are already documented as such.

PYK 2017-12-30 2023-04-13: After a pointer to an existing Tcl_Obj has been returned to a caller, that caller should increment the reference count of that object before passing it to a function that might decrement it. In the following example from incr Tcl, the result of Tcl_GetObjResult is passed directly to Tcl_GetObjectFromObj, which, in case of an error, passes the object to Tcl_SetObjResult, which in turn frees it, but then accesses the bytes member of the argument it was passed, which happens to be the same object it just freed:

Tcl_GetObjectFromObj(interp, Tcl_GetObjResult(interp));

This is a form of aliasing , where a routine ends up with two different handles to the same Tcl_Obj. To avoid errors in this case, increment the reference count before passing the Tcl_Obj to another routine:

resPtr = Tcl_GetObjResult(interp);
Tcl_IncrRefCount(resPtr);
Tcl_GetObjectFromObj(interp,resPtr);
Tcl_DecrRefCount(resPtr);

In this example from itcl, a new Tcl_Obj with a reference count of 0 is passed to Tcl_DictObjPut, which may or may not increment the reference count:

objPtr = Tcl_NewStringObj("vars", -1);
Tcl_DictObjRemove(interp, mapDict, objPtr);
Tcl_DictObjPut(interp, mapDict, objPtr, infoPtr->infoVars4Ptr);
Tcl_DecrRefCount(objPtr);

Instead, the reference count should first be incremented:

objPtr = Tcl_NewStringObj("vars", -1);
Tcl_IncrRefCount(objPtr);
Tcl_DictObjRemove(interp, mapDict, objPtr);
Tcl_DictObjPut(interp, mapDict, objPtr, infoPtr->infoVars4Ptr);
Tcl_DecrRefCount(objPtr);

Tcl Does Not Garbage Collect !

I have been trying to understand clearly the rules for Tcl reference counting of objects, and how to properly use the increment and decrement ref count operations. I finally came across a question and answer posting elsewhere, in which Donal K. Fellows clearly explains a very, very, VERY important concept:

Tcl Does Not Garbage Collect!

What does this mean for the Tcl extension writer? If you create an object, and you never pass it back to Tcl as part of another object (eg a list object), or as the result object, it will not get freed. You MUST call Tcl_DecrRefCount() because this is where the memory deallocator gets called... and nowhere else!

It goes without saying that for a very experienced programmer such as myself to have to hunt around for this morsel of information means that the documentation for Tcl Objects in the Tcl C reference is not explicit enough in making this fact crystal clear.

It should also be pointed out that it is perfectly OK to call Tcl_DecrRefCount() to free an allocated object without first having called Tcl_IncrRefCount(). The deallocation will happen if the reference count is zero or negative. This can be the case if you must create a new object for the sole purpose of passing it as an argument to another Tcl API, but then have no further use for it. If that bit of code is called over and over, you will end up with many cats and hats wandering in the woods outside Mr. Tesla's mountain laboratory, homeless.

I hope someone finds this clarification useful.

MS: disagrees strongly with is perfectly OK - at least in the unqualified version above! If you created an object without calling Tcl_IncrRefCount(), and passed it somewhere, calling Tcl_DecrRefCount() on it is possibly disastrous: If some part of Tcl kept a reference to it you will be removing it and freeing the object, which will cause memory corruption further down the line when the reference count is decremented by the rightful owner of the reference! Similarly, if some part of Tcl did an incr/decr of the refCount, the object will already be free when you call Tcl_DecrRefCount() so that your call causes memory corruption. That advise is ONLY correct if you never ever pass the Tcl_Obj anywhere else.

I also disagree (less emphatically) about it being difficult to find that Tcl does not garbage collect. It is never implied that it does; if you have to hunt around for such a morsel of information in the C documentation you will also spend a lot of time. It is even harder to find the morsel of information that Tcl will not wash your dishes. Then again, neither will C, C++, Java or C# - and the docs keep absolute silence about that!

Page Authors

Joe English: Wrote up one of the original messages categorizing library routines by effect on reference counting.

Donal Fellows: Added the category hairy monsters, which later became bouncers.

Poor Yorick: Renamed hairy monster to bouncer.

Category Internals