What exactly is __weakref__ in Python?

Question

Surprisingly, there's no explicit documentation for __weakref__. Weak references are explained here. __weakref__ is also shortly mentioned in the documentation of __slots__. But I could not find anything about __weakref__ itself.

What exactly is __weakref__? - Is it just a member acting as a flag: If present, the object may be weakly-referenced? - Or is it a function/variable that can be overridden/assigned to get a desired behavior? How?

Answer 1

__weakref__ is just an opaque object that references all the weak references to the current object. In actual fact it's an instance of weakref (or sometimes weakproxy) which is both a weak reference to the object and part of a doubly linked list to all weak references for that object.

It's just an implementation detail that allows the garbage collector to inform weak references that its referent has been collected, and to not allow access to its underlying pointer anymore.

The weak reference can't rely on checking the reference count of the object it refers to. This is because that memory may have been reclaimed and is now being used by another object. Best case scenario the VM will crash, worst case the weak reference will allow access to an object it wasn't originally referring to. This is why the garbage collector must inform the weak reference its referent is no longer valid.

See weakrefobject.h for the structure and C-API for this object. And the implementation detail is here

Answer 2

[Edit 1: Explain the linked list nature and when weakrefs are re-used]

Interestingly enough, the official documentation is somewhat non-enlightening on this topic:

Without a __weakref__ variable for each instance, classes defining __slots__ do not support weak references to its instances. If weak reference support is needed, then add __weakref__ to the sequence of strings in the __slots__ declaration.

The type object documentation on the topic does not seem to help things along too much:

When a type’s __slots__ declaration contains a slot named __weakref__, that slot becomes the weak reference list head for instances of the type, and the slot’s offset is stored in the type’s tp_weaklistoffset.

Weak references form a linked list. The head of that list (the first weak reference to an object) is available via __weakref__. Weakrefs are re-used whenever possible, so the list (not a Python list!) typically is either empty or contains a single element.

Example:

When you first use weakref.ref(), you create a new weak reference chain for the target object. The head of this chain is the new weakref and gets stored in the target object's __weakref__:

>>> import weakref
>>> class A(object): pass
>>> a = A()
>>> b = weakref.ref(a)
>>> c = weakref.ref(b)
>>> print(b is c is a.__weakref__)
True

As we can see, b is re-used. We can force python to create a new weakref, by e.g. adding a callback parameter:

>>> def callback():
>>>   pass
>>> a = A()
>>> b = weakref.ref(a)
>>> c = weakref.ref(b, callback)
>>> print(b is c is a.__weakref__)
False

Now b is a.__weakref__, and c is the second reference in the chain. The reference chain is not directly accessible from Python code. We see only the head element of the chain (b), but not how the chain continues (b -> c).

So __weakref__ is the head of the internal linked list of all the weak references to the object. I cannot find any piece of official documentation where this role of __weakref__ is concisely explained, so one should probably not rely on this behavior, as it is an implementation detail.

Answer 3

The __weakref__ variable is an attribute which makes the object to support the weak references and preserving the weak references to object.

The python documentation has explained it as following:

when the only remaining references to a referent are weak references, garbage collection is free to destroy the referent and reuse its memory for something else.

Therefore, the duty of weak references is supplying the conditions for an object in order to be able to be garbage collected regardless of its type and the scope.

And about the __slots__, we can first look into the documentation, which explains it very well:

By default, instances of classes have a dictionary for attribute storage. This wastes space for objects having very few instance variables. The space consumption can become acute when creating large numbers of instances.

The default can be overridden by defining __slots__ in a class definition. The __slots__ declaration takes a sequence of instance variables and reserves just enough space in each instance to hold a value for each variable. Space is saved because __dict__ is not created for each instance.

Now, since by using __slots__ you will control the demanded storage for your attribute, it actually prevents the automatic creation of __dict__ and __weakref__ for each instance. Which the __weakref__ is the necessary variable of each object in order to be able to deal with weak references.

Also, in addition to all these the documentation for object.__slots__ class says:

This class variable can be assigned a string, iterable, or sequence of strings with variable names used by instances. __slots__ reserves space for the declared variables and prevents the automatic creation of __dict__ and __weakref__ for each instance.

So, In a nutshell, we can conclude that __slots__ are for managing the storage allocation manually and since __weakref__ is the license of accepting the weak references for objects which is related to storage (because of the ability of being garbage collected), therefore __slots__ will control the __weakref__ as well as controlling the __dict__ attribute.

Also documentation has shown you the way of making an object to support the weak references along side of using __slots__:

Without a __weakref__ variable for each instance, classes defining __slots__ do not support weak references to its instances. If weak reference support is needed, then add '__weakref__' to the sequence of strings in the __slots__ declaration.

Here is an example in python 3.X:

>>> class Test:
...     __slots__ = ['a', 'b']
... 
>>> 
>>> import weakref
>>> 
>>> t = Test()
>>> 
>>> r = weakref.ref(t)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot create weak reference to 'Test' object
>>> 
>>> class Test:
...     __slots__ = ['a', 'b', '__weakref__']
... 
>>> t = Test()
>>> r = weakref.ref(t)
>>> 
>>> t.__weakref__
<weakref at 0x7f735bc55d68; to 'Test' at 0x7f735bc51fc8>

But in python 2.7 there, although the documentation is like the aforementioned docs, creating a weak reference from instances that doesn't provide the __weakref__ variable in their __slots__ names doesn't raise a TypeError:

>>> class Test:
...    __slots__ = ['a', 'b']
... 
>>> t = Test()
>>> 
>>> r = weakref.ref(t)
>>> 
>>> r
<weakref at 0x7fe49f4185d0; to 'instance' at 0x7fe4a3e75f80>