Keeping large dictionary in Python affects application performance

Question

I\'m having some difficulties understanding (and ultimately solving) why having a large dictionary in memory makes creation of other dictionaries longer.

Here\'s the tes

Answer 1

"Dictionary creation" is really a red herring here. What the dictionary creation does in this case that's relevant is that it creates a hundred thousand new 125-element lists. Because lists can be involved in reference cycles, that creates 12.5 million list elements CPython's cyclic garbage collection has to examine whenever it scans a generation containing a dict. It doesn't matter that these lists are in dictionaries, it only matters that they exist.

So what you're timing is largely the time consumed by Python's cyclic garbage collection. It doesn't particularly matter that you keep on creating more dicts, it only matters that you're creating new mutable objects (which can be involved in cycles) much faster than you're destroying old mutable objects. That (an excess of allocations over deallocations) is what triggers CPython's cyclic gc).

Not much you can do about it, alas. Programs that go through well-delineated phases of creating mounds of new objects can benefit by disabling cyclic gc for the duration. Can't guess whether that applies to you, though.

Ah, forgot one: the dict in Foo makes such a big difference because Foo "sticks around". All the other dicts you create are thrown away immediately after being constructed (CPython's reference counting is responsible for that), so don't add to the time consumed by cyclic gc. But the dict in Foo does, because that dict doesn't go away. Change your loop to append the new dicts to a list, and you'll see that the time goes up for each dict, then drops a lot, then goes up again, etc. That reflects dicts moving to older generations inside Python's cyclic gc, so getting scanned by cyclic gc less frequently. It gets complicated ;-)

More details?

It's hard to be more precise, because the performance of cyclic gc in specific cases depends on mountains of implementation details that can - and do - change across releases.

The general advice to use "immutable objects" when possible is based on a rather deep ;-) understanding of how cyclic gc is implemented in CPython, and how it's evolved over the years.

It so happens that today (most recent versions of Python 2 and Python 3), strong efforts are made to exempt certain tuples and dicts from cyclic gc. That may change. Special-casing such things isn't free, so deciding whether to add more tricks like this is always a difficult balancing act. It's an easier decision for immutable objects, hence the advice to move towards those.

Tuples and dicts weren't special-cased at all until very late 2008, as detailed in this from the Python issue tracker.

And - surprise ;-) - that turned out to have horrible performance consequences in some rare cases, which were fixed by more special-casing in this issue in mid 2012.

Some good news is that a gc.is_tracked() function was added so you can at least investigate whether cyclic gc is going to scan a specific object. Here are some examples under Python 2.7.5. There's no guarantee they'll always work this way:

"Scalar" objects (no internal pointers) are never tracked - it's impossible for them to be in a cycle:

>>> import gc
>>> gc.is_tracked(4)
False
>>> gc.is_tracked("2323")
False

Tuples are initially tracked:

>>> t1 = ([1],)
>>> t2 = ((1.),)
>>> gc.is_tracked(t1), gc.is_tracked(t2)
(True, True)

However, if cyclic gc runs and determines that a tuple is immutable "all the way down", it untracks that tuple:

>>> gc.collect()
0
>>> gc.is_tracked(t1), gc.is_tracked(t2)
(True, False)

There's nothing that can be done to t2 to make it participate in a cycle, because it, and all its components (on & on, all the way down) are immutable. But t1 still needs to be tracked! Because t1[0] is mutable, t1 may be part of a cycle later:

>>> t1
([1],)
>>> t1[0][0] = t1
>>> t1
([([...],)],)

A different policy happens to be used for dicts. They're created untracked, if possible:

>>> d = {1: [2]}
>>> gc.is_tracked(d)
True

Because that dict has a mutable value, it could become part of a cycle later, so must be tracked by cyclic gc.

>>> d[1][0] = d
>>> d
{1: [{...}]}

But a dict with untracked keys and values is created untracked:

>>> d = {1: 2}
>>> gc.is_tracked(d)
False

This is tricky, because such a dict can still become part of a cycle later!

>>> d[2] = d
>>> gc.is_tracked(d)
True

It's not free to detect such changes.

Then there are combinations of the above:

>>> d = {(1, 2): (4, "abc", 5)}
>>> gc.is_tracked(d)
True
>>> gc.collect()
3
>>> gc.is_tracked(d)
False

In this case, d is tracked at first, because its keys and values (the tuples) are created tracked at first. But after cyclic gc runs the first time, it determines that the keys and values are "immutable all the way down", so untracks the dict.

Like I said, it gets complicated ;-)

BTW,

I understand that tuple creation is much faster than list creation

It should be only slightly slower to create a list. Lists and tuples have very similar implementations in CPython. tuples require a little less memory (which can be significant, in percentage terms, for very short sequences), and it can be a little faster to index a tuple than a list. But creation time? It's the difference between one malloc()-like function (for a tuple) versus two (for a list), independent of the number of elements. That can be significant for very short sequences, but not for large ones.