overloaded __iter__ is bypassed when deriving from dict

Question

Trying to create a custom case-insensitive dictionary, I came the following inconvenient and (from my point-of-view) unexpected behaviour. If deriving a class from dict, the overloaded __iter__, keys, values functions are ignored when converting back to dict. I have condensed it to the following test case:

import collections

class Dict(dict):
    def __init__(self):
        super(Dict, self).__init__(x = 1)

    def __getitem__(self, key):
        return 2

    def values(self):
        return 3

    def __iter__(self):
        yield 'y'

    def keys(self):
        return 'z'

    if hasattr(collections.MutableMapping, 'items'):
        items = collections.MutableMapping.items
    if hasattr(collections.MutableMapping, 'iteritems'):
        iteritems = collections.MutableMapping.iteritems

d = Dict()
print(dict(d))              # {'x': 1}
print(dict(d.items()))      # {'y': 2}

The values for keys,values and __iter__,__getitem__ are inconsistent only for demonstration which methods are actually called.

The documentation for dict.__init__ says:

If a positional argument is given and it is a mapping object, a dictionary is created with the same key-value pairs as the mapping object. Otherwise, the positional argument must be an iterator object.

I guess it has something to do with the first sentence and maybe with optimizations for builtin dictionaries.

Why exactly does the call to dict(d) not use any of keys, __iter__? Is it possible to overload the 'mapping' somehow to force the dict constructor to use my presentation of key-value pairs?

Why did I use this? For a case-insensitive but -preserving dictionary, I wanted to:

• store (lowercase => (original_case, value)) internally, while appearing as (any_case => value).
• derive from dict in order to work with some external library code that uses isinstance checks
• not use 2 dictionary lookups: lower_case=>original_case, followed by original_case=>value (this is the solution which I am doing now instead)

If you are interested in the application case: here is corresponding branch

Answer 1

In the file dictobject.c, you see in line 1795ff. the relevant code:

static int
dict_update_common(PyObject *self, PyObject *args, PyObject *kwds, char *methname)
{
    PyObject *arg = NULL;
    int result = 0;

    if (!PyArg_UnpackTuple(args, methname, 0, 1, &arg))
        result = -1;

    else if (arg != NULL) {
        _Py_IDENTIFIER(keys);
        if (_PyObject_HasAttrId(arg, &PyId_keys))
            result = PyDict_Merge(self, arg, 1);
        else
            result = PyDict_MergeFromSeq2(self, arg, 1);
    }
    if (result == 0 && kwds != NULL) {
        if (PyArg_ValidateKeywordArguments(kwds))
            result = PyDict_Merge(self, kwds, 1);
        else
            result = -1;
    }
    return result;
}

This tells us that if the object has an attribute keys, the code which is called is a mere merge. The code called there (l. 1915 ff.) makes a distinction between real dicts and other objects. In the case of real dicts, the items are read out with PyDict_GetItem(), which is the "most inner interface" to the object and doesn't bother using any user-defined methods.

So instead of inheriting from dict, you should use the UserDict module.

Answer 2

Is it possible to overload the 'mapping' somehow to force the dict constructor to use my presentation of key-value pairs?

No.

Being an inherent type, redefining the semantics of dict would certainly cause outright breakage elsewhere.

You've got a library that you can't override the behavior of dict in, that's tough, but redefining the language primitives isn't the answer. You'd probably find it irksome if someone screwed with the commutative property of integer addition behind your back; that's why they can't.

And with regard to your comment "UserDict (correctly) gives False in isinstance(d, dict) checks", of course it does because it isn't a dict and dict has very specific invariants which UserDict can't assure.