Python: How to remove all empty fields in a nested dict

Question

If I have a dict, which field\'s values may also be a dict or an array. How can I remove all empty fields in it?

\"Empty field\" means a field\'s value is empty arra

Answer 1

If you want a full-featured, yet succinct approach to handling real-world data structures which are often nested, and can even contain cycles and other kinds of containers, I recommend looking at the remap utility from the boltons utility package.

After pip install boltons or copying iterutils.py into your project, just do:

from boltons.iterutils import remap

data = {'veg': [], 'fruit': [{'apple': 1}, {'banana': None}], 'result': {'apple': 1, 'banana': None}}

drop_falsey = lambda path, key, value: bool(value)
clean = remap(data, visit=drop_falsey)
print(clean)

# Output:
{'fruit': [{'apple': 1}], 'result': {'apple': 1}}

This page has many more examples, including ones working with much larger objects from Github's API.

It's pure-Python, so it works everywhere, and is fully tested in Python 2.7 and 3.3+. Best of all, I wrote it for exactly cases like this, so if you find a case it doesn't handle, you can bug me to fix it right here.

Answer 2

@mojoken - How about this to overcome the boolean problem

def clean_empty(d):
if not isinstance(d, (dict, list)):
    return d
if isinstance(d, list):
    return [v for v in (clean_empty(v) for v in d) if isinstance(v, bool) or v]
return {k: v for k, v in ((k, clean_empty(v)) for k, v in d.items()) if isinstance(v, bool) or v}

Answer 3

Use a recursive function that returns a new dictionary:

def clean_empty(d):
    if not isinstance(d, (dict, list)):
        return d
    if isinstance(d, list):
        return [v for v in (clean_empty(v) for v in d) if v]
    return {k: v for k, v in ((k, clean_empty(v)) for k, v in d.items()) if v}

The {..} construct is a dictionary comprehension; it'll only include keys from the original dictionary if v is true, e.g. not empty. Similarly the [..] construct builds a list.

The nested (.. for ..) constructs are generator expressions that allow the code to compactly filter empty objects after recursing.

Note that any values set to numeric 0 (integer 0, float 0.0) will also be cleared. You can retain numeric 0 values with if v or v == 0.

Demo:

>>> sample = {
...     "fruit": [
...         {"apple": 1},
...         {"banana": None}
...     ],
...     "veg": [],
...     "result": {
...         "apple": 1,
...         "banana": None
...     }
... }
>>> def clean_empty(d):
...     if not isinstance(d, (dict, list)):
...         return d
...     if isinstance(d, list):
...         return [v for v in (clean_empty(v) for v in d) if v]
...     return {k: v for k, v in ((k, clean_empty(v)) for k, v in d.items()) if v}
... 
>>> clean_empty(sample)
{'fruit': [{'apple': 1}], 'result': {'apple': 1}}

Answer 4

def remove_empty_fields(data_):
    """
        Recursively remove all empty fields from a nested
        dict structure. Note, a non-empty field could turn
        into an empty one after its children deleted.

        :param data_: A dict or list.
        :return: Data after cleaning.
    """
    if isinstance(data_, dict):
        for key, value in data_.items():

            # Dive into a deeper level.
            if isinstance(value, dict) or isinstance(value, list):
                value = remove_empty_fields(value)

            # Delete the field if it's empty.
            if value in ["", None, [], {}]:
                del data_[key]

    elif isinstance(data_, list):
        for index in reversed(range(len(data_))):
            value = data_[index]

            # Dive into a deeper level.
            if isinstance(value, dict) or isinstance(value, list):
                value = remove_empty_fields(value)

            # Delete the field if it's empty.
            if value in ["", None, [], {}]:
                data_.pop(index)

    return data_