Python sax to lxml for 80+GB XML

Question

How would you read an XML file using sax and convert it to a lxml etree.iterparse element?

To provide an overview of the problem, I have built an XML ingestion tool using lxml for an XML feed that will range in the size of 25 - 500MB that needs ingestion on a bi-daily basis, but needs to perform a one time ingestion of a file that is 60 - 100GB's.

I had chosen to use lxml based on the specifications that detailed a node would not exceed 4 -8 GB's in size which I thought would allow the node to be read into memory and cleared when finished.

An overview if the code is below

elements = etree.iterparse(
    self._source, events = ('end',)
)
for event, element in elements:
    finished = True
    if element.tag == 'Artist-Types':
        self.artist_types(element)

def artist_types(self, element):
    """
    Imports artist types

    :param list element: etree.Element
    :returns boolean:
    """
    self._log.info("Importing Artist types")
    count = 0
    for child in element:
        failed = False
        fields = self._getElementFields(child, (
            ('id', 'Id'),
            ('type_code', 'Type-Code'),
            ('created_date', 'Created-Date')
        ))
        if self._type is IMPORT_INC and has_artist_type(fields['id']):
            if update_artist_type(fields['id'], fields['type_code']):
                count = count + 1
            else:
                failed = True
        else:
            if create_artist_type(fields['type_code'],
                fields['created_date'], fields['id']):
                count = count + 1
            else:
                failed = True
        if failed:
            self._log.error("Failed to import artist type %s %s" %
                (fields['id'], fields['type_code'])
            )
    self._log.info("Imported %d Artist Types Records" % count)
    self._artist_type_count = count
    self._cleanup(element)
    del element

Let me know if I can add any type of clarification.

Answer 1

iterparse is an iterative parser. It will emit Element objects and events and incrementally build the entire Element tree as it parses, so eventually it will have the whole tree in memory.

However, it is easy to have a bounded memory behavior: delete elements you don't need anymore as you parse them.

The typical "giant xml" workload is a single root element with a large number of child elements which represent records. I assume this is the kind of XML structure you are working with?

Usually it is enough to use clear() to empty out the element you are processing. Your memory usage will grow a little but it's not very much. If you have a really huge file, then even the empty Element objects will consume too much and in this case you must also delete previously-seen Element objects. Note that you cannot safely delete the current element. The lxml.etree.iterparse documentation describes this technique.

In this case, you will process a record every time a </record> is found, then you will delete all previous record elements.

Below is an example using an infinitely-long XML document. It will print the process's memory usage as it parses. Note that the memory usage is stable and does not continue growing.

from lxml import etree
import resource

class InfiniteXML (object):
    def __init__(self):
        self._root = True
    def read(self, len=None):
        if self._root:
            self._root=False
            return "<?xml version='1.0' encoding='US-ASCII'?><records>\n"
        else:
            return """<record>\n\t<ancestor attribute="value">text value</ancestor>\n</record>\n"""

def parse(fp):
    context = etree.iterparse(fp, events=('end',))
    for action, elem in context:
        if elem.tag=='record':
            # processing goes here
            pass

        #memory usage
        print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

        # cleanup
        # first empty children from current element
            # This is not absolutely necessary if you are also deleting siblings,
            # but it will allow you to free memory earlier.
        elem.clear()
        # second, delete previous siblings (records)
        while elem.getprevious() is not None:
            del elem.getparent()[0]
        # make sure you have no references to Element objects outside the loop

parse(InfiniteXML())

Answer 2

I found this helpful example at http://effbot.org/zone/element-iterparse.htm. Bold emphasis is mine.

Incremental Parsing #

Note that iterparse still builds a tree, just like parse, but you can safely rearrange or remove parts of the tree while parsing. For example, to parse large files, you can get rid of elements as soon as you’ve processed them:

for event, elem in iterparse(source):
    if elem.tag == "record":
        ... process record elements ...
        elem.clear()

The above pattern has one drawback; it does not clear the root element, so you will end up with a single element with lots of empty child elements. If your files are huge, rather than just large, this might be a problem. To work around this, you need to get your hands on the root element. The easiest way to do this is to enable start events, and save a reference to the first element in a variable:

# get an iterable 
context = iterparse(source, events=("start", "end"))

# turn it into an iterator 
context = iter(context)

# get the root element 
event, root = context.next()

for event, elem in context:
    if event == "end" and elem.tag == "record":
        ... process record elements ...
        root.clear()

(future releases will make it easier to access the root element from within the loop)

Answer 3

This is a couple of years old and I don't have enough reputation to comment directly on the accepted answer, but I tried using this to parse an OSM where I am finding all intersections in a country. My original issue was that I was running out of RAM, so I thought I'd have to use the SAX parser but found this answer instead. Strangely it wasn't parsing correctly, and using the suggested cleanup somehow was clearing the elem node before reading through it (still not sure how this was happening). Removed elem.clear() from the code and now it runs perfectly fine!