using lxml and iterparse() to parse a big (+- 1Gb) XML file

问题:

I have to parse a 1Gb XML file with a structure such as below and extract the text within the tags "Author" and "Content":

MM/DD/YYLast Name, NameLorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas dictum dictum vehicula.MM/DD/YYLast Name, NameLorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas dictum dictum vehicula.      [...]      MM/DD/YYLast Name, NameLorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas dictum dictum vehicula.

So far I've tried two things: i) reading the whole file and going through it with .find(xmltag) and ii) parsing the xml file with lxml and iterparse(). The first option I've got it to work, but it is very slow. The second option I haven't managed to get it off the ground.

Here's part of what I have:

for event, element in etree.iterparse(path_to_file, tag="BlogPost"):     if element.tag == "BlogPost":         print element.text     else:         print 'Finished' 

The result of that is only blank spaces, with no text in them.

I must be doing something wrong, but I can't grasp it. Also, In case it wasn't obvious enough, I am quite new to python and it is the first time I'm using lxml. Please, help!

回答1:

for event, element in etree.iterparse(path_to_file, tag="BlogPost"):     for child in element:         print child.tag, child.text     element.clear() 

the final clear will stop you from using too much memory.

[update:] to get "everything between ... as a string" i guess you want one of:

for event, element in etree.iterparse(path_to_file, tag="BlogPost"):     print etree.tostring(element)     element.close() 

or

for event, element in etree.iterparse(path_to_file, tag="BlogPost"):     print ''.join([etree.tostring(child) for child in element])     element.close() 

or perhaps even:

for event, element in etree.iterparse(path_to_file, tag="BlogPost"):     print ''.join([child.text for child in element])     element.close() 

回答2:

For future searchers: The top answer here suggests clearing the element on each iteration, but that still leaves you with an ever-increasing set of empty elements that will slowly build up in memory:

for event, element in etree.iterparse(path_to_file, tag="BlogPost"):     for child in element:         print child.tag, child.text     element.clear() 

^ This is not a scalable solution, especially as your source file gets larger and larger. The better solution is to get the root element, and clear that every time you load a complete record. This will keep memory usage pretty stable (sub-20MB I would say).

Here's a solution that doesn't require looking for a specific tag. This function will return a generator that yields all 1st child nodes (e.g. elements) underneath the root node (e.g. ). It does this by recording the start of the first tag after the root node, then waiting for the corresponding end tag, yielding the entire element, and then clearing the root node.

from lxml import etree  xmlfile = '/path/to/xml/file.xml'  def iterate_xml(xmlfile):     doc = etree.iterparse(xmlfile, events=('start', 'end'))     _, root = next(doc)     start_tag = None     for event, element in doc:         if event == 'start' and start_tag is None:             start_tag = element.tag         if event == 'end' and element.tag == start_tag:             yield element             start_tag = None             root.clear() 

回答3:

I prefer XPath for such things:

In [1]: from lxml.etree import parse  In [2]: tree = parse('/tmp/database.xml')  In [3]: for post in tree.xpath('/Database/BlogPost'):    ...:     print 'Author:', post.xpath('Author')[0].text    ...:     print 'Content:', post.xpath('Content')[0].text    ...:  Author: Last Name, Name Content: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas dictum dictum vehicula. Author: Last Name, Name Content: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas dictum dictum vehicula. Author: Last Name, Name Content: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas dictum dictum vehicula. 

I'm not sure if it's different in terms of processing big files, though. Comments about this would be appreciated.

Doing it your way,

for event, element in etree.iterparse(path_to_file, tag="BlogPost"):      for info in element.iter():          if info.tag in ('Author', 'Content'):              print info.tag, ':', info.text 

文章来源: using lxml and iterparse() to parse a big (+- 1Gb) XML file