Using Python's xml.etree to find element start and end character offsets

问题

I have XML data that looks like:

<xml>
The captial of <place pid="1">South Africa</place> is <place>Pretoria</place>.
</xml>

I would like to be able to extract:

The XML elements as they're currently provided in etree.
The full plain text of the document, between the start and end tags.
The location within the plain text of each start element, as a character offset.

(3) is the most important requirement right now; etree provides (1) fine.

I cannot see any way to do (3) directly, but hoped that iterating through the elements in the document tree would return many small string that could be re-assembled, thus providing (2) and (3). However, requesting the .text of the root node only returns text between the root node and the first element, e.g. "The capital of ".

Doing (1) with SAX could involve implementing a lot that's already been written many times over, in e.g. minidom and etree. Using lxml isn't an option for the package that this code is to go into. Can anybody help?

回答1:

iterparse() function is available in xml.etree:

import xml.etree.cElementTree as etree

for event, elem in etree.iterparse(file, events=('start', 'end')):
    if event == 'start':
       print(elem.tag) # use only tag name and attributes here
    elif event == 'end':
       # elem children elements, elem.text, elem.tail are available
       if elem.text is not None and elem.tail is not None:
          print(repr(elem.tail))

Another option is to override start(), data(), end() methods of etree.TreeBuilder():

from xml.etree.ElementTree import XMLParser, TreeBuilder

class MyTreeBuilder(TreeBuilder):

    def start(self, tag, attrs):
        print("&lt;%s>" % tag)
        return TreeBuilder.start(self, tag, attrs)

    def data(self, data):
        print(repr(data))
        TreeBuilder.data(self, data)

    def end(self, tag):
        return TreeBuilder.end(self, tag)

text = """<xml>
The captial of <place pid="1">South Africa</place> is <place>Pretoria</place>.
</xml>"""

# ElementTree.fromstring()
parser = XMLParser(target=MyTreeBuilder())
parser.feed(text)
root = parser.close() # return an ordinary Element

Output

<xml>
'\nThe captial of '
<place>
'South Africa'
' is '
<place>
'Pretoria'
'.\n'

回答2:

You need to look at the .tail property as well as .text: .textgives you the text directly after a start tag, .tail gives you the text directly after the end tag. This will provide you with your "many small strings".

Tip: you can use etree.iterwalk(elem) (does the same thing as with etree.iterparse() but over an existing tree instead) to iterate over the start and end tags. To the idea:

for event, elem in etree.iterwalk(xml_elem, events=('start', 'end')):
    if event == 'start':
        # it's a start tag
        print 'starting element', elem.tag
        print elem.text
    elif event == 'end':
        # it's an end tag
        print 'ending element', elem.tag
        if elem is not xml_elem:
            # dont' want the text trailing xml_elem
            print elem.tail

I guess you can complete the rest for yourself? Warning: .text and .tail can be None, so if you want to concatenate you will have to guard against that (use (elem.text or '')for example)

If you are familiar with sax (or have existing sax code that does what you need), lxml lets you produce sax events from an element or tree:

lxml.sax.saxify(elem, handler)

Some other things to look for when extracting all the text from an element: the .itertext() method, the xpath expression .//text() (lxml lets you return "smart strings" from xpath expressions: they allow you to check which element they belong to etc...).

回答3:

(2) is easy with SAX, see this snippet

from xml.sax.handler import ContentHandler
import xml.sax
import sys

class textHandler(ContentHandler):
    def characters(self, ch):
        sys.stdout.write(ch.encode("Latin-1"))

parser = xml.sax.make_parser()
handler = textHandler()
parser.setContentHandler(handler)
parser.parse("test.xml")

or the Example 1-1: bookhandler.py in this book http://oreilly.com/catalog/pythonxml/chapter/ch01.html

(3) is trickier, consult to this thread, it's Java, but there should be similar thing in Python SAX api How do I get the correct starting/ending locations of a xml tag with SAX?

回答4:

(3) can be done with XMLParser.CurrentByteIndex, like this:

import xml.etree.ElementTree as ET

class MyTreeBuilder(ET.TreeBuilder):
    def start(self, tag, attrs):
        print(parser.parser.CurrentByteIndex)
        ET.TreeBuilder.start(self, tag, attrs)

builder = MyTreeBuilder()
parser = ET.XMLParser(target=builder)
builder.parser = parser
tree = ET.parse('test.xml', parser=parser)

See also this answer for a SAX alternative. Take note however that the byte index is not the same as character index, and there may not be an efficient way to translate byte to character index in Python. (See also here.)

An (admittedly ugly) workaround to get character offsets instead of byte offsets is to recode bytes as characters. Assuming the actual encoding is utf8:

import xml.etree.ElementTree as ET

class MyTreeBuilder(ET.TreeBuilder):
    def start(self, tag, attrs):
        print(parser.parser.CurrentByteIndex)
        ET.TreeBuilder.start(self, tag, attrs)

builder = MyTreeBuilder()
parser = ET.XMLParser(target=builder)
builder.parser = parser
with open('test.xml', 'rb') as f:
    parser.feed(f.read().decode('latin1').encode('utf8'))

来源：https://stackoverflow.com/questions/8111556/using-pythons-xml-etree-to-find-element-start-and-end-character-offsets

标签

python

xml

sax

elementtree