Regular Expressions to parse template tags in XML

Question

I need to parse some XML to pull out embedded template tags for further parsing. I can't seem to bend Python's regular expressions to do what I want, though.

In English: when a template tag is contained anywhere in the row, remove all the XML for that specific row and leave only the template tag in its place.

I put together a test case to demonstrate. Here's the original XML:

<!-- regex_trial.xml -->
<w:tbl>
    <w:tr>
        <w:tc><w:t>Header 1</w:t></w:tc>
        <w:tc><w:t>Header 2</w:t></w:tc>
        <w:tc><w:t>Header 3</w:t></w:tc>
    </w:tr>
     <w:tr>
        <w:tc><w:t>{% for i in items %}</w:t></w:tc>
        <w:tc><w:t></w:t></w:tc>
        <w:tc><w:t></w:t></w:tc>
    </w:tr>
    <w:tr>
        <w:tc><w:t>{{ i.field1 }}</w:t></w:tc>
        <w:tc><w:t>{{ i.field2 }}</w:t></w:tc>
        <w:tc><w:t>{{ i.field3 }}</w:t></w:tc>
    </w:tr>
    <w:tr>
        <w:tc><w:t>{% endfor %}</w:t></w:tc>
        <w:tc><w:t></w:t></w:tc>
        <w:tc><w:t></w:t></w:tc>
    </w:tr>
</w:tbl>

This is the desired result:

<!-- regex_desired_result.xml -->
<w:tbl>
    <w:tr>
        <w:tc><w:t>Header 1</w:t></w:tc>
        <w:tc><w:t>Header 2</w:t></w:tc>
        <w:tc><w:t>Header 3</w:t></w:tc>
    </w:tr>
    {% for i in items %}
    <w:tr>
        <w:tc><w:t>{{ i.field1 }}</w:t></w:tc>
        <w:tc><w:t>{{ i.field2 }}</w:t></w:tc>
        <w:tc><w:t>{{ i.field3 }}</w:t></w:tc>
    </w:tr>
    {% endfor %}
</w:tbl>

Here is some python code I am using to test:

#!/usr/bin/env python
import re
f = open( 'regex_trial.xml', 'r' )
orig_xml = f.read()
f.close()
p = re.compile( '<w:tr.*?(?P<tag>{%.*?%}).*?</w:tr>', re.DOTALL )
new_xml = p.sub( '\g<tag>', orig_xml, 0 )
print new_xml

The actual result of this regex is:

<!-- regex_trial.xml -->
<w:tbl>
    {% for i in items %}
    {% endfor %}
</w:tbl>

Any help is greatly appreciated! If we can figure this out, we will be able to dynamically generate MS Word docx files on the fly from Django-powered sites. Thanks!!

Update: this is the final code that I used

from xml.etree import ElementTree
import cStringIO as StringIO

TEMPLATE_TAG = 'template_text'

tree = ElementTree.parse( 'regex_trial.xml' )
rows = tree.getiterator('tr')
for row in rows:
    for cell in row.getiterator('t'):
        if cell.text and cell.text.find( '{%' ) >= 0:
            template_tag = cell.text
            row.clear()
            row.tag = TEMPLATE_TAG
            row.text = template_tag
            break

output = StringIO.StringIO()
tree.write( output )
xml = output.getvalue()
xml = xml.replace('<%s>' % TEMPLATE_TAG, '')
xml = xml.replace('</%s>' % TEMPLATE_TAG, '')
print xml

Thanks for all the help!

Answer 1

Please don't use regular expressions for this problem.

I'm serious, parsing XML with a regex is hard, and it makes your code 50x less maintainable by anyone else.

lxml is the defacto tool that pythonistas use to parse XML... take a look at this article on Stack Overflow for sample usage. Or consider this answer, which should have been the answer that was accepted.

I hacked this up as a quick demo... it searches for <w:tc> with non-empty <w:t> children and prints good next to each element.

import lxml.etree as ET
from lxml.etree import XMLParser

def worthy(elem):
    for child in elem.iterchildren():
        if (child.tag == 't') and (child.text is not None):
            return True
    return False

def dump(elem):
    for child in elem.iterchildren():
        print "Good", child.tag, child.text

parser = XMLParser(ns_clean=True, recover=True)
etree = ET.parse('regex_trial.xml', parser)
for thing in etree.findall("//"):
    if thing.tag == 'tc' and worthy(thing):
        dump(thing)

Yields...

Good t Header 1
Good t Header 2
Good t Header 3
Good t {% for i in items %}
Good t {{ i.field1 }}
Good t {{ i.field2 }}
Good t {{ i.field3 }}
Good t {% endfor %}

Answer 2

Never ever parse HTML or XML or SGML with regular expressions.

Always use tools like lxml, libxml2 or Beautiful - they will ever do a smarter and better job than your code .