Parse several XML declarations in a single file by means of lxml.etree.iterparse

Question

I need to parse a file that contains various XML files, i.e., <xml></xml> <xml></xml> .. and so forth. While using etree.iterparse, I get the following (correct) error:

lxml.etree.XMLSyntaxError: XML declaration allowed only at the start of the document

Now, I can preprocess the input file and produce for each contained XML file a separate file. This might be the easiest solution. But I wonder if a proper solution for this 'problem' exists.

Thanks!

Answer 1

The sample data you've provided suggests one problem, while the question and the exception you've provided suggests another. Do you have multiple XML documents concatenated together, each with its own XML declaration, or do you have an XML fragment with multiple top-level elements?

If it's the former, then the solution's going to involve breaking the input stream up into multiple streams, and parsing each one individually. This doesn't necessarily mean, as one comment suggests, implementing an XML parser. You can search a string for XML declarations without having to parse anything else in it, so long as your input doesn't include CDATA sections that contain unescaped XML declarations. You can write a file-like object that returns characters from the underlying stream until it hits an XML declaration, and then wrap it in a generator function that keeps returning streams until EOF is reached. It's not trivial, but it's not hugely difficult either.

If you have an XML fragment with multiple top-level elements, you can just wrap them an XML element and parse the whole thing.

Of course, as with most problems involving bad XML input, the easiest solution may just be to fix the thing that's producing the bad input.

Answer 2

I used regex to solve this problem. Suppose that data is a string that contains your multiple xml documents, and that handle is a function that will do something with each document. After executing this loop, data will be empty, or will contain an incomplete XML document, and the handle function will have been called zero or more times.

while True:
  match = re.match (r'''
        \s*                 # ignore leading whitespace
        (                   # start first group
          <(?P<TAG>\S+).*?> # opening tag (with optional attributes)
            .*?             # stuff in the middle
          </(?P=TAG)>       # closing tag
        )                   # end of first xml document
        (?P<REM>.*)         # anything else
      ''',
    data, re.DOTALL | re.VERBOSE)
  if not match:
    break
  document = match.group (1)
  handle (document)
  data = match.group ('REM')