What is the best way to use Xpath for processing larger XML files?

问题

I have a requirement where I have to use an large XML (4 GB file) for finding values in it. Basically I have to write around 30 different Xpath and store the values in a list. When I try to parse an XML, it throws memory error. I have tried using lxml and ElementTree with start and end events, still no luck the processing time is too high and my Pycharm/Jupyter notebook throws me memory error.

Is there a better way to do it? Even though this implementation is not restricted to any programming language, I prefer Python because its my right hand. Thanks in advance.

Eg of a search: If I want the value of year where category is cooking. Then I use ./bookstore[@category=cooking]/book/year. So the value is 2005

Similarly I have to find the values of my tags based on my business requirements. In my requirement the XML is not simple as the below example.

<?xml version="1.0" encoding="UTF-8"?>

<bookstore>

<book category="cooking">
  <title lang="en">Everyday Italian</title>
  <author>Giada De Laurentiis</author>
  <year>2005</year>
  <price>30.00</price>
</book>

<book category="children">
  <title lang="en">Harry Potter</title>
  <author>J K. Rowling</author>
  <year>2005</year>
  <price>29.99</price>
</book>

<book category="web">
  <title lang="en">XQuery Kick Start</title>
  <author>James McGovern</author>
  <author>Per Bothner</author>
  <author>Kurt Cagle</author>
  <author>James Linn</author>
  <author>Vaidyanathan Nagarajan</author>
  <year>2003</year>
  <price>49.99</price>
</book>

</bookstore>

回答1:

With the example given, the problem can be readily handled using streaming in XSLT 3.0:

<xsl:mode streamable="yes">
<xsl:template match="book[@category='cooking']">
  <xsl:value-of select="year"/>
</xsl:template>

I know the example is a simplification, but we can only judge whether the real data/query is streamable by seeing the real data/query. The devil is in the detail.

回答2:

I don't know if this will meet your needs. Try it first. If there is a problem, we can continue to communicate.

from simplified_scrapy import SimplifiedDoc,req,utils
def getBook(lines,category,year):
  html = "".join(lines)
  book = SimplifiedDoc(html).book
  if book.category==category and book.year.text==year:
    return book.category,book.title.text,book.authors.text,book.year.text,book.price.text
lst = []
with open('bookstore.xml', 'r') as file: # bookstore.xml is your xml file path
  lines = []
  flag = False
  for line in file:
    if flag or line.find('<book ')>=0:
      flag = True
      lines.append(line)
    if line.find('</book>')>=0:
      b = getBook(lines,'web','2003')
      if b: 
        lst.append(b)
        break # If you only want the first one, add a break
      flag = False
      lines = []
print (lst)

Result:

[('web', 'XQuery Kick Start', ['James McGovern', 'Per Bothner', 'Kurt Cagle', 'James Linn', 'Vaidyanathan Nagarajan'], '2003', '49.99')]

来源：https://stackoverflow.com/questions/61228553/what-is-the-best-way-to-use-xpath-for-processing-larger-xml-files

标签

python

xml

xml-parsing