Parsing very large XML documents (and a bit more) in java

Question

(All of the following is to be written in Java)

I have to build an application that will take as input XML documents that are, potentially, very large. The document

Answer 1

Stax is the right way. I would recommend looking at Woodstox

Answer 2

You might be interested by XOM:

XOM is fairly unique in that it is a dual streaming/tree-based API. Individual nodes in the tree can be processed while the document is still being built. The enables XOM programs to operate almost as fast as the underlying parser can supply data. You don't need to wait for the document to be completely parsed before you can start working with it.

XOM is very memory efficient. If you read an entire document into memory, XOM uses as little memory as possible. More importantly, XOM allows you to filter documents as they're built so you don't have to build the parts of the tree you aren't interested in. For instance, you can skip building text nodes that only represent boundary white space, if such white space is not significant in your application. You can even process a document piece by piece and throw away each piece when you're done with it. XOM has been used to process documents that are gigabytes in size.

Answer 3

You could use a BufferedInputStream with a very large buffer size and use mark() before the extension class works and reset() afterwards.

If the parts the extension class needs is very far into the file, then this might become extremely memory intensive, 'though.

A more general solution would be to write your own BufferedInputStream-workalike that buffers to the disk if the data that is to be buffered exceeds some preset threshold.

Answer 4

I would write a custom implementation of InputStream that decrypts the bytes in the file and then use SAX to parse the resulting XML as it comes off the stream.

SAXParserFactory.newInstance().newSAXParser().parse(
  new DecryptingInputStream(), 
  new MyHandler()
);

Answer 5

Look at the XOM library. The example you are looking for is StreamingExampleExtractor.java in the samples directory of the source distribution. This shows a technique for performing a streaming parse of a large xml document only building specific nodes, processing them and discarding them. It is very similar to a sax approach, but has a lot more parsing capability built in so a streaming parse can be achieved pretty easily.

If you want to work at higher level look at NUX. This provides a high level streaming xpath API that only reads the amount of data into memory needed to evaluate the xpath.

Answer 6

This sounds like a job for StAX (JSR 173). StAX is a pull parser, which means that it works more or less like an event based parser like SAX, but that you have more control over when to stop reading, which elements to pull, ...

The usability of this solution will depend a lot on what your extension classes are actually doing, if you have control over their implementation, etc...

The main point is that if the document is very large, you probably want to use an event based parser and not a tree based, so you will not use a lot of memory.

Implementations of StAX can be found from SUN (SJSXP), Codehaus or a few other providers.