Evaluating HDF5: What limitations/features does HDF5 provide for modelling data?

Question

We are in evaluating technologies that we\'ll use to store data that we gather during the analysis of C/C++ code. In the case of C++, the amount of data can be relatively l

Answer 1

We produce HDF5 data on my project, but I don't directly deal with it usually. I can take a stab at the first two questions:

1. We use a write once, read many times model and the format seems to handle this well. I know a project that used to write both to an Oracle database and HDF5. Eventually they removed the Oracle output since performance suffered and no one was using it. Obviously, SQLite is not Oracle, but the HDF5 format was better suited for the task. Based on that one data point, a RDBMS may be better tuned for multiple inserts and updates.

2. The readers our customers use are robust when we add new data types. Some of the changes are anticipated, but we don't have to worry about breaking thing when adding more data fields. Our DBA recently wrote a Python program to read HDF5 data and populate KMZ files for visualization in Google Earth. Since it was a project he used to learn Python, I'd say it's not hard to build readers.

On the third question, I'll bow to Jason S's superior knowledge.

I'd say HDF5 is a completely reasonable choice, especially if you are already interested in it or plan to produce something for the scientific community.