What to put in a binary data file's header

Question

I have a simulation that reads large binary data files that we create (10s to 100s of GB). We use binary for speed reasons. These files are system dependent, converted from te

Answer 1

If they're that large, I'd reserve a healthy chunk (64K?) of space at the beginning of the file and put the metadata there in XML format followed by an end-of-file character (Ctrl-Z for DOS/Windows, ctrl-D for unix?). That way you can examine and parse the metadata easily with the wide range of toolsets out there for XML.

Otherwise I go with what other people have already said: timestamp for file creation, identifier for which machine it's created on, basically anything else that you can think of for diagnostic purposes. And ideally you would include the definition of the structure format itself. If you are changing the structure often, it's a big pain to maintain the proper version of code around to read various formats of old datafiles.

One big advantage of HDF5 as @highpercomp has mentioned, is that you just don't need to worry about changes in the structure format, as long as you have some convention of what the names and datatypes are. The structure names and datatypes are all stored in the file itself, so you can blow your C code to smithereens and it doesn't matter, you can still retrieve data from an HDF5 file. It lets you worry less about the format of data and more on the structure of data, i.e. I don't care about the sequence of bytes, that's HDF5's problem, but I do care about field names and the like.

Another reason I like HDF5 is you can choose to use compression, which takes a very small amount of time and can give you huge wins in storage space if the data is slowly-changing or mostly the same except for a few errant blips of interestingness.