What are important points when designing a (binary) file format? [closed]

Answer 1

I agree that these are good ideas:

1. Magic numbers at the beginning. Pretty much required in *nix:

2. File version number for backwards compatibility.

3. Endianness specification.

But your fourth one is overkill, because #2 lets you add fields as long as you change the version number (and as long as you don't need forward compatibility).

• possibly reserve some space for further per-file attributes that might be necessary in the future?

Also, the idea of imposing a block-structure on your file, expressed in many other answers, seems less like a universal requirement for binary files than a solution to a problem with certain kinds of payloads.

In addition to 1-3 above, I'd add these:

• simple checksum or other way of detecting that the contents are intact. Otherwise you can't trust magic bytes or version numbers. Be careful to spec which bytes are included in the checksum. Typically you would include all bytes in the file that don't already have error detection.

• version of your software (including the most granular number you have, e.g. build number) that wrote the file. You're going to get a bug report with an attached file from someone who can't open it and they will have no clue when they wrote the file because the error didn't occur then. But the bug is in the version that wrote it, not in the one trying to read it.

• Make it clear in the spec that this is a binary format, i.e. all values 0-255 are allowed for all bytes (except the magic numbers).

And here are some optional ones:

• If you do need forward compatibility, you need some way of expressing which "chunks" are "optional" (like png does), so that a previous version of your software can skip over them gracefully.

• If you expect these files to be found "in the wild", you might consider embedding some clue to find the spec. Imagine how helpful it would be to find the string http://www.w3.org/TR/PNG/ in a png file.

Answer 2

Make sure that you reserve a tag code (or better yet reserve a bit in each tag) that specifies a deleted/free block/chunk. Blocks can then be deleted by simply changing a block's current tag code to the deleted tag code or set the tag's deleted bit. This way you don't need to right away completely restructure your file when you delete a block.

Reserving a bit in the tag provides the the option of possibly undeleting the block (if you leave the block's data unchanged).

For security, however you might want to zero out the deleted block's data, in this case you would use a special deleted/free tag.

I agree with Stepan, that you should choose an endianess, but I would also have an endianess indicator in the file. If you use an endianess indicator you might consider using one of the UniCode Byte Order Marks also as an inidicator of any UniCode text encoding used for any text blocks. The BOM is usually the first few bytes of UniCoded text files, so if your BOM is the first entry in your file there might be a problem of some utility identifying your file as UniCode text (I don't think this is much an issue). I would treat/reserve the BOM as one of your normal tags (using either the UTF16 BOM if using the 16bit tags or the UTF32 BOM if using 32bit tags) with a 0 length block/chunk.

See also http://en.wikipedia.org/wiki/File_format

Answer 3

Another point, taken from .xz file spec (http://tukaani.org/xz/xz-file-format.txt): one of the first few bytes should be a non-character, "to prevent applications from misdetecting the file as a text file.". Note sure how many header bytes are usually inspected by editors and other tools, but using a non-binary byte in the first four or eight bytes seems useful.

Answer 4

One of the most important things to know before even starting is how your file will be used.

• Will random or sequential access be the norm?
• How often will the data be read?
• How often will the data be written?
• Will you write out the file in one go or will you be slowing writing it as data comes in.
• Will the file need to be portable? Not all formats need to be.
• Does it need to be compatible with other versions? Maybe updating the file is sufficient.
• Does it need to be easy to read/write?
• Size/Speed/Compexity tradeoff.

Most answers here give good advise on the portability/compatibility front so I am not going to add more. But consider the following (often overlooked) things.

• Some files are often written and rarely read (backups, logs, ...) and you may want to focus on filesize and easy-writing.
• Converting endianness is slow (relatively) if your file will never leave the host, or leaves rarely enough that conversion is a good option you can get a significant performance boost. Consider writing a number such as 0x1234 as part of the header so that you can detect (and instruct the user to convert) if this is the case.
• Sometimes easy reading is really useful. If you are doing logs or text documents, consider compressing all in one go rather than per-entry so that you can zcat | strings the file and see what is inside.

There are many things to keep in mind and designing a good format takes a lot of planning and foresight. The little things such as zcating a file and getting useful information or the small performance boost from using native integers can give your product an edge, however you need to be careful that you don't sacrifice something important to get it.