Efficient Way to Create Numpy Arrays from Binary Files

Question

I have very large datasets that are stored in binary files on the hard disk. Here is an example of the file structure:

File Header

149 Byte          


        

Answer 1

Numpy supports mapping binary from data directly into array like objects via numpy.memmap. You might be able to memmap the file and extract the data you need via offsets.

For endianness correctness just use numpy.byteswap on what you have read in. You can use a conditional expression to check the endianness of the host system:

if struct.pack('=f', np.pi) == struct.pack('>f', np.pi):
  # Host is big-endian, in-place conversion
  arrayName.byteswap(True)