Efficient Way to Create Numpy Arrays from Binary Files

Question

I have very large datasets that are stored in binary files on the hard disk. Here is an example of the file structure:

File Header

149 Byte          


        

Answer 1

Some hints:

• Don't use the struct module. Instead, use Numpy's structured data types and fromfile. Check here: http://scipy-lectures.github.com/advanced/advanced_numpy/index.html#example-reading-wav-files

• You can read all of the records at once, by passing in a suitable count= to fromfile.

Something like this (untested, but you get the idea):

import numpy as np

file = open(input_file, 'rb')
header = file.read(149)

# ... parse the header as you did ...

record_dtype = np.dtype([
    ('timestamp', '