Append numpy matrix to binary file without numpy header

问题

I continously receive new data as numpy matrices which I need to append to an existing file. Structure and data type of that file are fixed, so I need python to do the conversion for me.

For a single matrix, this works:

myArr = np.reshape(np.arange(15), (3,5))
myArr.tofile('bin_file.dat')

But suppose I'd want to keep appending the existing file with more and more arrays, then numpy.tofile will overwrite any content it finds in the file, instead of appending.

I found that I could also keep doing this:

bfile = open('bin_file.dat', 'ab')
np.save(bfile, myArr)
bfile.close()

which successfully appends to the binary file. But numpy.save on the other hand does not store the raw binary data, but also saves a header (I assume) which makes the file unreadable by foreign software (I need raw binary with float32).

Reading in the existing content of the file with numpy.fromfile, appending my data and saving it again is not an option, since the file becomes very large and all the I/O would take forever to process.

Is there anything like an append mode for numpy.tofile? Are there other possibilities I am currently missing?

回答1:

It's possible to append to a tofile file:

In [344]: with open('test1',mode='ba+') as f:
     ...:     np.arange(3).tofile(f)
     ...:     np.arange(5).tofile(f)
     ...:     
In [345]: np.fromfile('test1',dtype=int)
Out[345]: array([0, 1, 2, 0, 1, 2, 3, 4])

This saves the array data without shape or dtype information. So the load has to specify dtype. And any reshaping is up to you.

来源：https://stackoverflow.com/questions/51086704/append-numpy-matrix-to-binary-file-without-numpy-header