NumPy array/matrix of mixed types

Question

I\'m trying to create a NumPy array/matrix (Nx3) with mixed data types (string, integer, integer). But when I\'m appending this matrix by adding some data, I get an error: <

Answer 1

First, numpy stores array elements using fixed physical record sizes. So, record objects need to all be the same physical size. For this reason, you need to tell numpy the size of the string or save a pointer to a string stored somewhere else. In a record array, 'S' translates into a zero-length string, and that's probably not what you intended.

The append method actually copies the entire array to a larger physical space to accommodate the new elements. Try, for example:

import numpy as np
mtype = 'S10, i4, i4'
ta = np.zeros((0), dtype=mtype)
print id(ta)
ta = np.append(ta, np.array([('first', 10, 11)], dtype=mtype))
print id(ta)
ta = np.append(ta, np.array([('second', 20, 21)], dtype=mtype))
print id(ta)

Each time you append this way, the copy gets slower because you need to allocate and copy more memory each time it grows. That's why the id returns a different value every time you append. If you want any significant number of records in your array, you are much better off either allocating enough space from the start, or else accumulating the data in lists and then collecting the lists into a numpy structured array when you're done. That also gives you the opportunity to make the string length in mtype as short as possible, while still long enough to hold your longest string.