Numpy array memory issue

Question

I believe I am having a memory issue using numpy arrays. The following code is being run for hours on end:

    new_data = npy.array([new_x, new_y1, new_y2, new_y         


        

Answer 1

Update: I incorporated @EOL's excellent indexing suggestion into the answer.

The problem might be the way row_stack grows the destination. You might be better off handling the reallocation yourself. The following code allocates a big empty array, fills it, and grows it as it fills an hour at a time

numcols = 4
growsize = 60*60 #60 samples/min * 60 min/hour
numrows = 3*growsize #3 hours, to start with
private.data = npy.zeros([numrows, numcols]) #alloc one big memory block
rowctr = 0
while (recording):
    private.data[rowctr] = npy.array([new_x, new_y1, new_y2, new_y3])
    rowctr += 1
    if (rowctr == numrows): #full, grow by another hour's worth of data
        private.data = npy.row_stack([private.data, npy.zeros([growsize, numcols])])
        numrows += growsize

This should keep the memory manager from thrashing around too much. I tried this versus row_stack on each iteration and it ran a couple of orders of magnitude faster.