How can I append to a numpy array without reassigning the result to a new variable?

问题

I have a matrix M with dimensions (m, n) and I need to append new columns to it from a matrix L with dimensions (m, l). So basically I will end up with a matrix (m, n + l).

No problem in doing this, I can use:

numpy.concatenate
numpy.vstack
numpy.append

in the following fashion np.command(M, L) and it will return me a new matrix. The problem arises with the fact that I need to append many many matrices to the original matrix, and the size of these matrices L are not known beforehand.

So I ended up with

# M is my original matrix
while:
    # find out my L matrix
    M = np.append(M, L)
    # check if I do not need to append the matrix

Knowing that my matrix M has approximately 100k rows, and I add on average 5k columns, the process is super slow and takes more than couple of hours (I don't know exactly how long because I gave up after 2 hours).

The problem here is clearly in this append function (I tried it with vstack and nothing changes). Also if I just calculate matrices L (without appending them), I spend less than 10 minutes for the task. I assume that this reassigning of matrix is what makes it slow. Intuitively it makes sense because I am constantly recreating the matrix M and removing the old matrix. But I do not know how to get rid of the reassigning part.

One idea is that creating an empty matrix beforehand and then populating it with correct columns should be faster, but the problem is that I do not know with what dimensions I should create it (there is no way to predict the number of columns in my matrix).

So how can I improve performance here?

回答1:

There's no way to append to an existing numpy array without creating a copy.

The reason is that a numpy array must be backed by a contiguous block of memory. If I create a (1000, 10) array, then decide that I want to append another row, I'd need to be able to extend the chunk of RAM corresponding to the array so that it's big enough to accommodate (1001, 10) elements. In the general case this is impossible, since the adjacent memory addresses may already be allocated to other objects.

The only way to 'concatenate' arrays is to get the OS to allocate another chunk of memory big enough for the new array, then copy the contents of the original array and the new row into this space. This is obviously very inefficient if you're doing it repeatedly in a loop, especially since the copying step becomes more and more expensive as your array gets larger and larger.

Here are two possible work-arounds:

Use a standard Python list to accumulate your rows inside your while loop, then convert the list to an array in a single step, outside the loop. Appending to a Python list is very cheap compared with concatenating numpy arrays, since a list is just an array of pointers which don't necessarily have to reference adjacent memory addresses, and therefore no copying is required.
Take an educated guess at the number of rows in your final array, then allocate a numpy array that's slightly bigger and fill in the rows as you go along. If you run out of space, concatenate on another chunk of rows. Obviously the concatenation step is expensive, since you'll need to make a copy, but you're much better off doing this once or twice than on every iteration of your loop. When you're choosing the initial number of rows in your output array there will be a trade-off between avoiding over-allocating and unnecessary concatenation steps. Once you're done, you could then 'trim off' any unused rows using slice indexing.

来源：https://stackoverflow.com/questions/32177537/how-can-i-append-to-a-numpy-array-without-reassigning-the-result-to-a-new-variab

标签

python

performance

numpy