Numpy concatenate is slow: any alternative approach?

Question

I am running the following code:

for i in range(1000)
    My_Array=numpy.concatenate((My_Array,New_Rows[i]), axis=0)

The above code is slow

Answer 1

It depends on what New_Rows[i] is, and what kind of array do you want. If you start with lists (or 1d arrays) that you want to join end to end (to make a long 1d array) just concatenate them all at once. Concatenate takes a list of any length, not just 2 items.

 np.concatenate(New_Rows, axis=0)

or maybe use an intermediate list comprehension (for more flexibility)

 np.concatenate([row for row in New_Rows])

or closer to your example.

 np.concatenate([New_Rows[i] for i in range(1000)])

But if New_Rows elements are all the same length, and you want a 2d array, one New_Rows value per row, np.array does a nice job:

 np.array(New_Rows)
 np.array([i for i in New_Rows])
 np.array([New_Rows[i] for i in range(1000)])

np.array is designed primarily to build an array from a list of lists.

np.concatenate can also build in 2d, but the inputs need to be 2d to start with. vstack and stack can take care of that. But all those stack functions use some sort of list comprehension followed by concatenate.

In general it is better/faster to iterate or append with lists, and apply the np.array (or concatenate) just once. appending to a list is fast; much faster than making a new array.