Is there a way to circumvent Python list.append() becoming progressively slower in a loop as the list grows?

Question

I have a big file I\'m reading from, and convert every few lines to an instance of an Object.

Since I\'m looping through the file, I stash the instance to a list usi

Answer 1

There is nothing to circumvent: appending to a list is O(1) amortized.

A list (in CPython) is an array at least as long as the list and up to twice as long. If the array isn't full, appending to a list is just as simple as assigning one of the array members (O(1)). Every time the array is full, it is automatically doubled in size. This means that on occasion an O(n) operation is required, but it is only required every n operations, and it is increasingly seldom required as the list gets big. O(n) / n ==> O(1). (In other implementations the names and details could potentially change, but the same time properties are bound to be maintained.)

Appending to a list already scales.

Is it possible that when the file gets to be big you are not able to hold everything in memory and you are facing problems with the OS paging to disk? Is it possible it's a different part of your algorithm that doesn't scale well?