Efficient data structure for storing N lists where N is very large

问题

I will need to store N lists, where N is large (1 million). For example,

[2,3]
[4,5,6]
...
[4,5,6,7]

Each item is a list of about 0-10000 elements. I wanted to use a numpy array of lists, like

np.array([[2,3],[4,5,6])

Then I got efficiency issues when trying to append to the lists in the numpy array. Also I was told here: Efficiently append an element to each of the lists in a large numpy array, to not use numpy array of lists.

What would be a good data structure for storing such data, in terms of memory and time efficiency?

回答1:

Unless the elements youre storing follow some pattern you must use nested list since there is no other way to get those elements out of the others.

In Python:

listOfLists = [[1,2,3],
               [4,5,6],
               [7,8,9]]

So whenever you want to operate with this list you can use numpy functions

>>> np.mean(listOfLists)
5.0
>>> np.max(listOfLists)
9

回答2:

Maybe use a dictionary:

d={}
for i in range(N):
  d[i]=your_nth_list

And you will simply append them by:

d[k].append(additional_items)

(It's efficient for 10.000.000 lists of 1000 items each)

回答3:

try nested list

nestedList = [[2,3],[4,5,6]]

回答4:

You could use nested lists but they are not efficent in terms of complexity. In fact, it is linear, you could use dictionaries to get better results :

dict={}
for i in range(numer_of_lists) :
  dict[str(i)]=your_i-th_list

Then access the i-th element withdict[str(i)] Then, appening an element will be as easy

`

来源：https://stackoverflow.com/questions/64440765/efficient-data-structure-for-storing-n-lists-where-n-is-very-large