Fast and efficient way to read a space separated file of numbers into an array?

Question

I need a fast and efficient method to read a space separated file with numbers into an array. The files are formatted this way:

4 6
1 2 3 4 5 6
2 5 4 3 21111 101         


        

Answer 1

What's your usage pattern for the data once it's loaded? Do you generally need to touch every array element or will you just make sparse/random access?

If you need to touch most array elements, loading it into memory will probably be the best way to go.

If you need to just access certain elements, you might want to lazy load the elements that you need into memory. One strategy would be to determine which of the two layouts the file uses (with/without newline) and create an algorithm to load a particular element directly from disk as needed (seek the given file offset, read and parse). To efficiently re-access the same element it could make sense to keep the element, once read, in a dictionary indexed by the offset. Check the dictionary first before going to the file for a particular value.

On general principal I would take the simple route unless your testing proves that you need to go a more complicated route (avoid premature optimization).