Loading large amount of data into memory - most efficient way to do this?

Question

I have a web-based documentation searching/viewing system that I\'m developing for a client. Part of this system is a search system that allows the client to search for a t

Answer 1

I tested three methods of loading the same 9,000,000 point dataset into Firefox 3.64.

1: Stephen's GetJSON Method
2) My function based push method
3) My pre-processed array appending method:

I ran my tests two ways: The first iteration of testing I imported 100 files containing 10,000 rows of data, each row containing 9 data elements [0,1,2,3,4,5,6,7,8]

The second interation I tried combining files, so that I was importing 1 file with 9 million data points.

This was a lot larger than the dataset I'll be using, but it helps demonstrate the speed of the various import methods.

Separate files:                 Combined file:

JSON:        34 seconds         34
FUNC-BASED:  17.5               24
ARRAY-BASED: 23                 46

Interesting results, to say the least. I closed out the browser after loading each webpage, and ran the tests 4 times each to minimize the effect of network traffic/variation. (ran across a network, using a file server). The number you see is the average, although the individual runs differed by only a second or two at most.