large-data

given large datasets that don't fit in memory, is there any library or api to perform sort in Java? the implementation would possibly be similar to linux utility sort. Java provides a general-purpose sorting routine which can be used as part of the larger solution to your problem. A common approach to sort data that's too large to all fit in memory is this: 1) Read as much data as will fit into main memory, let's say it's 1 Gb 2) Quicksort that 1 Gb (here's where you'd use Java's built-in sort from the Collections framework) 3) Write that sorted 1 Gb to disk as "chunk-1" 4) Repeat steps 1-3

I need to add row numbers to a large (ca. billion rows) dataset in BigQuery. When I try: SELECT * ROW_NUMBER() OVER (ORDER BY d_arf DESC) plarf FROM [trigram.trigrams8] I get "Resources exceeded during query execution.", because an analytic/window function needs to fit in one node. How can I add row numbers to a large dataset in BigQuery? You didn't give me a working query, so I had to create my own, so you'll need to translate it to your own problem space. Also I'm not sure why do you want to give a row number to each row in such a huge dataset, but challenge accepted: SELECT a.enc, plarf,

I want to read data from large xml file (20 GB) and manipulate them. I tired to use "xmlParse()" but it gave me memory issue before loading. Is there any efficient way to do this? My data dump looks like this, <tags> <row Id="106929" TagName="moto-360" Count="1"/> <row Id="106930" TagName="n1ql" Count="1"/> <row Id="106931" TagName="fable" Count="1" ExcerptPostId="25824355" WikiPostId="25824354"/> <row Id="106932" TagName="deeplearning4j" Count="1"/> <row Id="106933" TagName="pystache" Count="1"/> <row Id="106934" TagName="jitter" Count="1"/> <row Id="106935" TagName="klein-mvc" Count="1"/> <

I need to find for each point of the data set all its nearest neighbors. The data set contains approx. 10 million 2D points. The data are close to the grid, but do not form a precise grid... This option excludes (in my opinion) the use of KD Trees, where the basic assumption is no points have same x coordinate and y coordinate. I need a fast algorithm O(n) or better (but not too difficult for implementation :-)) ) to solve this problem ... Due to the fact that boost is not standardized, I do not want to use it ... Thanks for your answers or code samples... I would do the following: Create a

I'm new using R. I'm trying to add new lines to a file with my existing data in R. The problem is that my data is of about 30000 rows and 13000 cols. I already try to add a line with the writeLines function but the resulted file contains only the line added. Have you tried using the write function? line="blah text blah blah etc etc" write(line,file="myfile",append=TRUE) Rainer write.table , write.csv and others all have the append= argument, which appends append=TRUE and usually overwrites if append=FALSE . So which one you want to / have to use, depends on your data. By the way, cat() can

I'd like to create a 1D NumPy array that would consist of 1000 back-to-back repetitions of another 1D array, without replicating the data 1000 times. Is it possible? If it helps, I intend to treat both arrays as immutable. Paul You can't do this; a NumPy array must have a consistent stride along each dimension, while your strides would need to go one way most of the time but sometimes jump backwards. The closest you can get is either a 1000-row 2D array where every row is a view of your first array, or a flatiter object , which behaves kind of like a 1D array. (flatiters support iteration and