large-files

I have a large data set in GBs that I'd have to process before I analyse them. I tried creating a connector, which allows me to loop through the large datasets and extract chunks at a time.This allows me to quarantine data that satisfies some conditions. My problem is that I am not able to create an indicator for the connector that stipulates it is null and to execute close(connector) when the end of the dataset is reached. Moreover, for the first chunk of extracted data, I'd have to skip 17 lines since the file contains header that R is not able to read. A manual attempt that works: filename=

I'm trying to read a 17MB excel file (2003) with PHPExcel1.7.3c, but it crushes already while loading the file, after exceeding the 120 seconds limit I have. Is there another library that can do it more efficiently? I have no need in styling, I only need it to support UTF8. Thanks for your help Filesize isn't a good measure when using PHPExcel, it's more important to get some idea of the number of cells (rowsxcolumns) in each worksheet. If you have no need for styling, are you calling: $objReader->setReadDataOnly(true); before loading the file? If you don't need to access all worksheets, or

I have the following code: def application(env, start_response): path = process(env) fh = open(path,'r') start_response('200 OK', [('Content-Type','application/octet-stream')]) return fbuffer(fh,10000) def fbuffer(f, chunk_size): '''Generator to buffer file chunks''' while True: chunk = f.read(chunk_size) if not chunk: break yield chunk I'm not sure that it's right but the scraps of information I've found on the internet have led me to think it ought to work. Basically I want to stream a file out in chunks, and to do that I'm passing a generator back from my application function. However this

I'm writing a large file with node.js using a writable stream : var fs = require('fs'); var stream = fs.createWriteStream('someFile.txt', { flags : 'w' }); var lines; while (lines = getLines()) { for (var i = 0; i < lines.length; i++) { stream.write( lines[i] ); } } I'm wondering if this scheme is safe without using drain event? If it is not (which I think is the case), what is the pattern for writing an arbitrary large data to a file? That's how I finally did it. The idea behind is to create readable stream implementing ReadStream interface and then use pipe() method to pipe data to writable

I'm trying to implement Git to manage creative assets (Photoshop, Illustrator, Maya, etc.), and I'd like to exclude files from Git based on file size rather than extension, location, etc. For example, I don't want to exclude all .avi files, but there are a handful of massive +1GB avi files in random directories that I don't want to commit. Any suggestions? I'm new to .gitignore, so there may be better ways to do this, but I've been excluding files by file size using: find . -size +1G | cat >> .gitignore Obviously you'll have to run this code frequently if you're generating a lot of large files

I am looking for a text editor that will be able to load a 4+ Gigabyte file into it. Textpad doesn't work. I own a copy of it and have been to its support site, it just doesn't do it. Maybe I need new hardware, but that's a different question. The editor needs to be free OR, if its going to cost me, then no more than $30. For Windows. VonC glogg could also be considered, for a different usage: Caveat (reported by Simon Tewsi in the comments , Feb. 2013) One caveat - has two search functions, Main Search and Quick Find . The lower one, which I assume is Quick Find , is at least an order of