large-files

I have a large .csv file (about 300 MB), which is read from a remote host, and parsed into a target file, but I don't need to copy all the lines to the target file. While copying, I need to read each line from the source and if it passes some predicate, add the line to the target file. I suppose that Apache CSV ( apache.commons.csv ) can only parse whole file CSVFormat csvFileFormat = CSVFormat.EXCEL.withHeader(); CSVParser csvFileParser = new CSVParser("filePath", csvFileFormat); List<CSVRecord> csvRecords = csvFileParser.getRecords(); so I can't use BufferedReader . Based on my code, a new

I have an XML file that is approximately 12mb which has about 16000 product's. I need to process it into a database; however, at about 6000 rows it dies with a 500 error. I'm using the Kohana framework (version 3) just in case that has anything to do with it. Here's my code that I have inside the controller: $xml = new XMLReader(); $xml->open("path/to/file.xml"); $doc = new DOMDocument; // Skip ahead to the first <product> while ($xml->read() && $xml->name !== 'product'); // Loop through <product>'s while ($xml->name == 'product') { $node = simplexml_import_dom($doc->importNode($xml->expand(),

I'm using wshttpbinding for my service <wsHttpBinding> <binding name="wsHttpBinding_Windows" maxBufferPoolSize="9223372036854775807" maxReceivedMessageSize="2147483647"> <readerQuotas maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxStringContentLength="2147483647" maxNameTableCharCount="2147483647"/> <security mode="Message"> <message clientCredentialType="Windows"/> </security> </binding> </wsHttpBinding> <behavior name="ServiceBehavior"> <dataContractSerializer maxItemsInObjectGraph="6553600"/> <serviceThrottling maxConcurrentCalls="2147483647" maxConcurrentInstances="2147483647"

I have a large text file that I need to parse into a pipe delimited text file using python. The file looks like this (basically): product/productId: D7SDF9S9 review/userId: asdf9uas0d8u9f review/score: 5.0 review/some text here product/productId: D39F99 review/userId: fasd9fasd9f9f review/score: 4.1 review/some text here Each record is separated by two newline charters /n . I have written a parser below. with open ("largefile.txt", "r") as myfile: fullstr = myfile.read() allsplits = re.split("\n\n",fullstr) articles = [] for i,s in enumerate(allsplits[0:]): splits = re.split("\n.*?: ",s)

I'm trying to find a definitive answer and can't, so I'm hoping someone might know. I'm developing a C++ app using GCC 4.x on Linux (32-bit OS). This app needs to be able to read files > 2GB in size. I would really like to use iostream stuff vs. FILE pointers, but I can't find if the large file #defines (_LARGEFILE_SOURCE, _LARGEFILE64_SOURCE, _FILE_OFFSET_BITS=64) have any effect on the iostream headers. I'm compiling on a 32-bit system. Any pointers would be helpful. vladr This has already been decided for you when libstdc++ was compiled, and normally depends on whether or not _GLIBCXX_USE

I am trying to read some specific rows of a large csv file, and I don't want to load the whole file into memory. The index of the specific rows are given in a list L = [2, 5, 15, 98, ...] and my csv file looks like this: Col 1, Col 2, Col3 row11, row12, row13 row21, row22, row23 row31, row32, row33 ... Using the ideas mentioned here I use the following command to read the rows with open('~/file.csv') as f: r = csv.DictReader(f) # I need to read it as a dictionary for my purpose for i in L: for row in enumerate(r): print row[i] I immediately get the following error: IndexError Traceback (most

64bit file API is different on each platform. in windows: _fseeki64 in linux: fseeko in freebsd: yet another similar call ... How can I most effectively make it more convenient and portable? Are there any useful examples? Most POSIX-based platforms support the " _FILE_OFFSET_BITS " preprocessor symbol. Setting it to 64 will cause the off_t type to be 64 bits instead of 32, and file manipulation functions like lseek() will automatically support the 64 bit offset through some preprocessor magic. From a compile-time point of view adding 64 bit file offset support in this manner is fairly