large-data

Is it possible to use numpy.memmap to map a large disk-based array of strings into memory? I know it can be done for floats and suchlike, but this question is specifically about strings. I am interested in solutions for both fixed-length and variable-length strings. The solution is free to dictate any reasonable file format. If all the strings have the same length, as suggested by the term "array", this is easily possible: a = numpy.memmap("data", dtype="S10") would be an example for strings of length 10. Edit : Since apparently the strings don't have the same length, you need to index the

i have a db in phpmyadmin having 3000000 records. i want to export this to another pc. now when i export this only 200000 entries exported into .sql file and that is also not imported on the other pc. Answering this for anyone else who lands here. If you can only use phpMyAdmin because you do not have SSH access to the MySQL service or do not know how to use command line tools, then this might help. However as the comment above suggest, exporting a database of this size would be far easer with mysqldump. phpMyAdmin (I'm using v3.5.6) allows tables to be exported individually like so: Select

We're developing and maintaining a couple of systems, which need to export reports in Excel format to the end user. The reports are gathered from a MySQL database with some trivial processing and usually result in ~40000 rows of data with 10-15 columns, we're expecting the amount of data to grow steadily. At the moment we're using PHPExcel for the Excel generation, but it's not working for us anymore. After we go above 5000 rows, the memory consumption and loading times become untolerable, and can't be solved by indefinitely increasing PHP's maximum limits for memory usage and script execution

I'm using fread in data.table (1.8.8, R 3.0.1) in a attempt to read very large files. The file in questions has 313 rows and ~6.6 million cols of numeric data rows and the file is around around 12gb. This is a Centos 6.4 with 512GB of RAM. When I attempt to read in the file: g=fread('final.results',header=T,sep=' ') 'header' changed by user from 'auto' to TRUE Error: protect(): protection stack overflow I tried starting R with --max-ppsize 500000 , which is the max, but the same error. I also tried setting the stack size to unlimited via ulimit -s unlimited Virtual memory was already set to

Imagine you have the following array of integers: array(1, 2, 1, 0, 0, 1, 2, 4, 3, 2, [...] ); The integers go on up to one million entries; only instead of being hardcoded they've been pre-generated and stored in a JSON formatted file (of approximately 2MB in size). The order of these integers matters, I can't randomly generate it every time because it should be consistent and always have the same values at the same indexes. If this file is read back in PHP afterwards (e.g. using file_get_contents + json_decode ) it takes from 700 to 900ms just to get the array back — "Okay" I thought, "it's

Consider the code below. <GridBody Rows={rows} /> and imagine that rows.length would amount to any value 2000 or more with each array has about 8 columns in this example. I use a more expanded version of this code to render a part of a table that has been bottle necking my web application. var GridBody = React.createClass({ render: function () { return <tbody> {this.props.Rows.map((row, rowKey) => { return this.renderRow(row, rowKey); })} </tbody>; }, renderRow: function (row, rowKey) { return <tr key={rowKey}> {row.map((col, colKey) => { return this.renderColumn(col, colKey); })} </tr>; },

Closed. This question is off-topic. It is not currently accepting answers. Learn more . Want to improve this question? Update the question so it's on-topic for Stack Overflow. By "handle" I mean manipulate multi-columnar rows of data. How does R stack up against tools like Excel, SPSS, SAS, and others? Is R a viable tool for looking at "BIG DATA" (hundreds of millions to billions of rows)? If not, which statistical programming tools are best suited for analysis large data sets? If you look at the High-Performance Computing Task View on CRAN, you will get a good idea of what R can do in a sense