bigdata

I am able to connect tableau with my database but the table size is really large here. Everytime I try to load the table into tableau, it is crashing and I am not able to find any work around. The table size varies from 10 million - 400 million rows. How should I approach this issue any suggestion ? I found a simple solution for optimising Tableau to work with very large datasets (1 billion+ rows): Google BigQuery, which is essentially a managed data warehouse. Upload data to BigQuery (you can append multiple files into a single table). Link that table to Tableau as an external data source

I am scoping out a project with large, mostly-uncompressible time series data, and wondering if Django + Postgres with raw SQL is the right call. I have time series data that is ~2K objects/hour, every hour. This is about 2 million rows per year I store, and I would like to 1) be able to slice off data for analysis through a connection, 2) be able to do elementary overview work on the web, served by Django. I think the best idea is to use Django for the objects themselves, but drop to raw SQL to deal with the large time series data associated. I see this as a hybrid approach; that might be a

I have a vector x of 1,344 unique strings. I want to generate a matrix that gives me all possible groups of three values, regardless of order, and export that to a csv. I'm running R on EC2 on a m1.large instance w 64bit Ubuntu. When using combn(x, 3) I get an out of memory error: Error: cannot allocate vector of size 9.0 Gb The size of the resulting matrix is C1344,3 = 403,716,544 rows and three columns - which is the transpose of the result of combn() function. I thought of using the bigmemory package to create a file backed big.matrix so I can then assign the results of the combn() function

What are the differences between Sort Comparator and Group Comparator in Hadoop? Eswara Reddy Adapa To understand GroupComparator , see my answer to this question - What is the use of grouping comparator in hadoop map reduce SortComparator :Used to define how map output keys are sorted Excerpts from the book Hadoop - Definitive Guide: Sort order for keys is found as follows: If the property mapred.output.key.comparator.class is set, either explicitly or by calling setSortComparatorClass() on Job, then an instance of that class is used. (In the old API the equivalent method is

When due to very large data calculations will take a long time and, hence, we don't want them to crash, it would be valuable to know beforehand which reshape method to use. Lately, methods for reshaping data have been further developed regarding performance, e.g. data.table::dcast and tidyr::spread . Especially dcast.data.table seems to set the tone [1] , [2] , [3] , [4] . This makes other methods as base R's reshape in benchmarks seem outdated and almost useless [5] . Theory However , I've heard that reshape was still unbeatable when it comes to very large datasets (probably those exceeding