bigdata

After reading What is hive, Is it a database? , a colleague yesterday mentioned that he was able to filter a 15B table, join it with another table after doing a "group by", which resulted in 6B records, in only 10 minutes! I wonder if this would be slower in Spark, since now with the DataFrames, they may be comparable, but I am not sure, thus the question. Is Hive faster than Spark? Or this question doesn't have meaning? Sorry, for my ignorance. He uses the latest Hive, which from seems to be using Tez. Hive is just a framework that gives sql functionality to MapReduce type workloads. These

Does anyone have the list of hive error codes? For example, if we get a table not found error in hive, the value of "echo $?" will be 17. If you look at https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java you'll see most of the error codes. Although in order to map these to an exit code, you'd probably have to walk through the CLI code to trace that. 来源： https://stackoverflow.com/questions/49063409/does-anyone-have-a-list-of-hive-error-codes

I want to cluster a dataset (600000 observations), and for each cluster I want to get the principal components. My vectors are composed by one email and by 30 qualitative variables. Each quantitative variable has 4 classes: 0,1,2 and 3. So first thing I'm doing is to load the library FactoMineR and to load my data: library(FactoMineR) mydata = read.csv("/home/tom/Desktop/ACM/acm.csv") Then I'm setting my variables as qualitative (I'm excluding the variable 'email' though): for(n in 1:length(mydata)){mydata[[n]] <- factor(mydata[[n]])} I'm removing the emails from my vectors: mydata2 = mydata[2

What is the difference between the following transformations when they are executed right before writing RDD to a file? coalesce(1, shuffle = true) coalesce(1, shuffle = false) Code example: val input = sc.textFile(inputFile) val filtered = input.filter(doSomeFiltering) val mapped = filtered.map(doSomeMapping) mapped.coalesce(1, shuffle = true).saveAsTextFile(outputFile) vs mapped.coalesce(1, shuffle = false).saveAsTextFile(outputFile) And how does it compare with collect()? I'm fully aware that Spark save methods will store it with HDFS-style structure, however I'm more interested in data

My project is an ASP.NET MVC 4 project. While in localhost it works fine when I host it in Azure I get a timeout in ajax calls that take more than 4 minutes. I am sure that the problem is with azure because it doesn't matter what I'm doing in the server. even just set Thread.sleep(300000) I get a timeout. I read in: https://azure.microsoft.com/en-us/blog/new-configurable-idle-timeout-for-azure-load-balancer/ That a common practice to keep the connection active for a longer period is to use TCP Keep-alive and there is no other option for web apps. So I guess that what I need is help to keep

I want to run a mixed model (using lme4::lmer ) on 60M observations of the following format; all predictor/dependent variables are categorical (factors) apart from the continuous dependent variable tc ; patient is the grouping variable for a random intercept term. I have 64-bit R and 16Gb RAM and I'm working from a central server. RStudio is the most recent server version. model <- lmer(tc~sex+age+lho+atc+(1|patient), data=master,REML=TRUE) lho sex tc age atc patient 18 M 16.61 45-54 H 628143 7 F 10.52 12-15 G 2013855 30 M 92.73 35-44 N 2657693 19 M 24.92 70-74 G 2420965 12 F 17.44 65-69 A