bigdata

hi i wanted to know the basic difference between jobconf and job objects,currently i am submitting my job like this JobClient.runJob(jobconf); i saw other way of submitting jobs like this Configuration conf = getConf(); Job job = new Job(conf, "secondary sort"); job.waitForCompletion(true); return 0; and how can i specify the sortcomparator class for the job using jobconf? can any one explain me this concept? In short: JobConf and everything else in the org.apache.hadoop.mapred package is part of the old API used to write hadoop jobs, Job and everything in the org.apache.hadoop.mapreduce

I'm using Cassandra 2.0.9 for store quite big amounts of data, let's say 100Gb, in one column family. I would like to export this data to CSV in fast way. I tried: sstable2json - it produces quite big json files which are hard to parse - because tool puts data in one row and uses complicated schema (ex. 300Mb Data file = ~2Gb json), it takes a lot of time to dump and Cassandra likes to change source file names according its internal mechanism COPY - causes timeouts on quite fast EC2 instances for big number of records CAPTURE - like above, causes timeouts reads with pagination - I used

I have a small Scala program that runs fine on a single-node. However, I am scaling it out so it runs on multiple nodes. This is my first such attempt. I am just trying to understand how the RDDs work in Spark so this question is based around theory and may not be 100% correct. Let's say I create an RDD: val rdd = sc.textFile(file) Now once I've done that, does that mean that the file at file is now partitioned across the nodes (assuming all nodes have access to the file path)? Secondly, I want to count the number of objects in the RDD (simple enough), however, I need to use that number in a

I know what is Data Warehouse & what is Big Data. But I am confused with Data Warehouse Vs Big Data. Both are same with different names or both are different(Conceptually & Physically). I know that this is an older thread but there have been some developments in the last year or so. Comparing the data warehouse to Hadoop is like comparing apples to oranges. The data warehouse is a concept: clean, integrated data of high quality. I don't think the need for a data warehouse will go away anytime soon. Hadoop on the other hand is a technology. It is a distributed compute framework to process large

I couldn't find any plain English explanations regarding Apache Parquet files. Such as: What are they? Do I need Hadoop or HDFS to view/create/store them? How can I create parquet files? How can I view parquet files? Any help regarding these questions is appreciated. What is Apache Parquet? Apache Parquet is a binary file format that stores data in a columnar fashion. Data inside a Parquet file is similar to an RDBMS style table where you have columns and rows. But instead of accessing the data one row at a time, you typically access it one column at a time. Apache Parquet is one of the modern

Just wondering if anyone has any information on the status of project Rassilon, Neo4j's side project which focuses on improving horizontal scalability of Neo4j? It was first announced in January 2013 here . I'm particularly interested in knowing more about when the graph size limitation will be removed and when sharding across clusters will become available. Philip Rathle The node & relationship limits are going away in 2.1, which is the next release post 2.0 (which now has a release candidate). Rassilon is definitely still in the mix. That said, that work is not taking precedence over things