hadoop2

I have 100 mapper and 1 reducer running in a job. How to improve the job performance? As per my understanding: Use of combiner can improve the performance to great extent. But what else we need to configure to improve the jobs performance? With the limited data in this question ( Input file size, HDFS block size, Average map processing time, Number of Mapper slots & Reduce slots in cluster etc.), we can't suggest tips. But there are some general guidelines to improve the performance. If each task takes less than 30-40 seconds , reduce the number of tasks If a job has more than 1TB of input ,

I managed to launch a spark application on Yarn. However emory usage is kind of weird as you can see below : http://imgur.com/1k6VvSI What does memory reserved mean ? How can i manage to efficiently use all the memory available ? Thanks in advance. Check out this blog from Cloudera that explains the new memory management in YARN. Here's the pertinent bits: ... An implementation detail of this change that prevents applications from starving under this new flexibility is the notion of reserved containers. Imagine two jobs are running that each have enough tasks to saturate more than the entire

I'm a dummy on Ubuntu 16.04, desperately attempting to make Spark work. I've tried to fix my problem using the answers found here on stackoverflow but I couldn't resolve anything. Launching spark with the command ./spark-shell from bin folder I get this message WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable". I'm using Java version is java version "1.8.0_101 Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode). Spark is the latest version: 2.0.1 with

The Namenode in the Hadoop architecture is a single point of failure. How do people who have large Hadoop clusters cope with this problem?. Is there an industry-accepted solution that has worked well wherein a secondary Namenode takes over in case the primary one fails ? Yahoo has certain recommendations for configuration settings at different cluster sizes to take NameNode failure into account. For example: The single point of failure in a Hadoop cluster is the NameNode. While the loss of any other machine (intermittently or permanently) does not result in data loss, NameNode loss results in

I'm new to hadoop and I'm trying to understand it. Im talking about hadoop 2. When I have an input file which I wanto to do a MapReduce, in the MapReduce programm I say the parameter of the Split, so it will make as many map tasks as splits,right? The resource manager knows where the files are and will send the tasks to the nodes who have the data, but who says how many nodes will do the tasks? After the maps are donde there is the shuffle, which node will do a reduce task is decided by the partitioner who does a hash map,right? How many nodes will do reduce tasks? Will nodes who have done