MapReduce

问题 We are using 3 cluster machine and mapreduce.tasktracker.reduce.tasks.maximum property is set to 9. When I set no of reducer is equal to or less than 9 job is getting succeeded but if I set greater than 9 then it is failing with the exception "Task attempt_201701270751_0001_r_000000_0 failed to ping TT for 60 seconds. Killing!". Can any one guide me what will be the problem 回答1: There seem to be some bug in hadoop -0.20. https://issues.apache.org/jira/browse/MAPREDUCE-1905 (for reference ).

问题 My PHP server is hosted on Job Tracker machine and I am trying to run the map reduce job through my web page by calling the command line executing the jar command, but I am getting no response and job is not starting. However if I run a command to list the hdfs using same methodology it is running fine. Please guide me. Following command is not responding me anything and job is not running: exec("HADOOP_DIR/bin/hadoop jar /usr/local/MapReduce.jar Mapreduce [input Path] [output Path]"); But if

问题 I have created a 4-node hadoop cluster . I start all datanodes,namenode resource manager,etc. To find whether all of my nodes are working or not , I tried the following procedure: Step 1. I run my program when all nodes are active Step 2. I run my program when only master is active . The completion time in both cases were almost same . So, I would like to know if there is any other means by which I can know how many nodes are actually used while running the program. 回答1: Discussed in the chat

问题 I have 7 nodes in my Hadoop cluster [8GB RAM and 4VCPUs to each nodes], 1 Namenode + 6 datanodes. EDIT-1@ARNON: I followed the link, mad calculation according to the hardware configruation on my nodes and have added the update mapred-site and yarn-site.xml files in my question. Still my application is crashing with the same exection My mapreduce application has 34 input splits with a block size of 128MB. mapred-site.xml has the following properties: mapreduce.framework.name = yarn mapred

问题 I'm trying to count the items from two arrays within the same model: Model: {_id:1 name:"fun", objectsTypeA: [ objectId_1 objectId_2 ], objectsTypeB: [ objectId_5 objectId_9 ] }, {_id:2 name:"boring", objectsTypeA: [ objectId_3 objectId_4 ], objectsTypeB: [] } I'm trying to get the following result: [ { name:"fun", id: 1, count:4 }, { name:"boring", id: 2, count: 2 ] What I got so far is this: Object.aggregate([ {$project: {_id:1, name:1, objectsTypeA:1}}, {$unwind:'$objectsTypeA'}, {$group:

问题 The output folder has part-00000 file with no content! Here is the command trace where I see no exception, [cloudera@localhost ~]$ hadoop jar testmr.jar TestMR /tmp/example.csv /user/cloudera/output 14/02/06 11:45:24 WARN conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id 14/02/06 11:45:24 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 14/02/06 11:45:24 WARN mapred.JobClient: Use GenericOptionsParser for parsing the

问题 With reference to similar questions: Running a Hadoop Job From another Java Program and Calling a mapreduce job from a simple java program I too have a mapreduce job jar file in a Hadoop remote machine, and I'm creating a web application that, with a button click event, will call out to the jar file and execute the job. This web app is running on a separate machine. I've tried the suggestions from both of the posts above but could not get it to work, even working on the wordcount example

问题 This question already has an answer here : Output a list from a Hadoop Map Reduce job using custom writable (1 answer) Closed 4 years ago . How can we pass an arraylist as value from the mapper to the reducer. My code basically has certain rules to work with and would create new values(String) based on the rules.I am maintaining all the outputs(generated after the rule execution) in a list and now need to send this output(Mapper value) to the Reducer and do not have a way to do so. Can some

问题 I'm trying to learn MapReduce function in MongoDB. Instead of using an aggregation, I want to group documents in collection by key defined by myself using MapReduce function. My collection Cool is: /* 1 */ { "_id" : ObjectId("55d5e7287e41390ea7e83a55"), "id" : "a", "cool" : "a1" } /* 2 */ { "_id" : ObjectId("55d5e7287e41390ea7e83a56"), "id" : "a", "cool" : "a2" } /* 3 */ { "_id" : ObjectId("55d5e7287e41390ea7e83a57"), "id" : "b", "cool" : "b1" } /* 4 */ { "_id" : ObjectId(

问题 I know that I can submit a Cascading job by packaging it into a JAR, as detailed in the Cascading user guide. That job will then run on my cluster if I manually submit it using hadoop jar CLI command. However, in the original Hadoop 1 Cascading version, it was possible to submit a job to the cluster by setting certain properties on the Hadoop JobConf . Setting fs.defaultFS and mapred.job.tracker caused the local Hadoop library to automatically attempt to submit the job to the Hadoop1