MapReduce

问题 This link "http://had00b.blogspot.com/2013/07/random-subset-in-mapreduce.html" talks about how one can implement reservoir sampling using map reduce framework. I feel their solution is complicated and the following simpler approach would work. Problem: Given very large number of samples, generate a set of size k such that each sample has equal probability of being present in the set. Proposed solution: Map operation: For each input number n, output (i, n) where i is randomly chosen in range 0

问题 OOZIE wordcount example gives JA009: RPC response exceeds maximum data length. We have doubled the ipc.maximum.data.length and restarted the NameNode. 2018-12-05 17:55:45,914 WARN MapReduceActionExecutor:523 - SERVER[******] USER[******] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000004-181205174411487-oozie-******-W] ACTION[0000004-181205174411487-oozie-******-W@mr-node] No credential properties found for action : 0000004-181205174411487-oozie-******-W@mr-node, cred : null 2018-12-05 18:10:46

问题 I have a jar in which one MapReduce class works fine while the other class with same structure - proper use of Tool, use of getConf(), etc. - fails with error 'Found interface org.apache.hadoop.mapreduce.jobcontext but class expected'. Any specific places that I should look for to fix this? Just about any help/clue would be great! Edit: Other people with the same issue (no answer as yet on that thread either): https://groups.google.com/forum/#!msg/hipi-users/LSvktkk1YdI/yssjjc7cjeIJ 回答1: you

问题 I am running 3 node cluster on Ubuntu 12.04 LTS Server and hadoop 1.2.1 installed on it with JDK 1.7, now for very first check whether map-reduce jobs are executing or not I executed wordcount from hadoop-examples-1.2.1.jar and got the stunning error: 14/02/20 20:26:52 INFO mapred.JobClient: Running job: job_201402202023_0002 14/02/20 20:26:53 INFO mapred.JobClient: map 0% reduce 0% 14/02/20 20:26:57 INFO mapred.JobClient: Task Id :attempt_201402202023_0002_m_000005_0, Status : FAILED java

问题 We are using HDP 2.4 and have many map reduce jobs written in various ways ( java MR / Hive / etc. ) . The logs are collect in hadoop file system under the application ID. I want to collect all the logs of application and append in single file (hdfs or OS files of one machine) so that I can analyze my application log in a single location with out hassle . Also advise me the best way to achieve in HDP 2.4 ( Stack version info => HDFS 2.7.1.2.4 / YARN 2.7.1.2.4 / MapReduce2 2.7.1.2.4 / Log

问题 In our HBase table, each row has a column called crawl identifier. Using a MapReduce job, we only want to process at any one time rows from a given crawl. In order to run the job more efficiently we gave our scan object a filter that (we hoped) would remove all rows except those with the given crawl identifier. However, we quickly discovered that our jobs were not processing the correct number of rows. I wrote a test mapper to simply count the number of rows with the correct crawl identifier,

问题 I have 4 system cluster and Apache Nutch 2.3.1 is configured to crawl few website. After crawling, I have to change their score little big by some custom job. In job, the mapper is just combining the documents based on domain as key. While is reducer, I sum their effective text bytes and find the average. Later I assign the log of average bytes as score. But reducer job took 14 hours and then timeout has occured. While in Nutch builtin job e.g., updatedb is finished in 3 to 4 hours. Where is

问题 I am trying to use a Map Reduce to count number documents according to one of the field values per date. First, here are the results from a couple of regular find() functions: db.errors.find({ "cDate" : ISODate("2012-11-20T00:00:00Z") }).count(); returns 579 (ie. there are 579 documents for this date) db.errors.find( { $and: [ { "cDate" : ISODate("2012-11-20T00:00:00Z") }, {"Type":"General"} ] } ).count() returns 443 (ie. there are 443 documents for this date where Type="General") Following

问题 I wrote MR script which should load data from HBase and dump them into Hive. Connecting to HBase is ok, but when I try to save data into HIVE table, I get following error message: Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, org.apache.hive.hcatalog.common.HCatException : 2004 : HCatOutputFormat not initialized, setOutput has to be called org.apache.oozie.action.hadoop.JavaMainException: org.apache.hive.hcatalog.common.HCatException :

问题 I'm trying to execute MapReduce job using oozie workflow in hue. When I submit the job, oozie successfully executes but I don't get the expected output. It seems that either mapper or reducer never invoked.here is my workflow.xml: <workflow-app name="wordCount" xmlns="uri:oozie:workflow:0.4"> <start to="wordcount"/> <action name="wordcount"> <map-reduce> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.input.dir</name> <value>