MapReduce

问题 I have many small input files, and I want to combine them using some input format like CombineFileInputFormat to launch fewer mapper tasks. I know I can use Java API to do this, but I don't know whether there's a streaming jar library to support this function while I'm using Hadoop streaming. 回答1: Hadoop streaming uses TextInputFormat by default but any other input format can be used, including CombineFileInputFormat . You can change the input format from the command line, using the option

问题 I have five map reduce that I am running each separately. I want to pipeline them all together. So, output of one job goes to next job. Currently, I wrote shell script to execute them all. Is there a way to write this in java? Please provide an example. Thanks 回答1: You may find JobControl to be the simplest method for chaining these jobs together. For more complex workflows, I'd recommend checking out Oozie. 回答2: Hi I had similar requirement One way to do this is after submitting first job

问题 i have configured hadoop on two machine. i can access both machine without password using ssh.i have successfully formatted namenode using following command:-- bin/hadoop namenode -format then i tried to run pi example which shipped with hadoop.tar sandip@master:~/hadoop-1.0.4$ bin/hadoop jar hadoop-examples-1.0.4.jar pi 5 500 Number of Maps = 5 Samples per Map = 500 13/04/14 04:13:04 INFO ipc.Client: Retrying connect to server: master/192.168.188.131:9000. Already tried 0 time(s). 13/04/14

问题 I have 6 fields in a csv file: first is student name ( String ) others are student's marks like subject 1 , subject 2 etc I am writing mapreduce in java, splitting all fields with comma and sending student name in key and marks in value of map. In reduce I'm processing them outputting student name in key and theire marks plus total, average, etc in value of reduce . I think there may be an alternative, and more efficient way to do this. Has anyone got an idea of a better way to do this these

问题 Mapper: The Mapper class is a generic type, with four formal type parameters that specify the input key, input value, output key, and output value types of the map function public class MaxTemperatureMapper extends Mapper<LongWritable, Text, Text, IntWritable> { private static final int MISSING = 9999; @Override public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); String year = line.substring(15, 19); int

问题 I'm writing a mapreduce job over HBase using table mapper. I want to skip rows that don't have specific columns. For example, if the mapper reads from the "meta" family, "source" qualifier column, the mapper should expect something to be in that column. I know I can add columns to the scan object, but I expect this merely limits which rows can be seen by the scan, not which columns need to be there. What filter can I use to skip rows without the columns I need? Also, the filter concept itself

问题 I want to write Java program which reads input from HDFS, processes it using MapReduce and writes the output into a MongoDb. Here is the scenario: I have a Hadoop Cluster which has 3 datanodes. A java program reads the input from the HDFS, processes it using MapReduce. Finally, write the result into a MongoDb. Actually, reading from HDFS and processing it with MapReduce are simple. But I gets stuck about writing the result into a MongoDb. Is there any Java API supported to write the result

问题 I have a file called data that looks like this: (note there are tabs after the 'personA') personA (1, 2, 3) personB (2, 1, 34) And I have an Apache pig script like this: A = LOAD 'data' AS (name: chararray, nodes: tuple(a:int, b:int, c:int)); C = foreach A generate nodes.$0; dump C; The output of which makes sense: (1) (2) However if I change the schema of the script to be like this: A = LOAD 'data' AS (name: chararray, nodes: tuple()); C = foreach A generate nodes.$0; dump C; Then the output

问题 I am new in MapReduce and I wanted to ask if someone can give me an idea to perform word length frequency using MapReduce. I've already have the code for word count but I wanted to use word length, this is what I've got so far. public class WordCount { public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, Context context) throws

问题 My cdh5.2 cluster has a problem to run hbase MR jobs. For example, I added the hbase classpath into the hadoop classpath: vi /etc/hadoop/conf/hadoop-env.sh add the line: export HADOOP_CLASSPATH="/usr/lib/hbase/bin/hbase classpath:$HADOOP_CLASSPATH" And when I am running: hadoop jar /usr/lib/hbase/hbase-server-0.98.6-cdh5.2.1.jar rowcounter "mytable" I get the following exception: 14/12/09 03:44:02 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java