hadoop2

问题 I'm running hadoop 2.6.1 on OS X 10.10.5. I'm getting this warning: WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable I've read that this problem can be caused by running a 32bit native library libhadoop.so.1.0.0 with a 64 bit version of hadoop. I've checked my version of libhadoop.so.1.0.0 and it is 64 bit. $ find ~/hadoop-2.6.1/ -name libhadoop.so.1.0.0 -ls 136889669 1576 -rwxr-xr-x 1 davidlaxer staff 806303 Sep

问题 I have deployed a CDH-5.9 cluster with MR as hive execution engine. I have a hive table named "users" with 50 rows. Whenever I execute the query select * from users works fine as follows : hive> select * from users; OK Adam 1 38 ATK093 CHEF Benjamin 2 24 ATK032 SERVANT Charles 3 45 ATK107 CASHIER Ivy 4 30 ATK384 SERVANT Linda 5 23 ATK132 ASSISTANT . . . Time taken: 0.059 seconds, Fetched: 50 row(s) But issuing select max(age) from users failed after submitting as mr job. The container log

问题 I was under the impression that combiners are just like reducers that act on the local map task, That is it aggregates the results of individual Map task in order to reduce the network bandwidth for output transfer. And from reading Hadoop- The definitive guide 3rd edition , my understanding seems correct. From chapter 2 (page 34) Combiner Functions Many MapReduce jobs are limited by the bandwidth available on the cluster, so it pays to minimize the data transferred between map and reduce

问题 Which of these is used to submit job for execution in job tracker. It would be great if one can explain how both these classes are being used in different use cases. 回答1: Question 1: JobClient Job control is done through the Job class in New API rather than the old class JobClient Job is job submitter's view of the Job. It allows the user to configure the job, submit it, control its execution, and query the state. The set methods only work until the job is submitted, afterwards they will

问题 Multiple Java instances are running on my machine and I want to check whether the Hadoop file is already opened in write ( fs.create(file) or fs.append(file) ) mode in any of the instances. I Tried in FileStatus of the Hadoop file, not found anything. Is there any way to check whether the Hadoop file is already opened for write? One way is to try to create/append a file again and catch the exception, but I have thousands of files and don't want to try every file. Also, if create/append is a

问题 I am trying to create a hive table on azure HDInsight from an avro file exported from raw google analytics data in BigQuery. It seems to work. I can created the table, and there are no errors when I run DESCRIBE. But when I try to select results, even if I select only two non-nested columns, I get a an error: "java.lang.IllegalArgumentException". Here's how I created the table: DROP TABLE IF EXISTS ga_sessions_20150106; CREATE EXTERNAL TABLE IF NOT EXISTS ga_sessions_20150106 ROW FORMAT SERDE

问题 This is a follow up question of Extracting rows containing specific value using mapReduce and hadoop Mapper function public static class MapForWordCount extends Mapper<Object, Text, Text, IntWritable>{ private IntWritable saleValue = new IntWritable(); private Text rangeValue = new Text(); public void map(Object key, Text value, Context con) throws IOException, InterruptedException { String line = value.toString(); String[] words = line.split(","); for(String word: words ) { if(words[3]

问题 I am able to rename my reducer output file correctly but r-00000 is still persisting . I have used MultipleOutputs in my reducer class . Here is details of the that .Not sure what am i missing or what extra i have to do? public class MyReducer extends Reducer<NullWritable, Text, NullWritable, Text> { private Logger logger = Logger.getLogger(MyReducer.class); private MultipleOutputs<NullWritable, Text> multipleOutputs; String strName = ""; public void setup(Context context) { logger.info(

问题 I would like to create a file during my program. However, I don't want this file to be written on HDFS but on the datanode filesystem where the map operation is executed. I tried the following approach : public void map(Object key, Text value, Context context) throws IOException, InterruptedException { // do some hadoop stuff, like counting words String path = "newFile.txt"; try { File f = new File(path); f.createNewFile(); } catch (IOException e) { System.out.println("Message easy to look up

问题 I'm trying to write unit tests using miniDFSCluster and it's throwing the error below java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z any pointers to resolve this issue? 回答1: With errors like this, I use three steps Find out what it is looking for In this case, *org.apache.hadoop.io.nativeio.NativeIO$Windows.access0* Find out what jar/lib it is in. I don't use the Windows version, but I believe it is in hadoop.dll - you'll have to