MapReduce

问题 What is the main difference between bucketing and indexing of a table in Hive? 回答1: The main difference is the goal: Indexing The goal of Hive indexing is to improve the speed of query lookup on certain columns of a table. Without an index, queries with predicates like 'WHERE tab1.col1 = 10' load the entire table or partition and process all the rows. But if an index exists for col1, then only a portion of the file needs to be loaded and processed. Indexes become even more essential when the

问题 I have setup YARN MapReduce mini-cluster with 1 node manager, 4 local and 4 log directories and so on based on hadoop 2.3.0 from CDH 5.1.0. It looks more or less working. What I failed to achieve is syslog logging from containers. I see container log directories, stdout and stderr files but not syslog with MapReduce container logging. Appropriate stderr warns I have no log4j configuration and contains no any other string: log4j:WARN No appenders could be found for logger (org.apache.hadoop

问题 I am trying to submit a remote job in mapreduce, but I get the error [1]. I even have set in hdfs-site.xml in the remote hadoop the content [2], and changed permissions [3], but the problem remains. The client is xeon, and the superuser is xubuntu. How I add a remote user permission to submit in mapreduce? How I set a group for xeon? [1] 2015-04-23 05:57:35,648 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user xeon [2] <property> <name>dfs.web.ugi</name>

问题 I have a list of Java Objects and I need to reduce it applying Aggregate Functions like a select over a DataBase. NOTE: The data were calculated from multiples Databases and services calls. I expect to have thousands of rows and each row always will have the same quantity of "cells" for each execution. This quantity changes between executions. Samples: Supposing I have my data represented in a List of Object[3] ( List<Object[]> ) my data could be: [{"A", "X", 1}, {"A", "Y", 5}, {"B", "X", 1},

问题 How can I set the number of map task with a org.apache.hadoop.mapreduce.Job ? The function does not seem to exist... But it exists for the org.apacache.hadoop.mapred.JobConf... Thanks ! 回答1: AFAIK, setNumMapTasks is not supported any more. It is merely a hint to the framework(even in the old API), and doesn't guarantee that you'll get only the specified number of maps. The map creation is actually governed by the InputFormat you are using in your job. You could tweak the following properties

问题 I'm following along with the DISCO example for counting words from a file: Counting Words as a map/reduce job I have no issues getting this working, however I want to try reading in a specific field from a text file that contains JSON strings. The file has lines like: {"favorited": false, "in_reply_to_user_id": 306846931, "contributors": null, "truncated": false, "text": "@CataDuarte8 No! av\u00edseme cuando vaya ah salir para yo salir igual!", "created_at": "Wed Apr 04 20:25:37 +0000 2012",

问题 the map reduce examples I see use aggregation functions like count, but what is the best way to get say the top 3 items in each category using map reduce. I'm assuming I can also use the group function but was curious since they state sharded environments cannot use group(). However, I'm actually interested in seeing a group() example as well. 回答1: For the sake of simplification, I'll assume you have documents of the form: {category: <int>, score: <int>} I've created 1000 documents covering

问题 Inside my map function, I am trying to read a file from the distributedcache, load its contents into a hash map. The sys output log of the MapReduce job prints the content of the hashmap. This shows that it has found the file, has loaded into the data structure and performed the needed operation. It iterates through the list and prints its contents. Thus proving that the operation was successful. However, I still get the below error after a few minutes of running the MR job: 13/01/27 18:44:21

问题 I have a valid jar which is running perfectly on another system running the same version of hadoop i.e hadoop-1.2.1 with the same settings. I am able to put the jar file in the hdfs filesystem and create input,output directories. But when I use the command 'hadoop jar HelloWorld.jar classname(main method) input output' it throws 'Invalid jar' error. After searching for the possible solutions for a long time I found out that the command is searching for the jar in local filesystem instead of

问题 I am trying to calculate the no of rows in a Hbase table. Can do that with scannner but it is a bulky process.Want to use RowCounter to fetch the row number from Hbase table.Is there any way by which I can use that in Java Code. Is there any example or code snippet available. Directly using rowcounter is plain simple by using the command :- /hbase org.apache.hadoop.hbase.mapreduce.RowCounter [TABLE_NAME] Please provide any code snippet to use the same in Java code. Thanks 回答1: You can find