MapReduce

问题 I have worked on Hadoop MR for quite some time and I have created and used custom(extension) Writable classes including MapWritable . Now I am required to translate the same MR that I have written in Java to Python. I do not have experience in python and am now exploring the various libraries for the same. I am looking into some options like Pydoop and Mrjob . However, I want to know if these libraries contain the option to create similar custom Writable classes and how to create them. If not

问题 I am currently dealing with a set of in-game-events for various matches. In the game it is possible to kill enemies and purchase items in a shop. What I have been trying to do right now, is to count the number of kills that have occured in a single match up until every purchasing event. { "_id" : ObjectId("5988f89ae5873exxxxxxx"), "gameId" : NumberLong(2910126xxx) "participantId" : 3, "type" : "ITEM_PURCHASED", "timestamp" : 656664 }, { "_id" : ObjectId("5988f89ae5873exxxxxxx"), "gameId" :

问题 I got a web widget with 15,000,000 hits/months and I log every session. When I want to generate a report I'd like to know how many unique IP there are. In normal SQL that would be easy as I'd just do a: SELECT COUNT(*) FROM (SELECT DISTINCT IP FROM SESSIONS) But as that's not possible with the app engine, I'm now looking into solutions on how to do it. It doesn't need to be fast. A solution I was thinking of was to have an empty Unique-IP table, then have a MapReduce job to go through all

问题 Please clarify I have set of input files (say 10) with specific names. I run word count job on all files at once (input path is folder). I am expecting 10 output files with same names as input files. I.e. File1 input should be counted and should be stored in a separate output file with "file1" name. And so on to all files. 回答1: There are 2 approaches you can take to achieve multiple outputs Use MultipleOutputs class - refer this document for information about multipleclassoutput (https:/

问题 I just want to clarify this quote "Code moves near data for computation", does this mean all java MR written by developer deployed to all servers in cluster ? If 1 is true, if someone changes a MR program, how its distributed to all the servers ? Thanks 回答1: Hadoop put MR job's jar to the HDFS - its distributed file system. The task trackers which needed it will take it from there. So it distributed to some nodes and then loaded on-demand by nodes which actually needs them. Usually this needs

问题 I am new to Hadoop and Java, and I feel there is something obvious I am just missing. I am using Hadoop 1.0.3 if that means anything. My goal for using hadoop is to take a bunch of files and parse them one file at a time (as opposed to line by line). Each file will produce multiple key-values, but context to the other lines is important. The key and value are multi-value/composite, so I have implemented WritableCompare for the key and Writable for the value. Because the processing of each

问题 As we know, new need to pack all needed class into the job-jar and upload it to server. it's so slow, i will to know whether there is a way which to specify the thirdpart jar include executing map-red job, so that i could only pack my classes with out dependencies. PS(i found there is a "-libjar" command, but i doesn't figure out how to use it. Here is the link http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/) 回答1: Those are called generic

问题 I am trying to write a job which can analyse some information from youtube data set.I believe i have correctly set the output keys from the map in the driver class,but still i am getting the above error i am posting the code and the exception here, The Mapper public class YouTubeDataMapper extends Mapper<LongWritable,Text,Text,IntWritable>{ private static final IntWritable one = new IntWritable(1); private Text category = new Text(); public void mapper(LongWritable key,Text value,Context

问题 I am completely new to Hadoop and MapReduce and am trying to work my way through it. I am trying to develop a mapreduce application in python, in which I use data from 2 .CSV files. I am just reading the two files in mapper and then printing the key value pair from the files to the sys.stdout The program runs fine when I use it on a single machine, but with the Hadoop Streaming, I get an error. I think I am making some mistake in reading files in the mapper on Hadoop. Please help me out with

问题 I have the code of KMeans and my task is to calculate the speedup, I've done it by running it on different numbers of nodes in my uni's clusters. But is it possible to change the number of mappers and/or reducers, so that I can check the change in speedup while running it on single node. While googling, I found that by using conf.setNumReduceTasks(2); I can change the numbers of reducers. but I havn't see any change in my output. (My output is the time in ms). The code I am using is from