mrjob

问题 I am using yelps MRJob library for achieving map-reduce functionality. I know that map reduce has an internal sort and shuffle algorithm which sorts the values on the basis of their keys. So if I have the following results after map phase (1, 24) (4, 25) (3, 26) I know the sort and shuffle phase will produce following output (1, 24) (3, 26) (4, 25) Which is as expected But if I have two similar keys and different values why does the sort and shuffle phase sorts the data on the basis of first

问题 I am running an MRJob on Hadoop cluster & I am getting the following error: No configs found; falling back on auto-configuration Looking for hadoop binary in $PATH... Found hadoop binary: /usr/local/hadoop/bin/hadoop Using Hadoop version 2.7.3 Looking for Hadoop streaming jar in /usr/local/hadoop... Found Hadoop streaming jar: /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.3.jar Creating temp directory /tmp/Mr_Jobs.hduser.20170227.030012.446820 Copying local files to hdfs://

问题 Using the mrjob to run python code on Amazon's Elastic MapReduce I have successfully found a way to upgrade the EMR image's numpy and scipy. Running from console the following commands work: tar -cvf py_bundle.tar mymain.py Utils.py numpy-1.6.1.tar.gz scipy-0.9.0.tar.gz gzip py_bundle.tar python my_mapper.py -r emr --python-archive py_bundle.tar.gz --bootstrap-python-package numpy-1.6.1.tar.gz --bootstrap-python-package scipy-0.9.0.tar.gz > output.txt This successfully bootstraps the latest

问题 I'm trying to learn to use Yelp's Python API for MapReduce, MRJob. Their simple word counter example makes sense, but I'm curious how one would handle an application involving multiple inputs. For instance, rather than simply counting the words in a document, multiplying a vector by a matrix. I came up with this solution, which functions, but feels silly: class MatrixVectMultiplyTast(MRJob): def multiply(self,key,line): line = map(float,line.split(" ")) v,col = line[-1],line[:-1] for i in

问题 Hey I'm fairly new to the world of Big Data. I came across this tutorial on http://musicmachinery.com/2011/09/04/how-to-process-a-million-songs-in-20-minutes/ It describes in detail of how to run MapReduce job using mrjob both locally and on Elastic Map Reduce. Well I'm trying to run this on my own Hadoop cluser. I ran the job using the following command. python density.py tiny.dat -r hadoop --hadoop-bin /usr/bin/hadoop > outputmusic And this is what I get: HADOOP: Running job: job

问题 I'm using the MRJob module for python 2.7. I have created a class that inherits from MRJob, and have correctly mapped everything using the inherited mapper function. Problem is, I would like to have the reducer function output a .csv file...here is the code for the reducer: def reducer(self, geo_key, info_list): info_list.insert(0, ['Name,Age,Gender,Height']) for set in info_list: yield set Then i run in the command line---> python -m map_csv <inputfile.txt> outputfile.csv I keep getting this

问题 It seems like the nature of the MapReduce framework is to work with many files. So when I get errors that tell me I'm using too many files, I suspect I'm doing something wrong. If I run the job with the inline runner and three directories, it works: $ python mr_gps_quality.py /Volumes/Logs/gps/ByCityLogs/city1/0[1-3]/*.log -r inline --no-output --output-dir city1_results/gps_quality/2015/03/ But if I run it using the local runner (and the same three directories), it fails: $ python mr_gps

问题 I would like to populate a database of Postgresql by using a mapper with MrJob and Hadoop 2.7.1. I currently using the following code: # -*- coding: utf-8 -*- #Script for storing the sparse data into a database by using Hadoop import psycopg2 import re from mrjob.job import MRJob args_d = False args_c = True args_s = True args_n = 'es_word_space' def unicodize(segment): if re.match(r'\\u[0-9a-f]{4}', segment): return segment.decode('unicode-escape') return segment.decode('utf-8') def create

问题 I'm running a fairly big MRJob job (1,755,638 keys) and the keys are being written to the reducers in sorted order. This happens even if I specify that Hadoop should use the hash partitioner, with: class SubClass(MRJob): PARTITIONER = "org.apache.hadoop.mapred.lib.HashPartitioner" ... I don't understand why the keys are sorted, when I am not asking for them to be sorted. 回答1: The HashPartitioner is used by default when you don't specify any partitioner explicitly. 回答2: Keys are not sorted by

问题 Simple map combine reduce program: Map column-1 with value column-3 and append '+' in each mapper output of same key and append '-' after reduce output of same key. input_1 and input_2 both files contain a 1 2 3 a 4 5 6 Code is from mrjob.job import MRJob import re import sys class MRWordFreqCount(MRJob): def mapper(self, _, line): line=re.sub("\s\s+"," ",line) s1=line.split() yield(s1[0],s1[2]) def combiner(self, accid, eventid): s="+" yield (accid, s.join(eventid)) def reducer(self, accid,