mrjob

I am trying to yield the probability each key,value pair generated from mapper has. So, lets say mapper yields: a, (r, 5) a, (e, 6) a, (w, 7) I need to add 5+6+7 = 18 and then find probabilities 5/18, 6/18, 7/18 so the final output from the reducer would look like: a, [[r, 5, 0.278], [e, 6, 0.33], [w, 7, 0.389]] so far, I can only get the reducer to sum all integers from the value. How can I make it to go back and divide each instance by the total sum? thanks! ask417 Pai's solution is technically correct, but in practice this will give you a lot of strife, as setting the partitioning can be a

I'm trying to understand the example for mrjob better from mrjob.job import MRJob class MRWordFrequencyCount(MRJob): def mapper(self, _, line): yield "chars", len(line) yield "words", len(line.split()) yield "lines", 1 def reducer(self, key, values): yield key, sum(values) if __name__ == '__main__': MRWordFrequencyCount.run() I run it by $ python word_count.py my_file.txt and it works as expected but I don't get how it automatically knows that it's going to read a text file and split it by each line. and I'm not sure what the _ does either. From what I understand, the mapper() generates the

This is an MRJob implementation of a simple Map-Reduce sorting functionality. In beta.py : from mrjob.job import MRJob class Beta(MRJob): def mapper(self, _, line): """ """ l = line.split(' ') yield l[1], l[0] def reducer(self, key, val): yield key, [v for v in val][0] if __name__ == '__main__': Beta.run() I run it using the text: 1 1 2 4 3 8 4 2 4 7 5 5 6 10 7 11 One can run this using: cat <filename> | python beta.py Now the issue is the output is sorted assuming that the key is of type string (which is probably the case here). The output is: "1" "1" "10" "6" "11" "7" "2" "4" "4" "2" "5" "5"

I am working on a map-reduce job, consisting multiple steps. Using mrjob every step receives previous step output. The problem is I don't want it to. What I want is to extract some information and use it in second step against all input and so on. Is it possible to do this using mrjob? Note : Since I don't want to use emr, this question is not much of help to me. UPDATE : If it would not be possible to do this on a single job, I need to do it in two separate jobs. In this case, is there any way to wrap these two jobs and manage intermediate outpus, etc? You can use Runners You will have to