hadoop streaming - how to inner join of two diff files using python

问题

I want to find out top website page visits based on user age group between 18 and 25. I have two files, one contains username, age and other file contains username, website name. Examples:

users.txt

John, 22

pages.txt

John, google.com

I have written the following in python, and it works as i expected in outside of hadoop.

import os
os.chdir("/home/pythonlab")

#Top sites visited by users aged 18 to 25

#read the users file
lines = open("users.txt")
users = [ line.split(",") for line in lines]      #user name, age (eg - john, 22)
userlist = [ (u[0],int(u[1])) for u in users]     #split the user name and age

#read the page visit file
pages = open("pages.txt")
page = [p.split(",") for p in pages]              #user name, website visited (eg - john,google.com)
pagelist  = [ (p[0],p[1]) for p in page]

#map user and page visits & filter age group between 18 and 25
usrpage = [[p[1],u[0]] for u in userlist for p in pagelist  if (u[0] == p[0] and u[1]>=18 and u[1]<=25) ]

for z in usrpage:
    print(z[0].strip('\r\n')+",1")     #print website name, 1

Sample output:

yahoo.com,1 google.com,1

Now I want to solve this using hadoop streaming.

My question is, how do I process these two named files (users.txt, pages.txt) in my mapper? We normally pass only input directory to hadoop streaming.

回答1:

You would need to look into using Hive. This would allow you to join multiple source files into one, just like you need. it allows you to join two data sources, almost like you do in SQL and then push the result into your mapper and reducer.

来源：https://stackoverflow.com/questions/16909577/hadoop-streaming-how-to-inner-join-of-two-diff-files-using-python