How do I make Hadoop find imported Python modules when using Python UDFs in Pig?

Question

I am using Pig (0.9.1) with UDFs written in Python. The Python scripts import modules from the standard Python library. I have been able to run the Pig scrips that call the

Answer 1

I encountered the same issue using Hadoop 1.2.1 and Pig 0.11.1 and found a workaround from PIG-2433, which was to add -Dmapred.child.env="JYTHONPATH=job.jar/Lib" to my Pig arguments. Example:

pig -Dmapred.child.env="JYTHONPATH=job.jar/Lib" script.pig