Example Oozie job works from Hue, but not from command line: SparkMain not found

问题

I've successfully run the example Spark workflow ("Copy a file by launching a Spark Java program") provided in the Hue Oozie workflow editor (in the Cloudera 5.5.1 QuickStart VM).

I'm now trying to run it manually using the oozie commandline tool:

oozie job -oozie http://localhost:11000/oozie -config job.properties -run

The workflow XML is basically unchanged - I have copied it to HDFS and have the following job.properties:

nameNode=hdfs://localhost:8020
jobTracker=localhost:8032
oozie.wf.application.path=/user/cloudera/workflows/spark-scala/spark-scala.xml
input=/user/hue/oozie/workspaces/data/sonnets.txt
output=here

The job is accepted and appears in the Hue web dashboard, but is killed after a few seconds, and the logs report:

Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found

What is the problem here?

回答1:

Oozie doesn't include the libraries for the Spark action, by default - you need to add the following to the job.properties:

oozie.use.system.libpath=true

(Clicking on the previously successful Hue workflow in the Hue Dashboard, you can select the Configuration tab to see the properties that Hue has provided)

来源：https://stackoverflow.com/questions/37410831/example-oozie-job-works-from-hue-but-not-from-command-line-sparkmain-not-found