Oozie

I'm trying to use subprocess in a python script which I call within an oozie shell action. Subprocess is supposed to read a file which is stored in Hadoop's HDFS. I'm using hadoop-1.2.1 in pseudo-distributed mode and oozie-3.3.2. Here is the python script, named connected_subprocess.py : #!/usr/bin/python import subprocess import networkx as nx liste=subprocess.check_output("hadoop fs -cat /user/root/output-data/calcul-proba/final.txt",shell=True).split('\n') G=nx.DiGraph() f=open("/home/rlk/liste_strongly_connected.txt","wb") for item in liste: try: app1,app2=item.split('\t') G.add_edge(app1

Between Apache Oozie, Spotify/Luigi and airbnb/airflow , what are the pros and cons for each of them? I have used oozie and airflow in the past for building a data ingestion pipeline using PIG and Hive. Currently, I am in the process of building a pipeline that looks at logs and extracts out useful events and puts them on redshift. I found that airflow was much easier to use/test/setup. It has a much cooler UI and lets users perform actions from the UI itself, which is not the case with Oozie. Any information about Luigi or other insights regarding stability and issues are welcome. Azkaban:

I am attempting to run a sqoop command with a free-form query, because I need to perform an aggregation. It's being submitted via the Hue interface, as an Oozie workflow. The following is a scaled-down version of the command and query. When the command is processed, the "--query" statement (enclosed in quotes) results in each portion of the query to be interpreted as unrecognized arguments, as shown in the error following the command. In addition, the target directory is being misinterpreted. What is preventing this from running, and what can be done to resolve it? The ${env} and ${shard}

The following is my workflow.xml <workflow-app xmlns="uri:oozie:workflow:0.3" name="import-job"> <start to="createtimelinetable" /> <action name="createtimelinetable"> <sqoop xmlns="uri:oozie:sqoop-action:0.3"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.compress.map.output</name> <value>true</value> </property> </configuration> <command>import --connect jdbc:mysql://10.65.220.75:3306/automation --table ABC --username root</command> </sqoop> <ok to="end"/> <error to="end"/> </action> <end name="end"/> </workflow-app>

I would like to know if it is possible/how to force a coordinator to materialize or instantiate workflow at regular intervals even if previous instantiated workflow are not done yet. Let me explain: I have a simple coordinator looking like this: <coordinator-app name="myApp" frequency="${coord:hours(3)}" start="2015-01-01T0:00Z" end="2016-01-01T00:00Z" timezone="UTC" xmlns="uri:oozie:coordinator:0.4"> <action> <workflow> <app-path>${myPath}/workflow.xml</app-path> </workflow> </action> </coordinator-app> The frequency is set to 3 hours. Every 3 hours, I expect the coordinator to "materialize"

By default, Oozie shared lib directory provides libraries for Hive, Pig, and Map-Reduce. If I want to run Spark job on Oozie, it might be better to add Spark lib jars to Oozie's shared lib instead of copy them to app's lib directory. How can I add Spark lib jars (including spark-core and its dependencies) to Oozie's shared lib? Any comment / answer is appreciated. Spark action is scheduled to be released with Oozie 4.2.0, even though the doc seems to be a bit behind. See related JIRA here : Oozie JIRA - Add spark action executor Cloudera's release CDH 5.4 has it already though, see official