Handling loops in oozie workflow

Question

I have an oozie use case for checking input data availability and trigger mapreduce job based on availability of data. So I wrote a shell script for checking input data and created an ssh action for it in oozie,

The number of retries and and retry intervals of Input data checking should be configurable and after each retry if the data is still missing I got to send an alert, after specified number of retries mapreduce job can start with the available data

I wrote a workflow as follows :

<start to="datacheck" />

<action name="datacheck">
    <ssh xmlns="uri:oozie:ssh-action:0.1">
        <host>${sshUserHost}</host>
        <command>${Oozieutilsscript}</command>
    </ssh>
    <ok to="datacheckswitch" />
    <error to="fail" />
</action>

<decision name="datacheckswitch">
    <switch>
        <case to="mapreduce">${(wf:actionData('datacheck')['datatransfer'] == "complete" )}</case>
        <case to="retry">${(wf:actionData('datacheck')['datatransfer'] == "incomplete" )}</case>        
        <default to="fail" />    
    </switch>
</decision>

<action name="retry">
    <ssh xmlns="uri:oozie:ssh-action:0.1">
        <host>${sshUserHost}</host>
        <command>${Oozieutilsscript1}</command>
    </ssh>
    <ok to="retryswitch" />
    <error to="fail" />
</action>

<decision name="retryswitch">
    <switch>
        <case to="datacheck">${(wf:actionData('datacheck')['retry'] == "notfinished" )}</case>
        <case to="datacheck">${(wf:actionData('datacheck')['retry'] == "finished" )}</case>     
        <default to="fail" />    
    </switch>
</decision>

<action name="mapreduce">
...............
</action>


<!--Kill and End portion-->
<kill name="fail">
    <message>Java failed, error message[${wf:errorMessage(wf:lastErrorNode())}</message>
</kill>
<end name="end" />

Only when I executed the workflow I came to know that oozie doesn't support cycles since its workflow is DAG. got the error Error: E0707 : E0707: Loop detected at parsing, node [datacheck] while parsing workflow.xml

Is there any different approach for handling this usecase ?

Any help is appreciated .

Answer 1

You can simulate loops using recursion. The key idea is that a workflow calls itself using a sub-workflow action that points to the workflow file that contains the action node.

The recursion must be stopped using a decision node.

On my blog you can find a complete example for this.

Answer 2

DAG = Directed Acyclic Graph. This means that there cannot be any cycles (loops) in the graph formed by your workflow.

http://en.wikipedia.org/wiki/Directed_acyclic_graph

Answer 3

You can implemented loops by using sub-workflow concept. The idea is to extract out the action(s) to be repeated into a subflow and call it from the main flow as many times as you want. Read more here: How-to: Shorten Your Oozie Workflow Definitions