Oozie shell action memory limit

问题

We have an oozie workflow with a shell action that needs more memory than what a map task is given by Yarn by default.

How can we give it more memory?

We have tried adding the following configuration to the action:

<configuration>
  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>6144</value> <!-- for example -->
  </property>
</configuration>

We have both set this as an inline (in the workflow.xml) configuration and as a jobXml. Neither has had any effect.

回答1:

We found the answer:

A shell action is executed as an oozie "launcher" map task, and this task does not use the normal configuration properties.

Instead you have to prefix the properties with "oozie.launcher" to make them apply to the launcher task.

So in our case, if we use the following configuration for our shell action, it works.

<configuration>
  <property>
    <name>oozie.launcher.mapreduce.map.memory.mb</name>
    <value>6144</value> <!-- for example -->
  </property>
</configuration>

This is not obvious from the oozie documentation. We found this here: http://downright-amazed.blogspot.com/2012/02/configure-oozies-launcher-job.html

回答2:

thank your point, just edit workflow.xml file, add :

<workflow-app name="simple-ONE-wf" xmlns="uri:oozie:workflow:0.1">
    <start to='ONE'/>`enter code here
    <action name="ONE">
        <spark xmlns="uri:oozie:spark-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>oozie.launcher.mapreduce.map.memory.mb</name>
                    <value>4096</value>
                </property>
                <property>
                    <name>oozie.launcher.mapreduce.map.java.opts</name>
                    <value>-Xmx3200m</value>
                </property>
                <property>
                    <name>oozie.launcher.mapreduce.map.java.opts</name>
                    <value>-XX:MaxPermSize=1g</value>
                </property>
                ...
            </configuration>
           ...
    </action>

    <kill name='kill'>
        <message>Something went wrong: ${wf:errorCode('firstdemo')}</message>
    </kill>
    <end name='end'/>
</workflow-app>

来源：https://stackoverflow.com/questions/24262896/oozie-shell-action-memory-limit

标签

Oozie

yarn