Override hadoop's mapreduce.fileoutputcommitter.marksuccessfuljobs in oozie

Question

mapreduce.fileoutputcommitter.marksuccessfuljobs
false

I wan

Answer 1

I ran into the same issue and ended up using a shell action to create the flag.

Here's a full example: http://nathan.vertile.com/blog/2014/09/02/oozie-data-pipeline-done-flag/

Answer 2

Hive unfortunately overrides this capability by setting it's own NullOutputComitter:

conf.setOutputCommitter(NullOutputCommitter.class);

see

src/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
src/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java

Looks like you'll have to create the flag manually. We have filed HIVE-3700 for this.

Answer 3

You can add 'dfs' command to your hive script, like

dfs -touchz '$table_base_path'/dt='${partition}'/_SUCCESS

https://archive.cloudera.com/cdh4/cdh/4/hive/language_manual/cli.html