Oozie

问题 I'm trying to create an OOZIE job with a custom inputformat. I am using the new API and have set : <property> <name>mapred.mapper.new-api</name> <value>true</value> </property> <property> <name>mapred.reducer.new-api</name> <value>true</value> </property> The property name I'm triying is : <property> <name>mapreduce.job.inputformat.class</name <value>org.lab41.dendrite.generator.kronecker.mapreduce.lib.input.QuotaInputFormat</value> </property> Is the correct property name? 回答1: you can see

问题 We are migrating from CDH3 to CDH4 and as part of this migration we are moving all the jobs that we have on CDH3. We have noticed one critical issue in this, when a work flow is executed through oozie for executing a python script which internally invoked a hive query(hive -e {query}), here in this hive query we are adding a custom jar using add jar {LOCAL PATH FOR JAR}, and created a temporary function for custom udf. And it looks ok till here. But when the query started executing with

问题 OOZIE wordcount example gives JA009: RPC response exceeds maximum data length. We have doubled the ipc.maximum.data.length and restarted the NameNode. 2018-12-05 17:55:45,914 WARN MapReduceActionExecutor:523 - SERVER[******] USER[******] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000004-181205174411487-oozie-******-W] ACTION[0000004-181205174411487-oozie-******-W@mr-node] No credential properties found for action : 0000004-181205174411487-oozie-******-W@mr-node, cred : null 2018-12-05 18:10:46

问题 I am using a keytab file to bypass kerberos to run a shell script through oozie. Shell script includes hbase shell commands so I nee to do a kinit before running hbase shell commands.This works fine for me. So, in a case where I launch the job using a coordinator and have several shell scripts in the workflow using a keytab(service account) will there be a conflict in oozie ?Since I launch the job using oozie coordinator but have keytab files in workflow of a service account. Thanks, 回答1: The

问题 so I'm trying to run a spark job on yarn-cluster mode (succeeded running it in local mode and yarn-client), but I am running into a problem where oozie launcher fails. Below is the error message from stderr . Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, java.lang.NoSuchMethodError: org.apache.spark.network.util.JavaUtils.byteStringAsBytes(Ljava/lang/String;)J org.apache.oozie.action.hadoop.JavaMainException: java.lang.NoSuchMethodError:

问题 I am currently setting up an Oozie workflow that uses a Spark action. The Spark code that I use works correctly, tested on both local and YARN. However, when running it as an Oozie workflow I am getting the following error: Main class [org.apache.oozie.action.hadoop.SparkMain], exit code [1] Having read up on this error, I saw that the most common cause was a problem with Oozie sharelibs. I have added all Spark jar files to the Oozie /user/oozie/share/lib/spark on hdfs, restarted Oozie and

问题 I have a hadoop cluster with 6 nodes. I'm pulling data out of MSSQL and back into MSSQL via Sqoop. Sqoop import commands work fine, and I can run a sqoop export command from the console (on one of the hadoop nodes). Here's the shell script I run: SQLHOST=sqlservermaster.local SQLDBNAME=db1 HIVEDBNAME=db1 BATCHID= USERNAME="sqlusername" PASSWORD="password" sqoop export --connect 'jdbc:sqlserver://'$SQLHOST';username='$USERNAME';password='$PASSWORD';database='$SQLDBNAME'' --table ExportFromHive

问题 I have an Oozie job that I start from java client which gets stuck at START action and it says that it is RUNNING, but the START node is in PREP state. Why is that and how to fix the problem? The Oozie workflow contains only one java action. The Hadoop version on the cluster is 2.4.0, and Oozie on cluster is 4.0.0. Here is the workflow.xml <workflow-app xmlns='uri:oozie:workflow:0.2' name='java-filecopy-wf'> <start to='java1'/> <action name='java1'> <java> <job-tracker>${jobTracker}</job

问题 I'm a newbie in Oozie and I've read some Oozie shell action examples but this got me confused about certain things. There are examples I've seen where there is no <file> tag. Some example, like in Cloudera here, repeats the shell script in file tag: <shell xmlns="uri:oozie:shell-action:0.2"> <exec>check-hour.sh</exec> <argument>${earthquakeMinThreshold}</argument> <file>check-hour.sh</file> </shell> While in Oozie's website, writes the shell script (the reference ${EXEC} from job.properties,

问题 Oozie is failing with following error when workflow.xml is provided from s3, But the same worked provided workflow.xml from HDFS. Same has worked with earlier versions of oozie, Is there anything changed from 4.3 version of oozie.? Env: HDP 3.1.0 Oozie 4.3.1 oozie.service.HadoopAccessorService.supported.filesystems=* Job.properties nameNode=hdfs://ambari-master-1a.xdata.com:8020 jobTracker=ambari-master-2a.xdata.com:8050 queue=default #OOZIE job details basepath=s3a://mybucket/test/oozie