pass username and password to sqoop meta connect from oozie

问题

<sqoop xmlns="uri:oozie:sqoop-action:0.3">
  <job-tracker>${jobTracker}</job-tracker>
    <name-node>${nameNode}</name-node>
      <arg>job</arg>
      <arg>--meta-connect</arg>
      <arg>jdbc:mysql://FQDN:3306/sqoop</arg>
      <arg>--exec</arg>
      <arg>fabric_inventory</arg>
</sqoop>

Now, to pass username and password for the --meta-connect here,

if I pass it as following in oozie.xml:

<arg>jdbc:mysql://FQDN:3306/sqoop?user=sqoop&amp;password=sqoop</arg>

<arg>jdbc:mysql://FQDN:3306/sqoop?user=sqoop&password=sqoop</arg>

It gives following exception:

Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter 2. it does not take

--username and --password prop and val in arg tag.

How do I pass the values in the correct manner?

This is what the complete exception looks like:

2016-05-11 10:21:44,920 ERROR [main] tool.ImportTool (ImportTool.java:run(613)) - Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
    at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:228)
    at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
    at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197)
    at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
    at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:241)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

回答1:

If you ever sort out the confusion between Sqoop1 and Sqoop2, then you will have to jump over the Oozie hurdle too.

My advice: stop toying with command-line arguments and use standard Hadoop config files.

1. On your Gateway node (for unit tests), edit /etc/sqoop/conf/sqoop-site.xml to set Client properties

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <!-- some stuff here that does not matter for Metastore -->
  ...

  <!-- now the Metastore config -->
  <property>
    <name>sqoop.metastore.client.enable.autoconnect</name>
    <value>true</value>
  </property>
  <property>
    <name>sqoop.metastore.client.autoconnect.url</name>
    <value>jdbc:hsqldb:hsql://FQDN:16000/sqoop</value>
  </property>
  <property>
    <name>sqoop.metastore.client.autoconnect.username</name>
    <value>sa</value>
  </property>
  <property>
    <name>sqoop.metastore.client.autoconnect.password</name>
    <value></value>
  </property>
  <property>
    <name>sqoop.metastore.client.record.password</name>
    <value>false</value>
  </property>
</configuration>

1b. Upload that file to HDFS somewhere (for use with Oozie jobs)

2. On the node that will actually run the global Metastore DB, also edit the file, and also add two extra Server properties (in this example the DB files are stored in /var/lib/...)

  <property>
     <name>sqoop.metastore.server.port</name> 
     <value>16000</value> 
  </property>
  <property>
     <name>sqoop.metastore.server.location</name> 
     <value>/var/lib/sqoop/data/shared.db</value> 
  </property>

2b. Make sure you CHECKPOINT that database from time to time (to reset the "script" file and flush the "redo log" file) then backup the "script" file as a snapshot of the current DB state, somewhere safe, in case you lose the node and its disk -- yes, these things happen

3. In your Oozie Sqoop actions, set the Client properties with a <job-xml> entry targeting the config file in HDFS.

If you are interested in the actual Sqoop source code that handles these props and the Metastore client, look there

回答2:

--meta-connect should have value in jdbc:hsqldb:hsql://<server-name>:<port>/sqoop format(default is hsqldb). You can change it to mysql in your case.

You can override below properties in sqoop-site.xml to pass the metastore username and password:

 <property>
    <name>sqoop.metastore.client.autoconnect.username</name>
    <value>SA</value>
    <description>The username to bind to the metastore.
    </description>
  </property>
  <property>
    <name>sqoop.metastore.client.autoconnect.password</name>
    <value></value>
    <description>The password to bind to the metastore.
    </description>
  </property>

--meta-connect will read the username and password from here and will connect to the database.

You can refer the default sqoop-site.xml for more details

来源：https://stackoverflow.com/questions/37153427/pass-username-and-password-to-sqoop-meta-connect-from-oozie

标签

sqoop

Oozie