pass username and password to sqoop meta connect from oozie

Question

  ${jobTracker}
    ${nameNode}
      &l         


        

Answer 1

If you ever sort out the confusion between Sqoop1 and Sqoop2, then you will have to jump over the Oozie hurdle too.

My advice: stop toying with command-line arguments and use standard Hadoop config files.

1. On your Gateway node (for unit tests), edit /etc/sqoop/conf/sqoop-site.xml to set Client properties

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <!-- some stuff here that does not matter for Metastore -->
  ...

  <!-- now the Metastore config -->
  <property>
    <name>sqoop.metastore.client.enable.autoconnect</name>
    <value>true</value>
  </property>
  <property>
    <name>sqoop.metastore.client.autoconnect.url</name>
    <value>jdbc:hsqldb:hsql://FQDN:16000/sqoop</value>
  </property>
  <property>
    <name>sqoop.metastore.client.autoconnect.username</name>
    <value>sa</value>
  </property>
  <property>
    <name>sqoop.metastore.client.autoconnect.password</name>
    <value></value>
  </property>
  <property>
    <name>sqoop.metastore.client.record.password</name>
    <value>false</value>
  </property>
</configuration>

1b. Upload that file to HDFS somewhere (for use with Oozie jobs)

2. On the node that will actually run the global Metastore DB, also edit the file, and also add two extra Server properties (in this example the DB files are stored in /var/lib/...)

  <property>
     <name>sqoop.metastore.server.port</name> 
     <value>16000</value> 
  </property>
  <property>
     <name>sqoop.metastore.server.location</name> 
     <value>/var/lib/sqoop/data/shared.db</value> 
  </property>

2b. Make sure you CHECKPOINT that database from time to time (to reset the "script" file and flush the "redo log" file) then backup the "script" file as a snapshot of the current DB state, somewhere safe, in case you lose the node and its disk -- yes, these things happen

3. In your Oozie Sqoop actions, set the Client properties with a <job-xml> entry targeting the config file in HDFS.

If you are interested in the actual Sqoop source code that handles these props and the Metastore client, look there

Answer 2

--meta-connect should have value in jdbc:hsqldb:hsql://<server-name>:<port>/sqoop format(default is hsqldb). You can change it to mysql in your case.

You can override below properties in sqoop-site.xml to pass the metastore username and password:

 <property>
    <name>sqoop.metastore.client.autoconnect.username</name>
    <value>SA</value>
    <description>The username to bind to the metastore.
    </description>
  </property>
  <property>
    <name>sqoop.metastore.client.autoconnect.password</name>
    <value></value>
    <description>The password to bind to the metastore.
    </description>
  </property>

--meta-connect will read the username and password from here and will connect to the database.

You can refer the default sqoop-site.xml for more details