问题
This mail is to discuss on a use case, on which my team is working. It's to export metadata and data from a HIVE server to RDBMS.
On doing that, export to MySQL and ORACLE is working good, but export to
Netezza is failing with error message:
17/02/09 16:03:07 INFO DAGScheduler: Job 1 finished: json at RdbmsSandboxExecution.java:80, took 0.433405 s
17/02/09 16:03:07 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 143 ms on localhost (1/1)
17/02/09 16:03:07 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
Exception in thread "main" java.sql.SQLException: No suitable driver
at java.sql.DriverManager.getDriver(DriverManager.java:278)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$2.apply(JdbcUtils.scala:50)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$2.apply(JdbcUtils.scala:50)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createConnectionFactory(JdbcUtils.scala:49)
at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:278)
at org.apache.spark.sql.DataFrame.createJDBCTable(DataFrame.scala:1767)
at com.zaloni.mica.datatype.conversion.RdbmsSandboxExecution.main(RdbmsSandboxExecution.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/02/09 16:03:07 INFO SparkContext: Invoking stop() from shutdown hook
17/02/09 16:03:07 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static/sql,null}
17/02/09 16:03:07 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL1/execution/json,null}
We are using DataFrame.createJDBCTable.
The spark-submit command which we are using is :
spark-submit --class <java_class_with_export_logic> --master local --deploy-mode client --conf spark.driver.extraClassPath=/absolute-path/nzjdbc3.jar --jars /absolute-path/nzjdbc3.jar /absolute-path/<application-jar <JDBC_URL>
来源：https://stackoverflow.com/questions/42135791/feasibility-of-hive-to-netezza-data-export-using-spark