How to integrate Apache Spark with MySQL for reading database tables as a spark dataframe? [closed]

Answer 1

For Java(using maven), add spark dependencies and sql driver dependencies in your pom.xml file,

    1.8
    1.6.3
    UTF-8

 
    
        mysql
        mysql-connector-java
        6.0.6
    
    
        org.apache.spark
        spark-sql_2.10
        ${spark.version}
    
    
        org.apache.spark
        spark-core_2.10
        ${spark.version}
    

    
        junit
        junit
        4.11
        test
    

Sample code, suppose your mysql locates at local, database name is test, user name is root and password is password, and two tables in test db are table1 and table2

SparkConf sparkConf = new SparkConf();
SparkContext sc = new SparkContext("local", "spark-mysql-test", sparkConf);
SQLContext sqlContext = new SQLContext(sc);

// here you can run sql query
String sql = "(select * from table1 join table2 on table1.id=table2.table1_id) as test_table";
// or use an existed table directly
// String sql = "table1";
DataFrame dataFrame = sqlContext
    .read()
    .format("jdbc")
    .option("url", "jdbc:mysql://127.0.0.1:3306/test?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true")
    .option("user", "root")
    .option("password", "password")
    .option("dbtable", sql)
    .load();

// continue your logical code
......