How to use a predicate while reading from JDBC connection?

生来就可爱ヽ(ⅴ<●) 提交于 2019-11-30 07:55:45

You can replace dbtable with query:

db_tbl <- sc %>%
  spark_read_jdbc(sc      = .,
              name    = "table_name",  
              options = list(url      = "jdbc:mysql://localhost:3306/schema_name",
                             user     = "root",
                             password = "password",
                             dbtable  = "(SELECT * FROM table_name WHERE field > 1) as my_query"))

but with simple condition like this Spark should push it automatically when you filter:

db_tbl %>% filter(field > 1)

Just make sure to set:

memory = FALSE

in spark_read_jdbc.

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!