scala api for delta lake optimize command

谁说我不能喝 提交于 2021-01-06 07:34:42

问题


The databricks docs say that you can change zordering of a delta table by doing:

spark.read.table(connRandom)
     .write.format("delta").saveAsTable(connZorder)

sql(s"OPTIMIZE $connZorder ZORDER BY (src_ip, src_port, dst_ip, dst_port)")

The problem with this is the switching between the scala and SQL api which is gross. What I want to be able to do is:

spark.read.table(connRandom)
     .write.format("delta").saveAsTable(connZorder)
     .optimize.zorderBy("src_ip", "src_port", "dst_ip", "dst_port")

but I cant find any documentation that says that this is possible.

Is there a scala api for delta lake optimization commands? If so, how do I replicate the aforementioned logic in scala?

来源:https://stackoverflow.com/questions/65098666/scala-api-for-delta-lake-optimize-command

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!