Egg/JAR equivalent for Sparklyr projects

旧巷老猫 提交于 2020-06-26 14:17:00

问题


We have a SparklyR project which is set up like this

# load functions
source('./a.R')
source('./b.R')
source('./c.R')
....

# main script computations  
sc -> spark_connect(...)
read_csv(sc, s3://path)

....

Running it on EMR

spark-submit --deploy-mode client s3://path/to/my/script.R

Running this script using spark-submit above fails since it seems to only take a single R script but we are sourcing functions from multiple files. Is there a way we can package this as an egg/jar file with all of the files and pass it as an argument to spark-submit?

来源:https://stackoverflow.com/questions/62400076/egg-jar-equivalent-for-sparklyr-projects

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!