Trying to find R equivalent for SetConf from Java

问题

In Java, you can do something like:

sc.setConf('spark.sql.parquet.binaryAsString','true')

What would the equivalent be in R? I've looked at the methods available to the sc object, and can't find any obvious way of doing this

Thanks

回答1:

You can set environment variables during SparkContext initialization. sparkR.init has a number of optional arguments including:

• sparkEnvir - a list of environment variables to set on worker nodes.
• sparkExecutorEnv - a list of environment variables to be used when launching executors

In your case something like this should do the trick:

sparkEnvir <- list('spark.sql.parquet.binaryAsString'='true')
sc <- sparkR.init(master, app_name, sparkEnvir=sparkEnvir)

回答2:

I found the solution to the problem.

We can do the following:

sql(sqlContext,'SET spark.sql.parquet.binaryAsString=true')

This fixes everything.

来源：https://stackoverflow.com/questions/33085937/trying-to-find-r-equivalent-for-setconf-from-java