How to refer to a Spark DataFrame by name in sparklyr and assign it to a variable?

问题

Say I ran the following code and I forgot to assign the Spark dataframe iris to a variable in R and I can't use .Last.value to assign because I had run some other code right after copying the data to Spark.

library(sparklyr)
library(dplyr)
sc <- spark_connect(master = "local")
copy_to(sc, iris)
2+2 # ran some other code so can't use .Last.value

How do I assing the Spark dataframe "iris" to a variable in R called iris_tbl?

回答1:

copy_to provides additional name argument By default it is set to:

deparse(substitute(df))

so in your case the name will be iris. If you want more predictable behavior you should set the name manually:

copy_to(sc, iris, name = "foo")

Then you can access it dplyr way with tbl:

dplyr::tbl(sc, "foo")

or via Spark session:

sc %>% spark_session() %>% invoke("table", "foo") %>% sdf_register()

All production ready reader methods (copy_to shouldn't be used as anything else than a testing and development tool) require name, so you can reference tables the same way

spark_read_csv(sc, "bar", path)
tbl(sc, "bar")

来源：https://stackoverflow.com/questions/51867877/how-to-refer-to-a-spark-dataframe-by-name-in-sparklyr-and-assign-it-to-a-variabl