SparkR Stage X contains a task of very large size

问题

I'm getting this warning when invoking createOrReplaceTempView with a R data frame:

createOrReplaceTempView (as.Data.Frame(products), "prod")

Do I should ignore this warning? This is inefficient?

Thanks!

回答1:

Those are just warnings. If you want to try and avoid them, repartition you data and call an action on it before registering a temp table and executing some function on the data. The repartition will cause a shuffle.

For example,

set.seed(123)
df<- data.frame(thing1=rnorm(100000), thing2=rep("ThisIsAString", 100000), stringsAsFactors = FALSE)
sdf<- SparkR::createDataFrame(df) # Warnings for me
SparkR::getNumPartitions(sdf) # 1 partition
sdf<- SparkR::repartition(sdf, numPartitions=4L) # repartition, will cause a shuffle
SparkR::getNumPartitions(sdf) # spark now knows to repartition the data, this will happen once an action is called on the data, i.e. counting the rows
SparkR::cache(sdf) # Nothing has happened yet
SparkR::nrow(sdf) # Now cause the repartition and a count to happen. # Will be warned
SparkR::createOrReplaceTempView(sdf, "sdfTable") # Make a temp table as you have in your example

res<- SparkR::sql("SELECT thing1, thing2 FROM sdfTable WHERE thing1> 0.5") # SQL
SparkR::nrow(res) # no warnings, 31002 observations found. 
SparkR::getNumPartitions(res) # 4 partitions in the result

来源：https://stackoverflow.com/questions/49089935/sparkr-stage-x-contains-a-task-of-very-large-size