Spark: “Truncated the string representation of a plan since it was too large.” Warning when using manually created aggregation expression

Question

I am trying to build for each of my users a vector containing the average number of records per hour of day. Hence the vector has to have 24 dimensions.

My original

Answer 1

This config, along many others, has been moved to: SQLConf - sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

This can be set either in the config file or via command line in spark, using:

spark.conf.set("spark.sql.debug.maxToStringFields", 1000)