问题
I have to cross join 2 dataframe in Spark 2.0 I am encountering below error:
User class threw exception:
org.apache.spark.sql.AnalysisException: Cartesian joins could be prohibitively expensive and are disabled by default. To explicitly enable them, please set spark.sql.crossJoin.enabled = true;
Please help me where to set this configuration, I am coding in eclipse.
回答1:
As the error message clearly states you need to set spark.sql.crossJoin.enabled = true to your spark configuration
You can set the same something like below:
val sparkConf = new SparkConf().setAppName("Test")
sparkConf.set("spark.sql.crossJoin.enabled", "true")
Then get or create SparkSession by passing this SparkConf
val sparkSession = SparkSession.builder().config(sparkConf).getOrCreate()
Then do your join...
来源：https://stackoverflow.com/questions/42198834/how-to-enable-cartesian-join-in-spark-2-0