How to enable Cartesian join in Spark 2.0? [duplicate]

Question

This question already has an answer here:

• spark.sql.crossJoin.enabled for Spark 2.x 3 answers

I have to cross join 2 dataframe in Spark 2.0 I am encountering below error:

User class threw exception:

org.apache.spark.sql.AnalysisException: Cartesian joins could be prohibitively expensive and are disabled by default. To explicitly enable them, please set spark.sql.crossJoin.enabled = true; 

Please help me where to set this configuration, I am coding in eclipse.

Answer 1

As the error message clearly states you need to set spark.sql.crossJoin.enabled = true to your spark configuration

You can set the same something like below:

val sparkConf = new SparkConf().setAppName("Test")
sparkConf.set("spark.sql.crossJoin.enabled", "true")

Then get or create SparkSession by passing this SparkConf

val sparkSession = SparkSession.builder().config(sparkConf).getOrCreate()

Then do your join...