Cross Join for calculation in Spark SQL

Question

I have a temporary view with only 1 record/value and I want to use that value to calculate the age of the customers present in another big table (with 100

Answer 1

Hard to work out exactly your point, but if you cannot use Scala or pyspark and dataframes with .cache etc. then I think that instead of of using a temporary view, just create a single row table. My impression is you are using Spark %sql in a notebook on, say, Databricks.

This is my suspicion as it were.

That said a broadcastjoin hint may well mean the optimizer only sends out 1 row. See https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-sql-hint-framework.html#specifying-query-hints