What does “Correlated scalar subqueries must be Aggregated” mean?

Question

I use Spark 2.0.

I\'d like to execute the following SQL query:

val sqlText = \"\"\"
select
  f.ID as TID,
  f.BldgID as TBldgID,
  f.LeaseID as TLeas         


        

Answer 1

You have to make sure that your sub-query by definition (and not by data) only returns a single row. Otherwise Spark Analyzer complains while parsing the SQL statement.

So when catalyst can't make 100% sure just by looking at the SQL statement (without looking at your data) that the sub-query only returns a single row, this exception is thrown.

If you are sure that your subquery only gives a single row you can use one of the following aggregation standard functions, so Spark Analyzer is happy:

• first
• avg
• max
• min