Why is join not possible after show operator?

Question

The following code works fine until I add show after agg. Why is show not possible?

 val tempTableB = tableB.groupBy(\         


        

Answer 1

.show() is a function with, what we call in Scala, a side-effect. It prints to stdout and returns Unit(), just like println

Example:

val a  = Array(1,2,3).foreach(println)
a: Unit = ()

In scala, you can assume that everything is a function and will return something. In your case, Unit() is being returned and that's what's getting stored in tempTableB.

Answer 2

As @philantrovert has already answered with much detailed explanation. So I shall not explain.

What you can do if you want to see whats in tempTableB then you can do so after it has been assigned as below.

 val tempTableB = tableB.groupBy("idB")
  .agg(first("numB").as("numB")) 

 tempTableB.show

 tableA.join(tempTableB, $"idA" === $"idB", "inner")
 .drop("idA", "numA").show

It should work then