How to filter one spark dataframe against another dataframe

Question

I\'m trying to filter one dataframe against another:

scala> val df1 = sc.parallelize((1 to 100).map(a=>(s\"user $a\", a*0.123, a))).toDF(\"name\", \"sc         


        

Answer 1

You want a (regular) inner join, not an outer join :)

df1.join(df2, df1("user_id") === df2("valid_id"))