How to use NOT IN clause in filter condition in spark

Question

I want to filter a column of an RDD source :

val source = sql(\"SELECT * from sample.source\").rdd.map(_.mkString(\",\"))
val destination = sql(\"select * from          


        

Answer 1

You can try something similar in Java,

ds = ds.filter(functions.not(functions.col(COLUMN_NAME).isin(exclusionSet)));

where exclusionSet is a set of objects that needs to be removed from your dataset.