multiple conditions for filter in spark data frames

Question

I have a data frame with four fields. one of the field name is Status and i am trying to use a OR condition in .filter for a dataframe . I tried below queries but no luck.

Answer 1

If we want partial match just like contains, we can chain the contain call like this :

def getSelectedTablesRows2(allTablesInfoDF: DataFrame, tableNames: Seq[String]): DataFrame = {

    val tableFilters = tableNames.map(_.toLowerCase()).map(name => lower(col("table_name")).contains(name))
    val finalFilter = tableFilters.fold(lit(false))((accu, newTableFilter) => accu or newTableFilter)
    allTablesInfoDF.where(finalFilter)

  }