Use “IS IN” between 2 Spark dataframe columns

Question

I have the above dataframe:

from pyspark.sql.types import *

rdd = sc.parallelize([
        ('ALT', ['chien', 'chat'] , 'oiseau'),
        ('ALT', ['oiseau']        , 'oiseau'),
        ('TDR', ['poule','poulet'], 'poule' ),
        ('ALT', ['ours']          , 'chien' ),
        ('ALT', ['paon']          , 'tigre' ),
        ('TDR', ['tigre','lion']  , 'lion'  ),
        ('ALT', ['chat']          ,'chien'  ),
])
schema = StructType([StructField("ClientId",StringType(),True),
                     StructField("Animaux",ArrayType(StringType(),True),True),
                     StructField("Animal",StringType(),True),])
test = rdd.toDF(schema)
test.show()

+--------+---------------+------+
|ClientId|        Animaux|Animal|
+--------+---------------+------+
|     ALT|  [chien, chat]|oiseau|
|     ALT|       [oiseau]|oiseau|
|     TDR|[poule, poulet]| poule|
|     ALT|         [ours]| chien|
|     ALT|         [paon]| tigre|
|     TDR|  [tigre, lion]|  lion|
|     ALT|         [chat]| chien|
+--------+---------------+------+

I want to create a new column that will take the value "1" if the string in the column "Animal" is in the list in the column "Animaux" and "0" otherwise.

I tried:

test2=test.withColumn("isinlist", F.when("Animal in Animaux", 'ok').otherwise('pas ok'))
test2=test.withColumn("isinlist", F.when(test.Animal.isin(*test.Animaux), 'ok').otherwise('pas ok'))
test.where("Animal in (Animaux)").show()
test.where("Animal in Animaux").show()
test2=test.withColumn("isinlist", F.when(test.Animal.isin(test.Animaux), 'ok').otherwise('pas ok'))

But none of it works... Does anyone know how to do it without using udf... Is there a direct way to do it?

Answer 1

You can use array_contains:

from pyspark.sql.functions import expr

test.withColumn("isinlist", expr("array_contains(Animaux, Animal)")).show()
# +--------+---------------+------+--------+
# |ClientId|        Animaux|Animal|isinlist|
# +--------+---------------+------+--------+
# |     ALT|  [chien, chat]|oiseau|   false|
# |     ALT|       [oiseau]|oiseau|    true|
# |     TDR|[poule, poulet]| poule|    true|
# |     ALT|         [ours]| chien|   false|
# |     ALT|         [paon]| tigre|   false|
# |     TDR|  [tigre, lion]|  lion|    true|
# |     ALT|         [chat]| chien|   false|
# +--------+---------------+------+--------+

Source How to filter Spark dataframe if one column is a member of another column by zero323 (Scala).