Pyspark F.when X not in Y

Question

Hi everybody i have a question. I have to put a negative condition but I’m not sure about how to do it. Df.withColumn(“X”, F.when(F.col(ABC).isn([‘A’]),F.lit([‘B’]).otherwise(F.lit(“”)) This is if I want ABC in A but how to write not in A? I tried == False but the software tells me it’s not correct. Thx in advance

Answer 1

Use ~ option in when clause.

df.withColumn(“X”, F.when(~F.col(ABC).isin(['A']),F.lit(['B']).otherwise(F.lit(''))

Example:

df.show()
#+---+---+---+
#|  a|  b|  c|
#+---+---+---+
#|  a|123|  b|
#|  c|123|  d|
#+---+---+---+

#in 
df.filter(col("a").isin(['a'])).show()
#or
df.filter((col("a").isin(['a'])) == True).show()

#+---+---+---+
#|  a|  b|  c|
#+---+---+---+
#|  a|123|  b|
#+---+---+---+

#not in 
df.filter(~col("a").isin(['a'])).show()
#or
df.filter((col("a").isin(['a'])) == False).show()

#+---+---+---+
#|  a|  b|  c|
#+---+---+---+
#|  c|123|  d|
#+---+---+---+