Pyspark: reshape data without aggregation

Question

I want to reshape my data from 4x3 to 2x2 in pyspark without aggregating. My current output is the following:

co         


        

Answer 1

Using groupby and pivot is the natural way to do this, but if you want to avoid any aggregation you can achieve this with a filter and join

import pyspark.sql.functions as f

df.where("value_HIGH = 1").select("FAULTY", f.col("count").alias("value_HIGH_1"))\
    .join(
        df.where("value_HIGH = 0").select("FAULTY", f.col("count").alias("value_HIGH_1")),
        on="FAULTY"
    )\
    .show()
#+------+------------+------------+
#|FAULTY|value_HIGH_1|value_HIGH_1|
#+------+------------+------------+
#|     0|          12|         140|
#|     1|          21|         141|
#+------+------------+------------+