Pivot row to column level

问题

I have a spark dataframe t which is the result of a spark.sql("...") query. Here is the first few rows from t:

| yyyy_mm_dd | x_id | x_name      | b_app   | status        | has_policy | count |
|------------|------|-------------|---------|---------------|------------|-------|
| 2020-08-18 | 1    | first_name  | content | no_contact    | 1          | 23    |
| 2020-08-18 | 1    | first_name  | content | no_contact    | 0          | 346   |
| 2020-08-18 | 2    | second_name | content | implemented   | 1          | 64    |
| 2020-08-18 | 2    | second_name | content | implemented   | 0          | 5775  |
| 2020-08-18 | 3    | third_name  | content | implemented   | 1          | 54    |
| 2020-08-18 | 3    | third_name  | content | implemented   | 0          | 368   |
| 2020-08-18 | 4    | fourth_name | content | first_contact | 1          | 88    |
| 2020-08-18 | 4    | fourth_name | content | first_contact | 0          | 659   |

There is two rows per x_id and this is due to grouping on has_policy. I would like to pivot has_policy and count to columns so I can have one row per x_id instead. This is how the output would look:

| yyyy_mm_dd | x_id | x_name      | b_app   | status        | has_policy_count | has_no_policy_count |
|------------|------|-------------|---------|---------------|------------------|---------------------|
| 2020-08-18 | 1    | first_name  | content | no_contact    | 23               | 346                 |
| 2020-08-18 | 2    | second_name | content | implemented   | 64               | 5775                |
| 2020-08-18 | 3    | third_name  | content | implemented   | 54               | 368                 |
| 2020-08-18 | 4    | fourth_name | content | first_contact | 88               | 659                 |

I'm not sure if it would be easier to achieve this by converting to Pandas first or if we can operate on the Spark df as it is to get the same result?

---

Data types:

t.dtypes
[('yyyy_mm_dd', 'date'),
 ('xml_id', 'int'),
 ('xml_name', 'string'),
 ('b_app', 'string'),
 ('status', 'string'),
 ('has_policy', 'bigint'),
 ('count', 'bigint')]

回答1:

Assuming df is your dataframe. pivot is quite straight forward to use when you read the doc.

df.groupBy(
    "yyyy_mm_dd", "x_id", "x_name", "b_app", "status"
).pivot("has_policy", [0, 1]).sum("count")

来源：https://stackoverflow.com/questions/63684957/pivot-row-to-column-level