How to create dataframe from list[Map] based on condition

问题

I have a dataframe called DF1 like below.

DF1:

srcColumnZ|srcCoulmnY|srcCoulmnR| 
+---------+----------+----------+
|John     |Non Hf    |New york  |
|Steav    |Non Hf    |Mumbai    |
|Ram      |HF        |Boston    |

And also having one list of map with source to target column mapping like below.

List(Map(targetColumn -> columnNameX, sourceColumn -> List(srcColumnX, srcColumnY, srcColumnZ, srcColumnP, srcColumnQ, srcColumnR)), Map(targetColumn -> columnNameY, sourceColumn -> List(srcColumnY)), Map(targetColumn -> columnNameZ, selectvalue -> 5))

I wanted to create a dataframe based on the above list of Map and in that data frame I need columnNameX, columnNameY, columnNameZ as a column(according to above list) and the value of these column will be based on sourceColumn i.e. if sourceColumn is present like List(srcColumnX, srcColumnY, srcColumnZ, srcColumnP, srcColumnQ, srcColumnR)) then it will check all the column one by one in DF1 and whenever 1st column will match it will move all the values of that column into target column and same for next target column. And in case selectvalue present instead of source column it will hardcode that value into entire column. ie: in above list for target column(columnNameZ) selectvalue is present 5

Below is the expected output.

columnNameX|columnNameY|columnNameZ| 
+----------+-----------+-----------+
|John      |Non Hf     |5          |
|Steav     |Non Hf     |5          |
|Ram       |HF         |5          |

回答1:

The main thing here is that generate a query list from given map that you can do like below

//Input DF
val df=Seq(("John","Non Hf","New york"),("Steav","Non Hf","Mumbai"),("Ram","HF","Boston")).toDF("srcColumnZ", "srcColumnY", "srcColumnR")

//Input List

val mapList=List(Map("targetColumn" -> "columnNameX", "sourceColumn" -> List("srcColumnX", "srcColumnY", "srcColumnZ", "srcColumnP", "srcColumnQ", "srcColumnR")), Map("targetColumn" -> "columnNameY", "sourceColumn" -> List("srcColumnY")), Map("targetColumn" -> "columnNameZ", "selectvalue" -> 5))

//Get all the columns of df as list

val dfCols=df.columns.toList

//Then generate query list like below

val query = mapList.map { mp =>
            if (mp.contains("sourceColumn")) {
                val srcColumn = mp.getOrElse("sourceColumn", "sourceColumn key not found").toString.replace("List(", "").replace(")", "").split(",").map(_.trim).toList
                val srcCol = srcColumn.filter(dfCols.contains(_)).head
                df.col(srcCol.toString).alias(mp.getOrElse("targetColumn", "No Target column found").toString)
            } else {
                lit(mp.getOrElse("selectvalue", "No Target column found").toString.replace("(", "").replace(")", "").trim).alias(mp.getOrElse("targetColumn", "No Target column found").toString)
            }
        }

//Finally , fire the query

df.select(query:_*).show

//Sample output:

+-----------+-----------+-----------+
|columnNameX|columnNameY|columnNameZ|
+-----------+-----------+-----------+
|     Non Hf|     Non Hf|          5|
|     Non Hf|     Non Hf|          5|
|         HF|         HF|          5|
+-----------+-----------+-----------+

来源：https://stackoverflow.com/questions/51928103/how-to-create-dataframe-from-listmap-based-on-condition