casting a column dynamically in spark dataframe

Question

I want to to be able to create a new column out of an existing column(of type string) and cast it to a type dynamically.

resultDF = resultDF.withColumn(newColumn         


        

Answer 1

Why not use string descriptions?

scala> col("foo").cast("int")
res2: org.apache.spark.sql.Column = CAST(foo AS INT)

scala> col("foo").cast("string")
res3: org.apache.spark.sql.Column = CAST(foo AS STRING)

Otherwise use DataType, which will cover all primitive types and basic collections.