Spark SQL - Custom Datatype UUID

问题:

i am trying to convert the Column in the Dataset from varchar to UUID using the custom datatype in Spark SQL. But i see the conversion not happening. Please let me know if i am missing anything here.

val secdf = sc.parallelize( Array(("85d8b889-c793-4f23-93e9-ea18db640039","Revenue"), ("85d8b889-c793-4f23-93e9-ea18db640038","Income:123213"))).toDF("id", "report") val metadataBuilder = new MetadataBuilder() metadataBuilder.putString("database.column.type", "uuid") metadataBuilder.putLong("jdbc.type", java.sql.Types.OTHER) val metadata = metadataBuilder.build() val secReportDF = secdf.withColumn("id", col("id").as("id", metadata)) 

回答1:

i did the work around as we are not able to cast to UUID in Spark SQL, i have added the property in the Postgres JDBC client as stringtype=unspecified which solved my issue in Inserting UUID through Spark JDBC

文章来源: Spark SQL - Custom Datatype UUID