Spark structured streaming kafka convert JSON without schema (infer schema)

前端 未结 5 2108
时光说笑
时光说笑 2020-12-08 17:18

I read Spark Structured Streaming doesn\'t support schema inference for reading Kafka messages as JSON. Is there a way to retrieve schema the same as Spark Streaming does:

5条回答
  •  陌清茗
    陌清茗 (楼主)
    2020-12-08 17:53

    Taking Arnon's solution to the next step (since it's deprecated in spark's newer versions, and would require iterating the whole dataframe just for a type casting)

    spark.read.json(df.as[String])
    

    Anyways, as for now, it's still experimental.

提交回复
热议问题