How to convert spark SchemaRDD into RDD of my case class?

后端 未结 4 1722
长发绾君心
长发绾君心 2020-12-25 08:04

In the spark docs it\'s clear how to create parquet files from RDD of your own case classes; (from the docs)



        
4条回答
  •  悲哀的现实
    2020-12-25 08:48

    there is a simple method to convert schema rdd to rdd using pyspark in Spark 1.2.1.

    sc = SparkContext()  ## create SparkContext
    srdd = sqlContext.sql(sql)
    c = srdd.collect()  ## convert rdd to list
    rdd = sc.parallelize(c)
    

    there must be similar approach using scala.

提交回复
热议问题