How to use Hadoop InputFormats In Apache Spark?

后端 未结 2 1580
难免孤独
难免孤独 2021-02-20 06:13

I have a class ImageInputFormat in Hadoop which reads images from HDFS. How to use my InputFormat in Spark?

Here is my ImageInputFormat:

<
2条回答
  •  轻奢々
    轻奢々 (楼主)
    2021-02-20 07:10

    The SparkContext has a method called hadoopFile. It accepts classes implementing the interface org.apache.hadoop.mapred.InputFormat

    Its description says "Get an RDD for a Hadoop file with an arbitrary InputFormat".

    Also have a look at the Spark Documentation.

提交回复
热议问题