hadoop: difference between 0 reducer and identity reducer?

前端 未结 4 871
故里飘歌
故里飘歌 2020-12-01 05:50

I am just trying to confirm my understanding of difference between 0 reducer and identity reducer.

  • 0 reducer means reduce step will be skipped and mapper outp
4条回答
  •  陌清茗
    陌清茗 (楼主)
    2020-12-01 06:13

    Another use-case for using the Identity Reducer is to combine all the results into <# of reducers> output files. This can be handy if you are using Amazon Web Services to write to S3 directly, especially if the mapper output is small (e.g. a grep/search for a record), and you have a lot of mappers (e.g. 1000's).

提交回复
热议问题