Hadoop one Map and multiple Reduce

前端 未结 6 1005
陌清茗
陌清茗 2020-12-23 16:48

We have a large dataset to analyze with multiple reduce functions.

All reduce algorithm work on the same dataset generated by the s

6条回答
  •  天命终不由人
    2020-12-23 17:31

    You may use composite keys. Let's say you need two kinds of the reducers, 'R1' and 'R2'. Add ids for these as a prefix to your o/p keys in the mapper. So, in the mapper, a key 'K' now becomes 'R1:K' or 'R2:K'.

    Then, in the reducer, pass values to implementations of R1 or R2 based on the prefix.

提交回复
热议问题