Hadoop one Map and multiple Reduce

前端 未结 6 1021
陌清茗
陌清茗 2020-12-23 16:48

We have a large dataset to analyze with multiple reduce functions.

All reduce algorithm work on the same dataset generated by the s

6条回答
  •  难免孤独
    2020-12-23 17:37

    I still dont get your problem you can use following sequence:

    database-->map-->reduce(use cat or None depending on requirement) then store the data representation you have extracted. if you are saying that it is small enough to fit in memory then storing it on disk shouldnt be an issue.

    Also your use of MapReduce paradigm for the given problem is incorrect, using a single map function and multiple "different" reduce function makes no sense, it shows that you are just using map to pass out data to different machines to do different things. you dont require hadoop or any other special architecture for that.

提交回复
热议问题