hadoop-partitioning

What is the use of grouping comparator in hadoop map reduce

久未见 提交于 2019-11-27 06:38:17
I would like to know why grouping comparator is used in secondary sort of mapreduce. According to the definitive guide example of secondary sorting We want the sort order for keys to be by year (ascending) and then by temperature (descending): 1900 35°C 1900 34°C 1900 34°C ... 1901 36°C 1901 35°C By setting a partitioner to partition by the year part of the key, we can guarantee that records for the same year go to the same reducer. This still isn’t enough to achieve our goal, however. A partitioner ensures only that one reducer receives all the records for a year; it doesn’t change the fact

hadoop map reduce secondary sorting

烈酒焚心 提交于 2019-11-26 17:34:05
Can any one explain me how secondary sorting works in hadoop ? Why must one use GroupingComparator and how does it work in hadoop ? I was going through the link given below and got doubt on how groupcomapator works. Can any one explain me how grouping comparator works? http://www.bigdataspeak.com/2013/02/hadoop-how-to-do-secondary-sort-on_25.html Deepika C P Grouping Comparator Once the data reaches a reducer, all data is grouped by key. Since we have a composite key, we need to make sure records are grouped solely by the natural key. This is accomplished by writing a custom GroupPartitioner.

What is the use of grouping comparator in hadoop map reduce

不羁岁月 提交于 2019-11-26 12:05:57
问题 I would like to know why grouping comparator is used in secondary sort of mapreduce. According to the definitive guide example of secondary sorting We want the sort order for keys to be by year (ascending) and then by temperature (descending): 1900 35°C 1900 34°C 1900 34°C ... 1901 36°C 1901 35°C By setting a partitioner to partition by the year part of the key, we can guarantee that records for the same year go to the same reducer. This still isn’t enough to achieve our goal, however. A

hadoop map reduce secondary sorting

这一生的挚爱 提交于 2019-11-26 05:28:39
问题 Can any one explain me how secondary sorting works in hadoop ? Why must one use GroupingComparator and how does it work in hadoop ? I was going through the link given below and got doubt on how groupcomapator works. Can any one explain me how grouping comparator works? http://www.bigdataspeak.com/2013/02/hadoop-how-to-do-secondary-sort-on_25.html 回答1: Grouping Comparator Once the data reaches a reducer, all data is grouped by key. Since we have a composite key, we need to make sure records