Sort order with Hadoop MapRed

问题

Well,

I'd like to know how can I change the sort order of my simple WordCount program after the reduce task? I've already made another map to order by value instead by keys, but it still ordered in ascending order. Is there an easy simple method to do this (change the sort order)?!

Thanks Vellozo

回答1:

If you are using the older API (mapred.*), then set the OutputKeyComparatorClass in the job conf:

jobConf.setOutputKeyComparatorClass(ReverseComparator.class);

ReverseComparator can be something like this:

static class ReverseComparator extends WritableComparator {
        private static final Text.Comparator TEXT_COMPARATOR = new Text.Comparator();

        public ReverseComparator() {
            super(Text.class);
        }

        @Override
        public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
            try {
                return (-1)* TEXT_COMPARATOR
                        .compare(b1, s1, l1, b2, s2, l2);
            } catch (IOException e) {
                throw new IllegalArgumentException(e);
            }
        }

        @Override
        public int compare(WritableComparable a, WritableComparable b) {
            if (a instanceof Text && b instanceof Text) {
                return (-1)*(((Text) a)
                        .compareTo((Text) b)));
            }
            return super.compare(a, b);
        }
    }

In the new API (mapreduce.*), I think you need to use the Job.setSortComparator() method.

回答2:

This one is almost the same as above, just looks a bit simpler

class MyKeyComparator extends WritableComparator {
    protected DescendingKeyComparator() {
        super(Text.class, true);
    }

    @SuppressWarnings("rawtypes")
    @Override
    public int compare(WritableComparable w1, WritableComparable w2) {
        Text key1 = (Text) w1;
        Text key2 = (Text) w2;          
        return -1 * key1.compareTo(key2);
    }
}

Then add it it to the job

job.setSortComparatorClass(MyKeyComparator.class);

Text key1 = (Text) w1;
            Text key2 = (Text) w2;

you can change the above text type as per ur use.

来源：https://stackoverflow.com/questions/9493644/sort-order-with-hadoop-mapred

标签

sorting

Hadoop

MapReduce