Comparison method violates its general contract in Spark

问题

I tried to sort my List[Row] data set and here is how I made for it.

def getDiffMinute(ts1:Timestamp, ts2:Timestamp) : Long = {
    if(ts1==null || ts2==null) 0 
    else (ts1.getTime - ts2.getTime) / 60000
}
myList.sortWith( (r1: Row, r2: Row) => 
    MYUtils.getDiffMinute( r1.getAs[Timestamp]("time"), r2.getAs[Timestamp]("time")) < 0
)

Since getDiffMinute function return Long type data and wortWith need bool type, there is no way to get exception. Some data lists work so well, but others(especially big data like more than 1gb) does not work with this error.

Comparison method violates its general contract

Any Idea of this?

回答1:

I assume its because your comparator getDiffMinute is not properly written. In your case lets say B is null, then diff(A,B) = 0, diff(B,C) = 0 so diff (A,C) should be 0 too, but it can be anything if neither A and C are nulls.

more info: http://docs.oracle.com/javase/6/docs/api/java/util/Comparator.html#compare(T,%20T)

来源：https://stackoverflow.com/questions/36122843/comparison-method-violates-its-general-contract-in-spark