What is wrong with the pearson algorithm from “Programming Collective Intelligence”?

对着背影说爱祢 提交于 2019-12-05 11:08:58

There is nothing wrong in your result. You are trying to plot a line through 3 points. In second case you have all three points with the same coordinates, i.e. effectively one point. You can't say do these points correlate or anti-correlate, because you can draw infinite number of lines through one point (den in your code equals to zero).

If you look up Pearson correlation on wikipedia, you'll see that the formula uses the difference between each item in a series and the mean of the series. When all the items in the series are the same, you get division by zero, so your calculation fails.

If it is any clearer, you can use this code:

def simplified_sim_pearson(p1, p2):
    n = len(p1)
    assert (n != 0)
    sum1 = sum(p1)
    sum2 = sum(p2)
    m1 = float(sum1) / n
    m2 = float(sum2) / n
    p1mean = [(x - m1) for x in p1]
    p2mean = [(y - m2) for y in p2]
    numerator = sum(x * y for x, y in zip(p1mean, p2mean))
    denominator = math.sqrt(sum(x * x for x in p1mean) * sum(y * y for y in p2mean))
    return numerator / denominator if denominator else 0

def sim_pearson(prefs,p1,p2):
    p1 = prefs[p1]
    p2 = prefs[p2]
    si = set(p1.keys()).intersection(set(p2.keys()))
    p1_x = [p1[k] for k in sorted(si)]
    p2_x = [p2[k] for k in sorted(si)]
    return simplified_sim_pearson(p1_x, p2_x)



critics = {
    'user1':{
        'item1': 3,
        'item2': 5,
        'item3': 5,
        },
    'user2':{
        'item1': 4,
        'item2': 5,
        'item3': 5,
        }
}
critics2 = {
    'user1':{
        'item1': 5,
        'item2': 5,
        'item3': 5,
        },
    'user2':{
        'item1': 5,
        'item2': 5,
        'item3': 5,
        }
}
critics3 = {
    'user1':{
        'item1': 1,
        'item2': 3,
        'item3': 5,
        },
    'user2':{
        'item1': 5,
        'item2': 3,
        'item3': 1,
        }
}

print sim_pearson(critics, 'user1', 'user2', )
print sim_pearson(critics2, 'user1', 'user2', )
print sim_pearson(critics3, 'user1', 'user2', )

By the way, using Excel to determine the correct answer is a good way to validate most calculations. In this case, you would have used correl.

The algorithm gives the correct result. 0 means that there is no correlation between them (or at least you can't tell from what you know).

Generally (depending on what domain you apply this algorithm) you can consider everything between -0.9 < x < 0.09 as "No correlation observable".

Correlation does not imply causation. Had to say it. You need to develop an understanding of correlation statistics. Correlation can be between -1 and 1 and a value of 0 falls in this range and is a perfectly reasonable result. A correlation of 0 implies that there is not statistically significant relationship between the 2 variables. Remember to avoid doing statistics with less that 30 samples.

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!