Calculating Precision, Recall and F-score in one pass - python

Question

Accuracy, precision, recall and f-score are measures of a system quality in machine-learning systems. It depends on a confusion matrix of True/False Positives/Negatives.

Given a binary classification task, I have tried the following to get a function that returns accuracy, precision, recall and f-score:

gold = [1] + [0] * 9
predicted = [1] * 10

def evaluation(gold, predicted):
  true_pos = sum(1 for p,g in zip(predicted, gold) if p==1 and g==1)
  true_neg = sum(1 for p,g in zip(predicted, gold) if p==0 and g==0)
  false_pos = sum(1 for p,g in zip(predicted, gold) if p==1 and g==0)
  false_neg = sum(1 for p,g in zip(predicted, gold) if p==0 and g==1)
  try:
    recall = true_pos / float(true_pos + false_neg)
  except:
    recall = 0
  try:
    precision = true_pos / float(true_pos + false_pos)
  except:
    precision = 0
  try:
    fscore = 2*precision*recall / (precision + recall)
  except:
    fscore = 0
  try:
    accuracy = (true_pos + true_neg) / float(len(gold))
  except:
    accuracy = 0
  return accuracy, precision, recall, fscore

But it seems like I have redundantly looped through the dataset 4 times to get the True/False Positives/Negatives.

Also the multiple try-excepts to catch the ZeroDivisionError is a little redundant.

So what is the pythonic way to get the counts of the True/False Positives/Negatives without multiple loops through the dataset?

How do I pythonically catch the ZeroDivisionError without the multiple try-excepts?

---

I could also do the following to count the True/False Positives/Negatives in one loop but is there an alternative way without the multiple if?:

for p,g in zip(predicted, gold):
    if p==1 and g==1:
        true_pos+=1
    if p==0 and g==0:
        true_neg+=1
    if p==1 and g==0:
        false_pos+=1
    if p==0 and g==1:
        false_neg+=1

Answer 1

what is the pythonic way to get the counts of the True/False Positives/Negatives without multiple loops through the dataset?

I would use a collections.Counter, roughly what you're doing with all of the ifs (you should be using elifs, as your conditions are mutually exclusive) at the end:

counts = Counter(zip(predicted, gold))

Then e.g. true_pos = counts[1, 1].

How do I pythonically catch the ZeroDivisionError without the multiple try-excepts?

For a start, you should (almost) never use a bare except:. If you're catching ZeroDivisionErrors, then write except ZeroDivisionError. You could also consider a "look before you leap" approach, checking whether the denominator is 0 before trying the division, e.g.

accuracy = (true_pos + true_neg) / float(len(gold)) if gold else 0

Answer 2

This is a pretty natural use case for the bitarray package.

import bitarray as bt

tp = (bt.bitarray(p) & bt.bitarray(g)).count()
tn = (~bt.bitarray(p) & ~bt.bitarray(g)).count()
fp = (bt.bitarray(p) & ~bt.bitarray(g)).count()
fn = (~bt.bitarray(p) & bt.bitarray(g)).count()

There's some type conversion overhead, but after that, the bitwise operations are much faster.

For 100 instances, timeit on my PC gives 0.036 for your method and 0.017 using bitarray at 1000 passes. For 1000 instances, it goes to 0.291 and 0.093. For 10000, 3.177 and 0.863. You get the idea.

It scales pretty well, using no loops, and doesn't have to store a large intermediate representation building a temporary list of tuples in zip.

Answer 3

Depending on your needs, there are several libraries that will calculate precision, recall, F-score, etc. One that I have used is scikit-learn. Assuming that you have aligned lists of actual and predicted values, then it is as simple as...

from sklearn.metrics import precision_recall_fscore_support as pr
bPrecis, bRecall, bFscore, bSupport = pr(gold, predicted, average='binary')

One of the advantages of using this library is that different flavors of metrics (such as micro-averaging, macro-averaging, weighted, binary, etc.) come free out of the box.