可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效，请关闭广告屏蔽插件后再试):
问题:
I wrote a confusion matrix calculation code in Python:
def conf_mat(prob_arr, input_arr): # confusion matrix conf_arr = [[0, 0], [0, 0]] for i in range(len(prob_arr)): if int(input_arr[i]) == 1: if float(prob_arr[i]) = 0.5: conf_arr[1][0] = conf_arr[1][0] +1 else: conf_arr[1][1] = conf_arr[1][1] +1 accuracy = float(conf_arr[0][0] + conf_arr[1][1])/(len(input_arr))
prob_arr is an array that my classification code returned and a sample array is like this:
[1.0, 1.0, 1.0, 0.41592955657342651, 1.0, 0.0053405015805891975, 4.5321494433440449e-299, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.70943426182688163, 1.0, 1.0, 1.0, 1.0]
input_arr is the original class labels for a dataset and it is like this:
[2, 1, 1, 1, 1, 1, 2, 1, 1, 2, 1, 1, 2, 1, 2, 1, 1, 1]
What my code is trying to do is: i get prob_arr and input_arr and for each class (1 and 2) I check if they are misclassified or not.
But my code only works for two classes. If I run this code for a multiple classed data, it doesn't work. How can I make this for multiple classes?
For example, for a data set with three classes, it should return: [[21,7,3],[3,38,6],[5,4,19]]
回答1:
Scikit-Learn provides a confusion_matrix function
from sklearn.metrics import confusion_matrix y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2] y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2] confusion_matrix(y_actu, y_pred)
which output a Numpy array
array([[3, 0, 0], [0, 1, 2], [2, 1, 3]])
But you can also create a confusion matrix using Pandas:
import pandas as pd y_actu = pd.Series([2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2], name='Actual') y_pred = pd.Series([0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2], name='Predicted') df_confusion = pd.crosstab(y_actu, y_pred)
You will get a (nicely labeled) Pandas DataFrame:
Predicted 0 1 2 Actual 0 3 0 0 1 0 1 2 2 2 1 3
If you add margins=True like
df_confusion = pd.crosstab(y_actu, y_pred, rownames=['Actual'], colnames=['Predicted'], margins=True)
you will get also sum for each row and column:
Predicted 0 1 2 All Actual 0 3 0 0 3 1 0 1 2 3 2 2 1 3 6 All 5 2 5 12
You can also get a normalized confusion matrix using:
df_conf_norm = df_confusion / df_confusion.sum(axis=1) Predicted 0 1 2 Actual 0 1.000000 0.000000 0.000000 1 0.000000 0.333333 0.333333 2 0.666667 0.333333 0.500000
You can plot this confusion_matrix using
def plot_confusion_matrix(df_confusion, title='Confusion matrix', cmap=plt.cm.gray_r): plt.matshow(df_confusion, cmap=cmap) # imshow #plt.title(title) plt.colorbar() tick_marks = np.arange(len(df_confusion.columns)) plt.xticks(tick_marks, df_confusion.columns, rotation=45) plt.yticks(tick_marks, df_confusion.index) #plt.tight_layout() plt.ylabel(df_confusion.index.name) plt.xlabel(df_confusion.columns.name) plot_confusion_matrix(df_confusion)
Or plot normalized confusion matrix using:
plot_confusion_matrix(df_conf_norm)
You might also be interested by this project https://github.com/pandas-ml/pandas-ml and its Pip package https://pypi.python.org/pypi/pandas_ml
With this package confusion matrix can be pretty-printed, plot. You can binarize a confusion matrix, get class statistics such as TP, TN, FP, FN, ACC, TPR, FPR, FNR, TNR (SPC), LR+, LR-, DOR, PPV, FDR, FOR, NPV and some overall statistics
In [1]: from pandas_ml import ConfusionMatrix In [2]: y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2] In [3]: y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2] In [4]: cm = ConfusionMatrix(y_actu, y_pred) In [5]: cm.print_stats() Confusion Matrix: Predicted 0 1 2 __all__ Actual 0 3 0 0 3 1 0 1 2 3 2 2 1 3 6 __all__ 5 2 5 12 Overall Statistics: Accuracy: 0.583333333333 95% CI: (0.27666968568210581, 0.84834777019156982) No Information Rate: ToDo P-Value [Acc > NIR]: 0.189264302376 Kappa: 0.354838709677 Mcnemar's Test P-Value: ToDo Class Statistics: Classes 0 1 2 Population 12 12 12 P: Condition positive 3 3 6 N: Condition negative 9 9 6 Test outcome positive 5 2 5 Test outcome negative 7 10 7 TP: True Positive 3 1 3 TN: True Negative 7 8 4 FP: False Positive 2 1 2 FN: False Negative 0 2 3 TPR: (Sensitivity, hit rate, recall) 1 0.3333333 0.5 TNR=SPC: (Specificity) 0.7777778 0.8888889 0.6666667 PPV: Pos Pred Value (Precision) 0.6 0.5 0.6 NPV: Neg Pred Value 1 0.8 0.5714286 FPR: False-out 0.2222222 0.1111111 0.3333333 FDR: False Discovery Rate 0.4 0.5 0.4 FNR: Miss Rate 0 0.6666667 0.5 ACC: Accuracy 0.8333333 0.75 0.5833333 F1 score 0.75 0.4 0.5454545 MCC: Matthews correlation coefficient 0.6831301 0.2581989 0.1690309 Informedness 0.7777778 0.2222222 0.1666667 Markedness 0.6 0.3 0.1714286 Prevalence 0.25 0.25 0.5 LR+: Positive likelihood ratio 4.5 3 1.5 LR-: Negative likelihood ratio 0 0.75 0.75 DOR: Diagnostic odds ratio inf 4 2 FOR: False omission rate 0 0.2 0.4285714
回答2:
Scikit-learn (which I recommend using anyways) has it included in the metrics module:
>>> from sklearn.metrics import confusion_matrix >>> y_true = [0, 1, 2, 0, 1, 2, 0, 1, 2] >>> y_pred = [0, 0, 0, 0, 1, 1, 0, 2, 2] >>> confusion_matrix(y_true, y_pred) array([[3, 0, 0], [1, 1, 1], [1, 1, 1]])
回答3:
If you don't want scikit-learn to do the work for you...
import numpy actual = numpy.array(actual) predicted = numpy.array(predicted) # calculate the confusion matrix; labels is numpy array of classification labels cm = numpy.zeros((len(labels), len(labels))) for a, p in zip(actual, predicted): cm[a][p] += 1 # also get the accuracy easily with numpy accuracy = (actual == predicted).sum() / float(len(actual))
Or take a look at a more complete implementation here in NLTK.
回答4:
This function creates confusion matrices for any number of classes.
def create_conf_matrix(expected, predicted, n_classes): m = [[0] * n_classes for i in range(n_classes)] for pred, exp in zip(predicted, expected): m[pred][exp] += 1 return m def calc_accuracy(conf_matrix): t = sum(sum(l) for l in conf_matrix) return sum(conf_matrix[i][i] for i in range(len(conf_matrix))) / t
In contrast to your function above, you have to extract the predicted classes before calling the function, based on your classification results, i.e. sth. like
[1 if p
回答5:
回答6:
Nearly a decade has passed, yet the solutions (without sklearn) to this post are convoluted and unnecessarily long. Computing a confusion matrix can be done cleanly in Python in a few lines. The following function is an example:
def compute_confusion_matrix(true, pred): '''Computes a confusion matrix using numpy for two np.arrays true and pred. Results are identical (and similar in computation time) to: "from sklearn.metrics import confusion_matrix" However, this function avoids the dependency on sklearn.''' K = len(np.unique(true)) # Number of classes result = np.zeros((K, K)) for i in range(len(true)): result[true[i]][pred[i]] += 1 return result
回答7:
You should map from classes to a row in your confusion matrix.
Here the mapping is trivial:
def row_of_class(classe): return {1: 0, 2: 1}[classe]
In your loop, compute expected_row, correct_row, and increment conf_arr[expected_row][correct_row]. You'll even have less code than what you started with.
回答8:
In a general sense, you're going to need to change your probability array. Instead of having one number for each instance and classifying based on whether or not it is greater than 0.5, you're going to need a list of scores (one for each class), then take the largest of the scores as the class that was chosen (a.k.a. argmax).
You could use a dictionary to hold the probabilities for each classification:
prob_arr = [{classification_id: probability}, ...]
Choosing a classification would be something like:
for instance_scores in prob_arr : predicted_classes = [cls for (cls, score) in instance_scores.iteritems() if score = max(instance_scores.values())]
This handles the case where two classes have the same scores. You can get one score, by choosing the first one in that list, but how you handle that depends on what you're classifying.
Once you have your list of predicted classes and a list of expected classes you can use code like Torsten Marek's to create the confusion array and calculate the accuracy.
回答9:
You can make your code more concise and (sometimes) to run faster using numpy. For example, in two-classes case your function can be rewritten as (see mply.acc()):
def accuracy(actual, predicted): """accuracy = (tp + tn) / ts , where: ts - Total Samples tp - True Positives tn - True Negatives """ return (actual == predicted).sum() / float(len(actual))
, where:
actual = (numpy.array(input_arr) == 2) predicted = (numpy.array(prob_arr)
回答10:
I wrote a simple class to build a confusion matrix without the need to depend on a machine learning library.
The class can be used such as:
labels = ["cat", "dog", "velociraptor", "kraken", "pony"] confusionMatrix = ConfusionMatrix(labels) confusionMatrix.update("cat", "cat") confusionMatrix.update("cat", "dog") ... confusionMatrix.update("kraken", "velociraptor") confusionMatrix.update("velociraptor", "velociraptor") confusionMatrix.plot()
The class ConfusionMatrix:
import pylab import collections import numpy as np class ConfusionMatrix: def __init__(self, labels): self.labels = labels self.confusion_dictionary = self.build_confusion_dictionary(labels) def update(self, predicted_label, expected_label): self.confusion_dictionary[expected_label][predicted_label] += 1 def build_confusion_dictionary(self, label_set): expected_labels = collections.OrderedDict() for expected_label in label_set: expected_labels[expected_label] = collections.OrderedDict() for predicted_label in label_set: expected_labels[expected_label][predicted_label] = 0.0 return expected_labels def convert_to_matrix(self, dictionary): length = len(dictionary) confusion_dictionary = np.zeros((length, length)) i = 0 for row in dictionary: j = 0 for column in dictionary: confusion_dictionary[i][j] = dictionary[row][column] j += 1 i += 1 return confusion_dictionary def get_confusion_matrix(self): matrix = self.convert_to_matrix(self.confusion_dictionary) return self.normalize(matrix) def normalize(self, matrix): amin = np.amin(matrix) amax = np.amax(matrix) return [[(((y - amin) * (1 - 0)) / (amax - amin)) for y in x] for x in matrix] def plot(self): matrix = self.get_confusion_matrix() pylab.figure() pylab.imshow(matrix, interpolation='nearest', cmap=pylab.cm.jet) pylab.title("Confusion Matrix") for i, vi in enumerate(matrix): for j, vj in enumerate(vi): pylab.text(j, i+.1, "%.1f" % vj, fontsize=12) pylab.colorbar() classes = np.arange(len(self.labels)) pylab.xticks(classes, self.labels) pylab.yticks(classes, self.labels) pylab.ylabel('Expected label') pylab.xlabel('Predicted label') pylab.show()
回答11:
Only with numpy, we can do as follow considering efficiency:
def confusion_matrix(pred, label, nc=None): assert pred.size == label.size if nc is None: nc = len(unique(label)) logging.debug("Number of classes assumed to be {}".format(nc)) confusion = np.zeros([nc, nc]) # avoid the confusion with `0` tran_pred = pred + 1 for i in xrange(nc): # current class mask = (label == i) masked_pred = mask * tran_pred cls, counts = unique(masked_pred, return_counts=True) # discard the first item cls = [cl - 1 for cl in cls][1:] counts = counts[1:] for cl, count in zip(cls, counts): confusion[i, cl] = count return confusion
For other features such as plot, mean-IoU, see my repositories.
回答12:
A Simple Multiclass Implementation
A confusion matrix can be computed incredibly simply with vanilla Python. All we need to do is pair up the unique classes found in the actual vector into a 2-dimensional list. From there, we simply iterate through the zipped actual and predicted vectors and populate the counts.
# A Simple Confusion Matrix Implementation def confusionmatrix(actual, predicted, normalize = False): """ Generate a confusion matrix for multiple classification @params: actual - a list of integers or strings for known classes predicted - a list of integers or strings for predicted classes normalize - optional boolean for matrix normalization @return: matrix - a 2-dimensional list of pairwise counts """ unique = sorted(set(actual)) matrix = [[0 for _ in unique] for _ in unique] imap = {key: i for i, key in enumerate(unique)} # Generate Confusion Matrix for p, a in zip(predicted, actual): matrix[imap[p]][imap[a]] += 1 # Matrix Normalization if normalize: sigma = sum([sum(matrix[imap[i]]) for i in unique]) matrix = [row for row in map(lambda i: list(map(lambda j: j / sigma, i)), matrix)] return matrix
Usage
# Input Below Should Return: [[2, 1, 0], [0, 2, 1], [1, 2, 1]] cm = confusionmatrix( [1, 1, 2, 0, 1, 1, 2, 0, 0, 1], # actual [0, 1, 1, 0, 2, 1, 2, 2, 0, 2] # predicted ) # And The Output print(cm) [[2, 1, 0], [0, 2, 1], [1, 2, 1]]
Note: the actual classes are along the columns and the predicted classes are along the rows.
Actual 0 1 2 # # # [[2, 1, 0], # 0 [0, 2, 1], # 1 Predicted [1, 2, 1]] # 2
Class Names Can be Strings or Integers
# Input Below Should Return: [[2, 1, 0], [0, 2, 1], [1, 2, 1]] cm = confusionmatrix( ["B", "B", "C", "A", "B", "B", "C", "A", "A", "B"], # actual ["A", "B", "B", "A", "C", "B", "C", "C", "A", "C"] # predicted ) # And The Output print(cm) [[2, 1, 0], [0, 2, 1], [1, 2, 1]]
You Can Also Return The Matrix With Proportions (Normalization)
# Input Below Should Return: [[0.2, 0.1, 0.0], [0.0, 0.2, 0.1], [0.1, 0.2, 0.1]] cm = confusionmatrix( ["B", "B", "C", "A", "B", "B", "C", "A", "A", "B"], # actual ["A", "B", "B", "A", "C", "B", "C", "C", "A", "C"], # predicted normalize = True ) # And The Output print(cm) [[0.2, 0.1, 0.0], [0.0, 0.2, 0.1], [0.1, 0.2, 0.1]]
Extracting Statistics From a Multiple Classification Confusion Matrix
Once you have the matrix, you can compute a bunch of statistics to assess your classifier. That said, extracting the values out of a confusion matrix setup for multiple classification can be a bit of a headache. Here's a function that returns both the confusion matrix and statistics by class:
# Not Required, But Nice For Legibility from collections import OrderedDict # A Simple Confusion Matrix Implementation def confusionmatrix(actual, predicted, normalize = False): """ Generate a confusion matrix for multiple classification @params: actual - a list of integers or strings for known classes predicted - a list of integers or strings for predicted classes @return: matrix - a 2-dimensional list of pairwise counts statistics - a dictionary of statistics for each class """ unique = sorted(set(actual)) matrix = [[0 for _ in unique] for _ in unique] imap = {key: i for i, key in enumerate(unique)} # Generate Confusion Matrix for p, a in zip(predicted, actual): matrix[imap[p]][imap[a]] += 1 # Get Confusion Matrix Sum sigma = sum([sum(matrix[imap[i]]) for i in unique]) # Scaffold Statistics Data Structure statistics = OrderedDict(((i, {"counts" : OrderedDict(), "stats" : OrderedDict()}) for i in unique)) # Iterate Through Classes & Compute Statistics for i in unique: loc = matrix[imap[i]][imap[i]] row = sum(matrix[imap[i]][:]) col = sum([row[imap[i]] for row in matrix]) # Get TP/TN/FP/FN tp = loc fp = row - loc fn = col - loc tn = sigma - row - col + loc # Populate Counts Dictionary statistics[i]["counts"]["tp"] = tp statistics[i]["counts"]["fp"] = fp statistics[i]["counts"]["tn"] = tn statistics[i]["counts"]["fn"] = fn statistics[i]["counts"]["pos"] = tp + fn statistics[i]["counts"]["neg"] = tn + fp statistics[i]["counts"]["n"] = tp + tn + fp + fn # Populate Statistics Dictionary statistics[i]["stats"]["sensitivity"] = tp / (tp + fn) if tp > 0 else 0.0 statistics[i]["stats"]["specificity"] = tn / (tn + fp) if tn > 0 else 0.0 statistics[i]["stats"]["precision"] = tp / (tp + fp) if tp > 0 else 0.0 statistics[i]["stats"]["recall"] = tp / (tp + fn) if tp > 0 else 0.0 statistics[i]["stats"]["tpr"] = tp / (tp + fn) if tp > 0 else 0.0 statistics[i]["stats"]["tnr"] = tn / (tn + fp) if tn > 0 else 0.0 statistics[i]["stats"]["fpr"] = fp / (fp + tn) if fp > 0 else 0.0 statistics[i]["stats"]["fnr"] = fn / (fn + tp) if fn > 0 else 0.0 statistics[i]["stats"]["accuracy"] = (tp + tn) / (tp + tn + fp + fn) if (tp + tn) > 0 else 0.0 statistics[i]["stats"]["f1score"] = (2 * tp) / ((2 * tp) + (fp + fn)) if tp > 0 else 0.0 statistics[i]["stats"]["fdr"] = fp / (fp + tp) if fp > 0 else 0.0 statistics[i]["stats"]["for"] = fn / (fn + tn) if fn > 0 else 0.0 statistics[i]["stats"]["ppv"] = tp / (tp + fp) if tp > 0 else 0.0 statistics[i]["stats"]["npv"] = tn / (tn + fn) if tn > 0 else 0.0 # Matrix Normalization if normalize: matrix = [row for row in map(lambda i: list(map(lambda j: j / sigma, i)), matrix)] return matrix, statistics
Computed Statistics
Above, the confusion matrix is used to tabulate statistics for each class, which are returned in an OrderedDict with the following structure:
OrderedDict( [ ('A', { 'stats' : OrderedDict([ ('sensitivity', 0.6666666666666666), ('specificity', 0.8571428571428571), ('precision', 0.6666666666666666), ('recall', 0.6666666666666666), ('tpr', 0.6666666666666666), ('tnr', 0.8571428571428571), ('fpr', 0.14285714285714285), ('fnr', 0.3333333333333333), ('accuracy', 0.8), ('f1score', 0.6666666666666666), ('fdr', 0.3333333333333333), ('for', 0.14285714285714285), ('ppv', 0.6666666666666666), ('npv', 0.8571428571428571) ]), 'counts': OrderedDict([ ('tp', 2), ('fp', 1), ('tn', 6), ('fn', 1), ('pos', 3), ('neg', 7), ('n', 10) ]) }), ('B', { 'stats': OrderedDict([ ('sensitivity', 0.4), ('specificity', 0.8), ('precision', 0.6666666666666666), ('recall', 0.4), ('tpr', 0.4), ('tnr', 0.8), ('fpr', 0.2), ('fnr', 0.6), ('accuracy', 0.6), ('f1score', 0.5), ('fdr', 0.3333333333333333), ('for', 0.42857142857142855), ('ppv', 0.6666666666666666), ('npv', 0.5714285714285714) ]), 'counts': OrderedDict([ ('tp', 2), ('fp', 1), ('tn', 4), ('fn', 3), ('pos', 5), ('neg', 5), ('n', 10) ]) }), ('C', { 'stats': OrderedDict([ ('sensitivity', 0.5), ('specificity', 0.625), ('precision', 0.25), ('recall', 0.5), ('tpr', 0.5), ('tnr', 0.625), ( 'fpr', 0.375), ( 'fnr', 0.5), ('accuracy', 0.6), ('f1score', 0.3333333333333333), ('fdr', 0.75), ('for', 0.16666666666666666), ('ppv', 0.25), ('npv', 0.8333333333333334) ]), 'counts': OrderedDict([ ('tp', 1), ('fp', 3), ('tn', 5), ('fn', 1), ('pos', 2), ('neg', 8), ('n', 10) ]) }) ] )
