Confusion Matrix, Precision/Recall & ROC

Scores and a threshold

Most classifiers, such as logistic regression or a neural network, do not output “yes” or “no” directly. They output a score, often a probability, and we decide:

predict POSITIVE if score ≥ threshold, otherwise NEGATIVE

The threshold is our choice, and it changes every number we report. In the 3D model, orange balls are examples that are really positive and blue ones are really negative, placed by their score. Everything to the right of the yellow line is predicted positive.

The confusion matrix

Every prediction falls into one of four boxes:

	Predicted positive	Predicted negative
Actually positive	TP true positive (found it)	FN false negative (missed it)
Actually negative	FP false positive (false alarm)	TN true negative (correctly ignored)

From these four counts come all the usual measures:

Measure	Formula	Question it answers
Accuracy	(TP + TN) / all	How many predictions are right?
Precision	TP / (TP + FP)	When it says “positive”, how often is it right?
Recall (sensitivity, TPR)	TP / (TP + FN)	Of all real positives, how many did it find?
F1	2 · P · R / (P + R)	One number balancing precision and recall (their harmonic mean)
FPR (false positive rate)	FP / (FP + TN)	Of all real negatives, how many raised a false alarm?
Specificity	TN / (TN + FP) = 1 − FPR	Of all real negatives, how many were correctly ignored?

Example. A spam filter flags 50 e-mails. 40 are really spam (TP = 40) and 10 are not (FP = 10). It also misses 20 spam e-mails (FN = 20). Precision = 40/50 = 0.8; recall = 40/60 ≈ 0.67.

Moving the threshold

For the Good model (10 positives, 10 negatives):

Threshold	TP	FN	FP	TN	Precision	Recall	F1	Accuracy
0.3	10	0	5	5	0.667	1.0	0.8	0.75
0.5	8	2	2	8	0.8	0.8	0.8	0.8
0.7	6	4	0	10	1.0	0.6	0.75	0.8
0.85	3	7	0	10	1.0	0.3	0.462	0.65

This is the precision–recall trade-off. A low threshold catches every positive but raises many false alarms. A high threshold is almost never wrong when it says “positive”, but misses many. Which is better depends on the cost of each mistake:

• Cancer screening: a missed case (FN) is far worse than a false alarm, so prefer high recall (low threshold).
• Spam filter: losing a real e-mail (FP) is worse than seeing some spam, so prefer high precision (high threshold).

The ROC curve

Instead of picking one threshold, look at all of them. Start with a threshold above every score (nothing is positive: the point (0, 0)) and lower it past one example at a time:

• passing a positive moves the point up (TPR increases by 1/P),
• passing a negative moves it right (FPR increases by 1/N).

At threshold 0 everything is positive: the point (1, 1). The curve of (FPR, TPR) points is the ROC curve (receiver operating characteristic, a name from radar engineering).

• A perfect model goes straight up to (0, 1) and then right: every positive outscores every negative.
• A random model follows the diagonal.
• The area under the curve, AUC, summarises it in one number. It equals the probability that a randomly chosen positive gets a higher score than a randomly chosen negative.

Data	AUC	At threshold 0.5
Good model	0.92	precision 0.8, recall 0.8
Weak model	0.55	precision 0.556, recall 0.5
Imbalanced	0.875	precision 0.5, recall 0.75

AUC does not depend on the threshold, which makes it good for comparing models. To choose the threshold itself you still need to decide how much each kind of error costs.

The accuracy trap

The Imbalanced data has only 4 positives and 16 negatives. At threshold 0.5 the model gets accuracy 0.8. A rule that ignores the input and always says negative gets… also 0.8, with recall 0: it never finds a single positive. Accuracy hides this; recall, F1 and the confusion matrix show it at once. Whenever classes are imbalanced (fraud, disease, defects), report precision, recall and F1, or use a precision–recall curve, which is more informative than ROC when positives are rare.

Code

DATA = {
    'good': ([0.95, 0.9, 0.86, 0.81, 0.77, 0.7, 0.64, 0.55, 0.42, 0.33],      # scores of the positives
             [0.6, 0.5, 0.45, 0.38, 0.3, 0.24, 0.18, 0.12, 0.08, 0.03]),      # scores of the negatives
    'weak': ([0.9, 0.75, 0.66, 0.58, 0.52, 0.47, 0.41, 0.35, 0.28, 0.2],
             [0.85, 0.7, 0.62, 0.55, 0.49, 0.44, 0.37, 0.3, 0.22, 0.1]),
    'imbalanced': ([0.92, 0.78, 0.61, 0.4],
                   [0.83, 0.66, 0.52, 0.47, 0.43, 0.36, 0.33, 0.29, 0.25, 0.21, 0.17, 0.14, 0.11, 0.07, 0.05, 0.02]),
}

def confusion(pos, neg, t):
    """Predict 'positive' when score >= t."""
    tp = sum(s >= t for s in pos)
    fp = sum(s >= t for s in neg)
    return tp, len(pos) - tp, fp, len(neg) - fp          # TP, FN, FP, TN

def metrics(tp, fn, fp, tn):
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn)                             # = true positive rate
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
    return {'accuracy': round((tp + tn) / (tp + fn + fp + tn), 3), 'precision': round(precision, 3),
            'recall': round(recall, 3), 'F1': round(f1, 3), 'FPR': round(fp / (fp + tn), 3)}

def roc(pos, neg):
    """(FPR, TPR) for every threshold, from 'nothing is positive' to 'everything is positive'."""
    points = [(0.0, 0.0)]
    for t in sorted(set(pos + neg), reverse=True):
        tp, fn, fp, tn = confusion(pos, neg, t)
        points.append((fp / len(neg), tp / len(pos)))
    return points

def auc(points):
    """Area under the ROC curve with the trapezoid rule."""
    return sum((x2 - x1) * (y1 + y2) / 2 for (x1, y1), (x2, y2) in zip(points, points[1:]))

for name, (pos, neg) in DATA.items():
    cm = confusion(pos, neg, 0.5)
    print(name, 'TP FN FP TN =', cm, metrics(*cm), 'AUC =', round(auc(roc(pos, neg)), 3))
# good TP FN FP TN = (8, 2, 2, 8) {'accuracy': 0.8, 'precision': 0.8, 'recall': 0.8, 'F1': 0.8, 'FPR': 0.2} AUC = 0.92
# weak TP FN FP TN = (5, 5, 4, 6) {'accuracy': 0.55, 'precision': 0.556, 'recall': 0.5, 'F1': 0.526, 'FPR': 0.4} AUC = 0.55
# imbalanced TP FN FP TN = (3, 1, 3, 13) {'accuracy': 0.8, 'precision': 0.5, 'recall': 0.75, 'F1': 0.6, 'FPR': 0.188} AUC = 0.875

pos, neg = DATA['imbalanced']
print(metrics(*confusion(pos, neg, 1.01)))   # predict "negative" for everybody
# {'accuracy': 0.8, 'precision': 0.0, 'recall': 0.0, 'F1': 0.0, 'FPR': 0.0}

The same numbers come from scikit-learn: confusion_matrix, precision_score, recall_score, f1_score, roc_curve and roc_auc_score in sklearn.metrics.

Common mistakes

• Swapping precision and recall. Precision divides by everything you predicted positive; recall divides by everything that is positive.
• Reporting accuracy alone on imbalanced data. Always look at the confusion matrix.
• Thinking 0.5 is the right threshold. It is only a default. Choose it from the costs of false positives and false negatives.
• Reading AUC as accuracy. AUC measures how well the scores rank positives above negatives, at every threshold at once.
• Mixing up the axes of the ROC plot. x is the false positive rate, y is the true positive rate (recall).