Evaluation Metrics

Why accuracy is not enough

Accuracy is the fraction of predictions that are correct. It is easy to compute and easy to misread.

Suppose 10 of 100 emails are spam. A "model" that always says "not spam" is right 90 times out of 100. Accuracy 90%, and it catches nothing.

import numpy as np
y_true = np.array([1]*10 + [0]*90)
y_pred = np.zeros(100, dtype=int)         # always predict "negative"
print((y_true == y_pred).mean())

0.9

This is class imbalance: one class is much rarer than the other. Fraud, disease and defects are all rare, and accuracy hides failure on the class you care about.