One hundred transactions, one of them fraudulent. Two models, and the number that cannot tell them apart.
actual = [1] + [0] * 99 # one fraud in a hundred lazy = [0] * 100 # predicts 'no' every time # 99 of 100 right -> accuracy 0.99, recall 0.0 useful = [1] + [0] * 98 + [1] # catches it, one false alarm # 99 of 100 right -> accuracy 0.99, recall 1.0 # Identical accuracy. One model is worthless and the other works. # # The baseline is the tell: always guessing the majority class also # scores 0.99, so 0.99 is the floor, not an achievement.
- Accuracy weights every row equally, and the rows are not equal
It counts the 99 easy correct answers alongside the one that mattered, so the rare class contributes one per cent of the score and a hundred per cent of the value. That is not a flaw in the arithmetic — accuracy is exactly the right metric when the classes are balanced and the two kinds of mistake cost the same. Fraud, disease and defects are none of those.
- Always compare against the majority-class baseline
The cheapest possible model — predict whatever is commonest and ignore the input — has an accuracy equal to the share of the majority class. If your model is not clearly beating that, it has learned nothing, and reporting the baseline next to the accuracy makes it impossible to mistake one for the other. It costs one line and it would have caught this in the first week.
- Recall is the number the fraud team actually wanted
'Of the fraud that happened, how much did we catch?' has nothing to do with the 99 honest transactions, so it cannot be inflated by them. The lazy model scores 0.0 and the useful one 1.0 — which is the distinction accuracy could not make. Adding precision alongside it stops the opposite failure, a model that flags everything.
The accuracy calculation is correct — do not change it. What is missing is everything that would have shown the model was not working: how much of the fraud it caught, and what score a model that does nothing would get.
Count the true positives, false positives and false negatives as in the confusion-matrix exercise. Recall is tp / (tp + fn), precision is tp / (tp + fp), and each needs its own zero guard.
The baseline is the share of the commonest label in `actual` — count how many are positive, take max(positives, total - positives), and divide by the total. Note it is computed from the actual labels only: it is a property of the data, not of the model.
def evaluate(actual, predicted): if len(actual) != len(predicted): raise ValueError('actual and predicted must be the same length') total = len(actual) correct = sum(1 for a, p in zip(actual, predicted) if a == p) tp = sum(1 for a, p in zip(actual, predicted) if a == 1 and p == 1) fp = sum(1 for a, p in zip(actual, predicted) if a != 1 and p == 1) fn = sum(1 for a, p in zip(actual, predicted) if a == 1 and p != 1) positives = sum(1 for a in actual if a == 1) majority = ____(positives, total - positives) return { 'accuracy': correct / total if total else 0.0, 'recall': tp / (tp + ____) if tp + fn else 0.0, 'precision': tp / (tp + ____) if tp + fp else 0.0, 'baseline': majority / total if ____ else 0.0, }
def evaluate(actual, predicted):
if len(actual) != len(predicted):
raise ValueError(
f"actual has {len(actual)} labels, predicted has {len(predicted)}"
)
total = len(actual)
correct = sum(1 for a, p in zip(actual, predicted) if a == p)
true_positive = sum(
1 for a, p in zip(actual, predicted) if a == 1 and p == 1
)
false_positive = sum(
1 for a, p in zip(actual, predicted) if a != 1 and p == 1
)
false_negative = sum(
1 for a, p in zip(actual, predicted) if a == 1 and p != 1
)
# What the laziest possible model would score: always guess the
# commonest label. Anything not clearly above this has learned nothing.
positives = sum(1 for a in actual if a == 1)
majority = max(positives, total - positives)
found = true_positive + false_negative
flagged = true_positive + false_positive
return {
"accuracy": correct / total if total else 0.0,
"recall": true_positive / found if found else 0.0,
"precision": true_positive / flagged if flagged else 0.0,
"baseline": majority / total if total else 0.0,
}
It is unchanged, and it was right all along — the fix adds context rather than correcting arithmetic. That is worth sitting with: the most expensive bugs in data work are frequently not wrong calculations but correct calculations answering a question nobody meant to ask, and no amount of unit testing the formula would have found this one.
It never looks at the predictions, because it describes the data: how skewed is this problem, and therefore how high is the floor? Reporting it beside the accuracy turns 'we scored 0.99' into 'we scored 0.99 against a floor of 0.99', which is a sentence nobody can misread. max() rather than assuming the negatives dominate, because the majority class is whichever one it is.
Accuracy is fooled by imbalance, recall is fooled by a model that flags everything, precision is fooled by one lucky cautious prediction. Together they cannot all be high for a model that is not working — which is the only real argument for reporting several numbers instead of one.
None of this is a reason to stop reporting accuracy: on a balanced problem with symmetric costs it is the clearest single number there is. The rule is to know your class balance before you choose, and to report the baseline whenever it is not 50%. For a genuinely rare event, the better headline metrics are the area under the precision-recall curve or, if you want one number that cannot be gamed by predicting one class, Matthews correlation — sklearn has both, along with balanced_accuracy_score, which averages the recall of each class and gives the lazy model exactly 0.5.
- Add balanced accuracy — the mean of the recall on each class — and check that the lazy model scores 0.5 rather than 0.99.
- Report the class balance itself, so the shape of the problem is visible without inferring it from the baseline.
- Take a positive label argument as the confusion-matrix exercise does, then try it on a dataset where the rare class is labelled 'fraud' rather than 1.