Which metric should I optimize?

The one that reflects the real cost of mistakes in your application. Write it down before modeling. For imbalanced data, prefer precision, recall, F1, or PR-AUC over plain accuracy.

What is a good ROC-AUC?

It depends on the problem. Compare against your baseline and what the business needs; values above 0.9 are strong in many settings, but a high score on leaked or unrealistic data is meaningless.

Evaluating Machine Learning Models: Accuracy, Precision, Recall, and More

Introduction

Accuracy alone can be dangerously misleading. If only 1% of transactions are fraud, a model that always says not-fraud is 99% accurate and useless. This lesson covers the metrics that actually tell you whether a model is good, for both classification and regression.

Prerequisites

A trained scikit-learn classifier
The train/test split lesson

Step-by-Step Guide

Step 1: Read a confusion matrix

A confusion matrix counts true negatives, false positives, false negatives, and true positives. In scikit-learn the rows are the true classes and the columns are the predicted classes.

from sklearn.metrics import confusion_matrix

y_pred = model.predict(X_test)
print(confusion_matrix(y_test, y_pred))
# [[TN FP]
#  [FN TP]]

Step 2: Understand precision, recall, and F1

Precision answers: of everything predicted positive, how much really was? Recall answers: of all real positives, how many did we catch? F1 is their harmonic mean. Choose by the cost of errors: favor recall when missing a case is costly (disease screening), and precision when false alarms are costly (spam filtering).

from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred))

Step 3: Use ROC-AUC for ranking quality

ROC-AUC measures how well the model ranks positives above negatives across all thresholds. It needs predicted probabilities, not labels. A value of 0.5 is random guessing and 1.0 is perfect.

from sklearn.metrics import roc_auc_score

y_proba = model.predict_proba(X_test)[:, 1]
print("ROC-AUC:", roc_auc_score(y_test, y_proba))

Step 4: Evaluate regression models

For predicting numbers, use mean absolute error (average size of the error), root mean squared error (penalizes big misses more), and R-squared (share of variance explained).

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

pred = regressor.predict(X_test)
print("MAE :", mean_absolute_error(y_test, pred))
print("RMSE:", mean_squared_error(y_test, pred) ** 0.5)
print("R2  :", r2_score(y_test, pred))

Step 5: Compare against a baseline

Always compare your model with a trivial baseline, such as predicting the most common class or the average value. If you cannot beat the baseline by a meaningful margin, the model is not adding value.

from sklearn.dummy import DummyClassifier

baseline = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
print("Baseline accuracy:", baseline.score(X_test, y_test))

FAQs

Which metric should I optimize?: The one that reflects the real cost of mistakes in your application. Write it down before modeling. For imbalanced data, prefer precision, recall, F1, or PR-AUC over plain accuracy.
What is a good ROC-AUC?: It depends on the problem. Compare against your baseline and what the business needs; values above 0.9 are strong in many settings, but a high score on leaked or unrealistic data is meaningless.

Summary

Pick metrics that match the cost of errors, read the confusion matrix, use probabilities for ROC-AUC, and always compare with a baseline. A single accuracy number rarely tells the whole story.