What is a good score?

It depends on the problem. Compare against a simple baseline and against what the business needs. A 95 percent accurate model is useless if 95 percent of cases belong to one class.

Data Science with Python Module 7: Evaluating and Improving Models

Introduction

A single accuracy number can mislead. This module shows how to pick metrics that match the problem, estimate performance reliably with cross-validation, tune a model with a search, and recognize the mistakes that make a model look better than it is.

Prerequisites

Module 6 completed

Step-by-Step Guide

Step 1: Pick metrics that fit the problem

For regression, MAE is the average error in the original units, RMSE punishes large errors more, and R squared is the share of variation explained. For classification, accuracy is misleading when classes are imbalanced. Precision asks how many predicted positives were correct, recall asks how many real positives were found, and F1 combines them. Choose based on which mistake costs more.

from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score

pred = clf.predict(Xc_test)
print(confusion_matrix(yc_test, pred))
print(classification_report(yc_test, pred))
print("ROC AUC:", round(roc_auc_score(yc_test, clf.predict_proba(Xc_test)[:, 1]), 3))

Step 2: Use cross-validation

One train and test split can be lucky or unlucky. Cross-validation splits the training data into several folds, trains on all but one fold, scores on the held-out fold, and repeats. The mean and spread of the scores give a more trustworthy estimate.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(clf, Xc_train, yc_train, cv=5, scoring="f1")
print(scores.round(3), "mean:", scores.mean().round(3), "std:", scores.std().round(3))

Step 3: Tune hyperparameters with a search

Hyperparameters are settings you choose before training, such as tree depth. GridSearchCV tries combinations and scores each with cross-validation. Tune on the training data only, then check the best model once on the test set at the end.

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid={"n_estimators": [100, 300], "max_depth": [None, 5, 10]},
    cv=5, scoring="f1", n_jobs=-1,
)
search.fit(Xc_train, yc_train)
print(search.best_params_, round(search.best_score_, 3))
print("test F1:", round(search.score(Xc_test, yc_test), 3))

Step 4: Recognize overfitting and underfitting

An overfit model scores far better on training data than on new data. An underfit model is poor on both. Remedies for overfitting include more data, a simpler model, stronger regularization, fewer features, and limiting tree depth. Remedies for underfitting include more useful features and a more flexible model.

from sklearn.tree import DecisionTreeClassifier

for depth in [1, 3, 6, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(Xc_train, yc_train)
    print(depth, round(tree.score(Xc_train, yc_train), 3), round(tree.score(Xc_test, yc_test), 3))

Step 5: Watch for data leakage

Leakage means information from outside the training data sneaks into the model, making results look unrealistically good. Typical causes: scaling or imputing before splitting, using a feature that is only known after the outcome, and the same customer appearing in both training and test sets. Keep every preprocessing step inside a pipeline, split before any learning, and ask of every feature: would I know this at prediction time?

# Wrong: the scaler sees the test data
# X_scaled = StandardScaler().fit_transform(X)
# X_train, X_test = train_test_split(X_scaled)

# Right: split first, then fit the pipeline on training data only
# model = make_pipeline(StandardScaler(), LogisticRegression())
# model.fit(X_train, y_train)

FAQs

What is a good score?: It depends on the problem. Compare against a simple baseline and against what the business needs. A 95 percent accurate model is useless if 95 percent of cases belong to one class.

Summary

Match metrics to the cost of mistakes, estimate performance with cross-validation, tune on training data, compare training and test scores to spot overfitting, and prevent leakage by splitting first and using pipelines.