Overfitting, Cross-Validation, and Hyperparameter Tuning

Step 1: Spot overfitting

Compare training and test scores. A large gap is the classic sign. An unconstrained decision tree is a good demonstration because it can memorize the training set.

from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(random_state=42).fit(X_train, y_train)
print("Train:", tree.score(X_train, y_train))   # likely 1.0
print("Test: ", tree.score(X_test, y_test))     # noticeably lower

shallow = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X_train, y_train)
print("Shallow test:", shallow.score(X_test, y_test))

Step 2: Estimate performance with cross-validation

K-fold cross-validation splits the data into k parts, trains on k-1 of them, tests on the remaining one, and repeats so every part is used for testing once. The average and spread of the scores are far more trustworthy than one split.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X, y, cv=5, scoring="f1")
print(scores.round(3))
print("Mean:", scores.mean().round(3), "Std:", scores.std().round(3))

Step 3: Tune hyperparameters with a grid search

Hyperparameters are settings you choose before training, such as tree depth. GridSearchCV tries every combination using cross-validation on the training data only, then refits the best one.

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

param_grid = {
    "n_estimators": [100, 300],
    "max_depth": [None, 5, 10],
}

grid = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid, cv=5, scoring="f1", n_jobs=-1,
)
grid.fit(X_train, y_train)

print(grid.best_params_)
print("CV score:", grid.best_score_)

Step 4: Touch the test set once

After tuning, evaluate the final model on the test set a single time. If you keep adjusting based on test results, the test set becomes part of training and your estimate becomes optimistic.

print("Final test score:", grid.score(X_test, y_test))

Step 5: Reduce overfitting

Common remedies are getting more data, simplifying the model (less depth, fewer features), adding regularization, and using ensembles. Try these in order of how cheap they are, and measure each change with cross-validation.