Why does the training score look better than the test score?

Models fit their training data more closely than new data. A large gap suggests overfitting, where the model memorized details that do not generalize. The next module shows how to measure and reduce it.

Data Science with Python Module 6: Machine Learning with scikit-learn

Introduction

Machine learning finds patterns in data that let you predict something new. In supervised learning you have examples with known answers: a number to predict (regression) or a category to predict (classification). scikit-learn gives every model the same simple interface. This module trains one of each and introduces pipelines.

Prerequisites

Module 5 completed

Step-by-Step Guide

Step 1: Split data into training and test sets

A model must be judged on data it has never seen. Hold out part of the data as a test set, and set random_state so your split is repeatable. For classification, stratify keeps class proportions the same in both parts.

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
print(X_train.shape, X_test.shape)

Step 2: Fit a regression model

Every scikit-learn model has fit to learn from training data and predict to make predictions. Linear regression finds the weights that best fit a straight-line relationship. Start with a simple model as a baseline before trying complex ones.

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score

model = LinearRegression()
model.fit(X_train, y_train)

pred = model.predict(X_test)
print("MAE:", round(mean_absolute_error(y_test, pred), 1))
print("R2 :", round(r2_score(y_test, pred), 3))

Step 3: Fit a classification model

Logistic regression predicts a category, here whether a tumor is malignant or benign. Because its solver works better with scaled inputs, put a StandardScaler in front of it. Compare accuracy with a dummy baseline that always guesses the most common class.

from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

Xc, yc = load_breast_cancer(return_X_y=True, as_frame=True)
Xc_train, Xc_test, yc_train, yc_test = train_test_split(
    Xc, yc, test_size=0.2, random_state=42, stratify=yc
)

baseline = DummyClassifier(strategy="most_frequent").fit(Xc_train, yc_train)
clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
clf.fit(Xc_train, yc_train)

print("baseline accuracy:", round(baseline.score(Xc_test, yc_test), 3))
print("model accuracy   :", round(clf.score(Xc_test, yc_test), 3))

Step 4: Why pipelines matter

A pipeline chains preprocessing steps and a model into one object. When you fit it, the scaler learns its statistics from the training data only, which prevents leaking information from the test set. It also means the exact same steps run at prediction time. Use ColumnTransformer to give numeric and text columns different preprocessing.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder

preprocess = ColumnTransformer([
    ("numbers", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()),
     ["age", "income"]),
    ("categories", OneHotEncoder(handle_unknown="ignore"), ["city"]),
])
# model = make_pipeline(preprocess, LogisticRegression(max_iter=1000))

Step 5: Try a tree-based model

Random forests combine many decision trees and often work well with little tuning and no scaling. They handle non-linear patterns and give feature importances. Keep the same interface: fit, predict, score.

from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(n_estimators=200, random_state=42)
forest.fit(Xc_train, yc_train)
print("forest accuracy:", round(forest.score(Xc_test, yc_test), 3))

importances = pd.Series(forest.feature_importances_, index=Xc.columns)
print(importances.sort_values(ascending=False).head(5))

FAQs

Why does the training score look better than the test score?: Models fit their training data more closely than new data. A large gap suggests overfitting, where the model memorized details that do not generalize. The next module shows how to measure and reduce it.

Summary

Split data into training and test sets, fit models with fit and predict, always compare against a baseline, and put preprocessing and the model in a pipeline so test data never leaks into training.