Why do we need a separate test set?

A model scores very well on data it trained on even if it learned nothing general. The test set simulates new data and gives an honest estimate of real-world performance.

Why set random_state?

Splits and many algorithms involve randomness. Fixing the seed makes your results repeatable so you can compare changes fairly.

Train/Test Split and Your First Model with scikit-learn

Introduction

A model must be judged on data it has never seen, otherwise you only measure how well it memorized. This lesson splits a dataset into training and test sets, trains a random forest classifier on scikit-learn's built-in breast cancer dataset, and checks its accuracy.

Prerequisites

Prepared features X and target y
scikit-learn installed

Step-by-Step Guide

Step 1: Load a dataset

scikit-learn ships with small datasets so you can practice without downloading files. This one has 30 numeric measurements and a binary label.

from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
print(X.shape, y.shape)   # (569, 30) (569,)

Step 2: Split into training and test sets

Hold out part of the data for testing. random_state makes the split repeatable, and stratify keeps the class proportions the same in both parts.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

Step 3: Train a model

Every scikit-learn model follows the same pattern: create it, fit it on the training data, then predict. Random forests need no feature scaling, which keeps this first example simple.

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=200, random_state=42)
model.fit(X_train, y_train)

Step 4: Evaluate on the test set

score returns accuracy for classifiers. Compare it with the training score: a big gap means the model memorized the training data.

print("Train accuracy:", model.score(X_train, y_train))
print("Test accuracy: ", model.score(X_test, y_test))

Step 5: Predict on new data

predict returns the class for each row. predict_proba returns the model's confidence for each class, which is often more useful than a hard label.

print(model.predict(X_test[:5]))
print(model.predict_proba(X_test[:5]).round(2))

FAQs

Why do we need a separate test set?: A model scores very well on data it trained on even if it learned nothing general. The test set simulates new data and gives an honest estimate of real-world performance.
Why set random_state?: Splits and many algorithms involve randomness. Fixing the seed makes your results repeatable so you can compare changes fairly.

Summary

Split first, train on the training portion only, and judge the model on the held-out test portion. The fit, predict pattern works the same way across scikit-learn models.