Should I fill missing values with the mean?

Sometimes, but it is not always right. The median resists outliers, a special category such as Unknown suits text, and for some models a missing-value flag column helps. The right choice depends on why the data is missing, so record your decision.

Data Science with Python Module 3: Loading and Cleaning Data

Introduction

Real data is messy: values are missing, numbers arrive as text, dates are inconsistent, and rows repeat. Cleaning is where most data science time goes, and good cleaning protects every later result. This module builds a messy table in code and cleans it step by step.

Prerequisites

Module 2 completed

Step-by-Step Guide

Step 1: Read data from files

pandas reads many formats. Check the first rows, the types, and the count of missing values straight after loading. For large files, read only the columns you need.

import pandas as pd

# df = pd.read_csv("orders.csv", parse_dates=["order_date"])
# df = pd.read_excel("orders.xlsx", sheet_name="Sheet1")
# df = pd.read_json("orders.json")
# df = pd.read_parquet("orders.parquet")
# df.to_csv("clean_orders.csv", index=False)

Step 2: Build a messy practice table

This table has the usual problems: a missing value, a duplicate row, numbers stored as text, inconsistent capitalization and spaces, and an impossible age.

import pandas as pd
import numpy as np

raw = pd.DataFrame({
    "customer": ["Asha ", "ravi", "Asha ", "Meera", "John", None],
    "age": ["34", "29", "34", "41", "290", "35"],
    "city": ["pune", "Delhi", "pune", "MUMBAI", "Delhi", "Pune"],
    "amount": ["1,200", "850", "1,200", "N/A", "430", "999"],
    "order_date": ["2026-01-05", "2026-01-07", "2026-01-05", "2026-02-11", "bad date", "2026-03-02"],
})
raw

Step 3: Find the problems first

Look before you change anything. Count missing values, find duplicates, and check the types. Keep the raw table untouched and work on a copy, so you can always compare.

df = raw.copy()

print(df.isna().sum())            # missing values per column
print(df.duplicated().sum())      # duplicate rows
print(df.dtypes)                  # everything is text

Step 4: Fix text, types, and dates

Standardize text with the str methods. Convert numbers with to_numeric, using errors set to coerce so bad values become NaN instead of raising an error, and do the same for dates.

df["customer"] = df["customer"].str.strip().str.title()
df["city"] = df["city"].str.strip().str.title()

df["amount"] = pd.to_numeric(df["amount"].str.replace(",", ""), errors="coerce")
df["age"] = pd.to_numeric(df["age"], errors="coerce")
df["order_date"] = pd.to_datetime(df["order_date"], errors="coerce")

Step 5: Handle duplicates, missing values, and outliers

Remove exact duplicates. For missing values, decide deliberately: drop the row, fill with a sensible value such as the median, or keep it as missing. Treat impossible values such as an age of 290 as errors and mark them missing. Write down every choice, because it changes your results.

df = df.drop_duplicates()

df.loc[~df["age"].between(0, 120), "age"] = np.nan    # impossible ages
df["age"] = df["age"].fillna(df["age"].median())
df["amount"] = df["amount"].fillna(df["amount"].median())
df = df.dropna(subset=["customer", "order_date"])      # rows we cannot use

df.info()
df

FAQs

Should I fill missing values with the mean?: Sometimes, but it is not always right. The median resists outliers, a special category such as Unknown suits text, and for some models a missing-value flag column helps. The right choice depends on why the data is missing, so record your decision.

Summary

Inspect first, then standardize text, convert types with errors coerced, remove duplicates, and handle missing and impossible values deliberately. Keep the raw data untouched and record every cleaning decision.