What is a good p-value cutoff?

The 0.05 cutoff is a convention, not a law. Choose it before the analysis, consider the cost of being wrong, and always look at how large and how uncertain the effect is.

Data Science with Python Module 5: Statistics for Data Science

Introduction

Statistics tells you how much to trust what you see in data. A difference between two groups might be real, or it might be luck in a small sample. This module covers the ideas you need most, with short SciPy examples built on generated data.

Prerequisites

Module 4 completed

Step-by-Step Guide

Step 1: Describe center and spread

The mean is the average, the median is the middle value, and the standard deviation measures typical distance from the mean. The median is more robust when there are extreme values. Report a measure of center together with a measure of spread.

import numpy as np

rng = np.random.default_rng(0)
income = np.append(rng.normal(50, 8, 200), [400, 500])    # two extreme values

print("mean  ", income.mean().round(1))
print("median", np.median(income).round(1))
print("std   ", income.std(ddof=1).round(1))

Step 2: Understand distributions and sampling

Many natural measurements follow a bell-shaped normal distribution. A sample is a small part of a larger population, and different samples give slightly different means. The standard error, the standard deviation divided by the square root of the sample size, describes how much a sample mean varies.

sample = rng.normal(loc=170, scale=10, size=100)    # heights in cm
mean = sample.mean()
se = sample.std(ddof=1) / np.sqrt(len(sample))
print(round(mean, 1), round(se, 2))

Step 3: Build a confidence interval

A 95 percent confidence interval is a range built so that, if you repeated the study many times, about 95 percent of such ranges would contain the true value. It expresses uncertainty better than a single number.

from scipy import stats

low, high = stats.t.interval(
    0.95, df=len(sample) - 1, loc=sample.mean(), scale=stats.sem(sample)
)
print(round(low, 1), round(high, 1))

Step 4: Run a hypothesis test

A hypothesis test asks whether a difference could easily happen by chance. The null hypothesis says there is no difference. The p-value is the probability of seeing a result at least this extreme if the null hypothesis were true. A small p-value, commonly below 0.05, is evidence against it, but it does not give the probability that you are right and it ignores how big the difference is.

group_a = rng.normal(100, 15, 80)     # for example, page load time with design A
group_b = rng.normal(95, 15, 80)      # design B

t_stat, p_value = stats.ttest_ind(group_a, group_b, equal_var=False)
print(round(t_stat, 2), round(p_value, 4))
print("difference in means:", round(group_a.mean() - group_b.mean(), 1))

Step 5: Avoid common mistakes

Correlation is not causation, because a third factor may drive both variables. Testing many hypotheses without adjustment will produce false positives by luck. Small samples give wide uncertainty. Always report effect sizes and intervals, not only p-values, and decide your questions before looking at the data.

# A simple effect size: Cohen's d for the two groups above
pooled_sd = np.sqrt((group_a.var(ddof=1) + group_b.var(ddof=1)) / 2)
print("Cohen's d:", round((group_a.mean() - group_b.mean()) / pooled_sd, 2))

FAQs

What is a good p-value cutoff?: The 0.05 cutoff is a convention, not a law. Choose it before the analysis, consider the cost of being wrong, and always look at how large and how uncertain the effect is.

Summary

Report center and spread, remember that samples vary, use confidence intervals to express uncertainty, read p-values carefully together with effect sizes, and never treat correlation as causation.