Why did I get a SettingWithCopyWarning?

It appears when you assign into something that might be a copy of a slice. Use loc to select and assign in one step, for example df.loc[mask, 'col'] = value, or call .copy() on a slice you want to change independently.

Data Science with Python Module 2: NumPy and pandas Basics

Introduction

pandas is the main tool for working with tabular data in Python, and it is built on NumPy arrays. This module shows how to create, select, filter, and summarize data, the operations you will use in every project.

Prerequisites

Module 1 completed
A running Jupyter notebook

Step-by-Step Guide

Step 1: Use NumPy arrays

A NumPy array holds numbers of one type and does math on the whole array at once, which is much faster than a Python loop. This is called vectorization.

import numpy as np

prices = np.array([2.5, 3.0, 4.0, 3.5])
units = np.array([10, 7, 5, 9])

revenue = prices * units          # element by element
print(revenue, revenue.sum(), revenue.mean())
print(np.arange(0, 10, 2))        # [0 2 4 6 8]
print(np.random.default_rng(42).normal(size=3))

Step 2: Create Series and DataFrames

A Series is one labeled column. A DataFrame is a table of columns that can have different types. Check its shape, types, and first rows after loading or building any table.

import pandas as pd

df = pd.DataFrame({
    "city": ["Pune", "Delhi", "Mumbai", "Pune", "Delhi"],
    "sales": [250, 400, 380, 270, 410],
    "month": ["Jan", "Jan", "Jan", "Feb", "Feb"],
})

print(df.shape)       # (5, 3)
print(df.dtypes)
print(df.head(3))
df.info()

Step 3: Select columns and rows

Use square brackets for columns, loc for labels and conditions, and iloc for positions. Prefer loc with a boolean condition for filtering, and combine conditions with & and | inside parentheses.

df["sales"]                          # one column (Series)
df[["city", "sales"]]                # several columns (DataFrame)
df.loc[df["sales"] > 300]            # filter rows
df.loc[(df["city"] == "Pune") & (df["month"] == "Feb"), "sales"]
df.iloc[0:2, 0:2]                    # by position

Step 4: Add columns and sort

Assign to a new column name to create it. Use sort_values to order rows and nlargest to get the top rows quickly. Most pandas methods return a new object instead of changing the original, so assign the result back if you want to keep it.

df["sales_k"] = df["sales"] / 1000
df["big"] = df["sales"] >= 380

df.sort_values("sales", ascending=False)
df.nlargest(2, "sales")

Step 5: Summarize with groupby

groupby splits the table into groups, applies a calculation to each, and combines the results: the split-apply-combine pattern. agg lets you calculate several statistics at once.

df.groupby("city")["sales"].sum()

df.groupby("city").agg(
    total_sales=("sales", "sum"),
    average_sales=("sales", "mean"),
    months=("month", "nunique"),
)

df.pivot_table(index="city", columns="month", values="sales", aggfunc="sum")

FAQs

Why did I get a SettingWithCopyWarning?: It appears when you assign into something that might be a copy of a slice. Use loc to select and assign in one step, for example df.loc[mask, 'col'] = value, or call .copy() on a slice you want to change independently.

Summary

NumPy does fast math on arrays, and pandas organizes data in DataFrames. Select with brackets, loc, and iloc, add columns, sort, and summarize with groupby, agg, and pivot_table.