Python Pandas Tutorial

Load, inspect, select, filter, group and summarise tabular data with pandas DataFrames, and treat missing values deliberately.

Lesson 76 of 107Advanced12 min read

pandas is Python's main library for working with tables of data. Its DataFrame is like a spreadsheet in code: labelled columns, labelled rows, and fast operations on whole columns. It reads CSV, Excel, JSON and SQL, and makes filtering, grouping and summarising a few lines of code:

Python
import pandas as pd
sales = pd.read_csv("market_sales.csv")
print(sales.groupby("zone")["units"].sum())

Install it with python -m pip install pandas. This tutorial covers the DataFrame model, selecting with loc and iloc, filtering rows, grouping and aggregating, and handling missing values. The examples analyse sales at a weekend market. pandas is built on NumPy, and this lesson uses pandas 3.0 behaviour, noting where older versions differ.

Table model

A DataFrame is a table of columns; each column is a Series, a one-dimensional labelled array of one type. Both have an index, the row labels:

Python
import pandas as pd
stalls = pd.DataFrame(
    {"stall": ["Herbs", "Bread", "Honey", "Cheese"],
     "zone": ["east", "west", "east", "west"],
     "sales": [120.0, None, 85.5, 210.0]},
    index=[101, 102, 103, 104],
)
print(stalls.shape)          # (4, 3)
print(stalls.dtypes)         # stall and zone: str, sales: float64
stalls.info()                # columns, non-null counts and types

Start every analysis by inspecting the data: head(), shape, dtypes, info() and describe() reveal wrong types, unexpected blanks and odd values before they distort results. In pandas 3.0 text columns get a dedicated str dtype; earlier versions show object.

Row labels aren't necessarily positions. Here the index is the stall number, so label 101 is the first row. When you read a CSV without specifying an index, pandas uses 0, 1, 2..., which makes labels and positions coincide, until you filter or sort and they no longer do.

Selection

There are three main ways to select data:

SyntaxSelects byExample
df["col"] / df[["a", "b"]]column name(s)stalls["sales"]
df.loc[rows, cols]labelsstalls.loc[103, "stall"] → "Honey"
df.iloc[rows, cols]integer positionsstalls.iloc[0, 0] → "Herbs"

Filtering uses a boolean Series as a mask:

Python
big = stalls[stalls["sales"] > 100]
east_big = stalls.loc[(stalls["zone"] == "east") & (stalls["sales"] > 100), ["stall", "sales"]]

Combine conditions with & and |, each in brackets, not with and/or. .isin(["east", "north"]) tests membership, and .query("sales > 100") is a readable alternative.

To change values, use one loc call: stalls.loc[stalls["zone"] == "east", "sales"] *= 1.1. Chained assignment such as stalls[mask]["sales"] = 0 never modifies the original in pandas 3.0, because of copy-on-write; it raises a warning instead.

Aggregation

groupby splits rows into groups by one or more columns, applies a calculation to each group, and combines the results:

Python
by_zone = stalls.groupby("zone")["sales"].sum()
summary = stalls.groupby("zone")["sales"].agg(["sum", "mean", "count"])
Text
        sum    mean  count
zone
east  205.5  102.75      2
west  210.0  210.00      1

Choose the aggregation that answers the business question. Total sales per zone uses sum; a typical stall's sales uses mean or median; how many stalls reported uses count, which excludes missing values, while size() counts every row. In the example, west's mean of 210 is based on one stall, because Bread's sales are missing, which is exactly the kind of detail a report should state.

Other summaries: value_counts() counts categories, pivot_table() builds cross-tabulations, and sort_values() ranks results.

Missing values

pandas represents missing data as NaN (or NA). Detect it before deciding what it means:

Python
print(stalls["sales"].isna().sum())       # 1 missing value
print(stalls["sales"].mean())             # 138.5: missing values are skipped
print(stalls["sales"].fillna(0).mean())   # 103.875: treats missing as zero

The two means differ because they encode different assumptions. A missing sales figure might mean the stall sold nothing, didn't trade that day, or simply didn't report. Only the first justifies filling with 0. Options include:

  • dropna() to exclude incomplete rows, when they genuinely don't belong in the analysis;
  • fillna(value) when there's a defensible default;
  • keeping the gap and reporting how many values were missing.

Write the policy down next to the analysis. Aggregations such as sum and mean skip missing values by default, which is usually right, but means a total can silently cover fewer rows than you think. The data cleaning lesson goes further.

Independent practice and a worked check

Task: create a market report with typed columns, a row filter, grouped totals and a documented missing value policy.

Python
import pandas as pd

df = pd.DataFrame({
    "stall": ["Herbs", "Bread", "Honey", "Cheese", "Soap"],
    "zone": ["east", "west", "east", "west", "east"],
    "units": [12, 30, None, 8, 15],
})
print(df.dtypes["units"])                    # float64: the missing value forces floats

policy = "Rows with missing units are excluded and counted, not treated as zero."
known = df.dropna(subset=["units"])
busy = known[known["units"] >= 10]
print(busy.groupby("zone")["units"].sum().to_dict())   # {'east': 27.0, 'west': 30.0}
print(f"Excluded: {len(df) - len(known)} row(s). {policy}")

For the worked check, the selector that uses integer positions is iloc; loc uses labels.

Reference and next practice

The pandas documentation's 10 minutes to pandas and indexing guide are good next steps. Continue with the Matplotlib tutorial and data cleaning. Practise with the Python exercises and Python quiz.

Pandas Tutorial Summary

  • Table model: A DataFrame organizes labeled columns and rows while a Series represents one labeled sequence.
  • Selection: loc selects by labels and iloc selects by integer positions.
  • Aggregation: groupby partitions rows into groups before aggregation.
  • Missing values: Missing data needs a deliberate interpretation before dropping or filling it.

Pandas Tutorial Examples

Each example shows the code, the output it produces and what each part does.

Example 1

Pandas Tutorial in a small program

Python
import pandas as pd
sales = pd.DataFrame({"zone": ["east", "west", "east"], "units": [6, 9, 4]})
print(sales.groupby("zone")["units"].sum().to_dict())
Output
{'east': 10, 'west': 9}
Line-by-line explanation
  1. Line 2A DataFrame from a dictionary of columns.
  2. Line 3Group by zone, select the units column, sum each group and convert to a plain dict.
Example 2

A market report from CSV text

Python
import io
import pandas as pd

csv_text = """stall,zone,units,price
Herbs,east,12,2.50
Bread,west,30,3.20
Honey,east,,6.00
Cheese,west,8,7.50
Soap,east,15,4.00
"""
sales = pd.read_csv(io.StringIO(csv_text))
print(sales.shape, sales["units"].isna().sum())

sales["revenue"] = sales["units"] * sales["price"]
report = (sales.dropna(subset=["units"])
               .groupby("zone")
               .agg(stalls=("stall", "count"), revenue=("revenue", "sum"))
               .sort_values("revenue", ascending=False))
print(report)
print("Top stall:", sales.loc[sales["revenue"].idxmax(), "stall"])
print("Positions 0-1:", sales.iloc[0:2]["stall"].tolist())
Output
(5, 4) 1
      stalls  revenue
zone
west       2    156.0
east       2     90.0
Top stall: Bread
Positions 0-1: ['Herbs', 'Bread']
Line-by-line explanation
  1. Line 11read_csv accepts a file path or any file-like object; the empty cell becomes NaN.
  2. Line 14Column arithmetic is vectorized; Honey's revenue is NaN because its units are missing.
  3. Lines 15 to 18Rows without units are excluded explicitly, then named aggregations build the report.
  4. Line 20idxmax() returns the index label of the largest revenue, which loc then uses.

Common Pandas Tutorial Mistakes

Mixing up loc and iloc after filtering

Don't
east = stalls[stalls["zone"] == "east"]   # index labels 101 and 103
print(east.loc[0])                        # KeyError: there is no label 0
Do
print(east.iloc[0])                       # first row by position
print(east.loc[101])                      # row with label 101

Filling missing sales with zero by default

Don't
stalls["sales"] = stalls["sales"].fillna(0)   # unreported becomes "sold nothing"
Do
missing = stalls["sales"].isna().sum()
average = stalls["sales"].mean()              # skips missing values
print(f"Average of reporting stalls: {average:.2f} ({missing} not reported)")

Pandas Tutorial in Real Projects

pandas in data work

Analysts use pandas to load exports from spreadsheets, databases and APIs, clean them, join tables (merge), compute summaries and feed charts or machine-learning models. Reports that once took a morning in a spreadsheet become a script that runs in seconds and gives the same answer every time.

Jupyter notebooks are the usual environment for exploration, with the finished analysis moved into scripts or pipelines. For very large data, libraries with a similar API, such as Polars, DuckDB and Dask, handle more rows or run faster, and the concepts from this lesson transfer directly.

Pandas Tutorial FAQ

What is the difference between loc and iloc in pandas?

loc selects by labels, such as index values and column names. iloc selects by integer positions, starting at 0. After filtering or sorting, labels and positions often differ, so pick the one that matches your intent.

What is the difference between a DataFrame and a Series?

A Series is one labelled column of values. A DataFrame is a table of several Series sharing the same row index. Selecting one column, df["sales"], returns a Series.

How do I read a CSV file with pandas?

df = pd.read_csv("file.csv"). Useful options include sep=";" for other delimiters, parse_dates=["date"] for date columns, dtype={"code": str} to keep leading zeros, and encoding="utf-8-sig" for Excel exports.

Why does my integer column become float in pandas?

NumPy integer columns can't hold NaN, so a column with missing values is stored as float64. Use the nullable integer type, df["units"].astype("Int64"), if you need integers with missing values.

What is SettingWithCopyWarning or ChainedAssignmentError?

They warn that you assigned to a temporary copy, as in df[df.x > 1]["y"] = 0, which doesn't change df. Use a single loc call instead: df.loc[df.x > 1, "y"] = 0. pandas 3.0 uses copy-on-write, so chained assignment never updates the original.

Pandas Tutorial Exercises

Answer each one and check it instantly. If you get stuck, open the hint before the answer.

Multiple choiceeasy

Which selector uses integer positions?

Multiple choicemedium

Which review concern is valid for pandas tutorial?

Pandas Tutorial Quiz

2 questions to check what you learned.

  1. Which selector uses integer positions?

    • loc
    • iloc
    • Python ignores the distinction

  2. Which statement correctly describes aggregation?

    • groupby partitions rows into groups before aggregation.
    • Only the example output matters, not the rule
    • Every operation has the same behavior

Related Python topics

75NumPy TutorialWork with NumPy arrays: shapes, vectorized operations, reductions along axes, filtering, broadcasting and the difference between views and copies.78Data Cleaning and AnalysisTurn messy records into trustworthy data: validate against a contract, convert types, handle gaps and duplicates, report rejected rows and reconcile counts.77Matplotlib TutorialCreate clear charts with Matplotlib's Figure and Axes, label them properly, combine several plots and save them as image files.49CSV Handling in DepthRead and write CSV files correctly with csv.reader, DictReader and DictWriter, including quoting, encodings and type conversion.
Esc
↑↓ to navigateEnter to open