Working with Missing Data

The Missing Data Problem

Missing values cause bugs, return unexpected results, and trip up even experienced analysts. But once you understand how pandas handles them, you can avoid the pitfalls.

What NaN Actually Means

NaN stands for "Not a Number," but it really means "missing" or "unknown." It’s not zero. It’s not an empty string. It’s the absence of a value.

Think of it this way:

• 0 — the value is zero
• "" (empty string) — the value is blank text
• NaN — we don’t know what the value is

This distinction matters because NaN behaves differently from actual values.

A Real Example

Let’s look at the fintech_app_users table:

Some users haven’t provided their phone numbers — those values show up as NaN. We don’t know their phone numbers. Maybe they never entered one. Maybe the data was lost. The point is: we don’t have a value.

Finding Missing Values: .isna()

Here’s the first surprise: you can’t use == to check for NaN. NaN == NaN evaluates to False in Python, so equality checks won’t find missing values. Instead, use .isna():

fintech_app_users[fintech_app_users["phone_number"].isna()]

.isna() returns True for every row where the value is missing, and False everywhere else.

Users Missing Phone Numbers

Excluding Missing Values: .notna()

To find rows where a value exists, use .notna() — the opposite of .isna():

This returns only rows where phone_number has a non-null value.

Wine Varieties Tasted by Roger Voss

NaN in Comparisons

NaN makes comparisons tricky. Any comparison with NaN returns False:

np.nan == np.nan    # False (not True!)
np.nan != np.nan    # True
np.nan > 5          # False
np.nan == "text"    # False

This means rows with NaN values are excluded from most filters, even when you might not expect it.

NaN Disappears from Both Sides of a Filter

# Filtering with == excludes NaN automatically
fintech_app_users[fintech_app_users["status"] == "active"]

Counting Missing Values

Before cleaning data, it helps to know how much is missing. A few useful patterns:

# Count NaN values per column
fintech_app_users.isna().sum()

# Percentage missing per column
fintech_app_users.isna().mean() * 100

# Total rows with any NaN
fintech_app_users.isna().any(axis=1).sum()

.isna().sum() counts missing values in each column. .isna().mean() gives you the proportion. These are the first things to run when exploring a new dataset.

Handling Missing Values with .fillna()

# Replace NaN in a specific column and keep the DataFrame
fintech_app_users["phone_number"] = (
    fintech_app_users["phone_number"].fillna("No phone")
)

This overwrites the column in place. You can also fill different columns with different defaults:

# Fill different columns with different values
df.fillna({"phone_number": "No phone", "email": "unknown@example.com"})

Dropping Missing Values with .dropna()

Sometimes missing data makes a row useless. .dropna() removes rows that contain any NaN:

# Drop rows where ANY column is NaN
fintech_app_users.dropna()

# Drop rows where a SPECIFIC column is NaN
fintech_app_users.dropna(subset=["phone_number"])

The subset parameter is important — without it, .dropna() removes any row that has a NaN in ANY column, which can be too aggressive.

Removing Incomplete Rows

Common Mistakes

Using == to check for NaN

# Wrong: equality check doesn't find NaN
fintech_app_users[fintech_app_users["phone_number"] == None]

# Correct: use .isna()
fintech_app_users[fintech_app_users["phone_number"].isna()]

Assuming an empty string equals NaN

# These are different:
df[df["name"] == ""]      # matches empty strings
df[df["name"].isna()]     # matches NaN/None

An empty string "" is a value. NaN is the absence of a value. They require different checks.

Forgetting that NaN propagates through math

# Any arithmetic with NaN returns NaN
# 5 + NaN = NaN
# NaN * 100 = NaN

# Use .fillna(0) before math if NaN should be treated as zero
df["revenue"].fillna(0) + df["bonus"].fillna(0)

Key Takeaways

• NaN means unknown or missing — not zero, not an empty string.
• Use .isna() and .notna(), never == None or == NaN.
• Comparisons with NaN return False — missing rows silently disappear from filters.
• Be explicit about whether missing rows should be included in your results.
• .fillna() replaces missing values with defaults. .dropna() removes rows with missing values.
• Always check .isna().sum() when exploring new data.

What’s Next

You now know how to filter data in all the ways that matter. In the next lesson, you’ll learn to sort your results and grab the top or bottom N rows — essential for any ranking or “top performers” question.