Pandas Interview Questions (2026): Data Wrangling Q&A With Examples

A Series is a one-dimensional, labeled array that holds a single dtype end to end, basically one column with an index attached. A DataFrame is a two-dimensional table built from multiple Series that share the same row index, and unlike a Series, every column in a DataFrame can carry its own dtype independently.

You can think of a DataFrame as a dict of Series under the hood, each one keyed by column name. That's also why selecting a single column hands you back a full Series rather than a plain list.

loc selects by label and also accepts a boolean array. iloc selects by integer position only, the same way plain Python list slicing works. Mix them up and the code often still runs without erroring, which is exactly what makes the mistake dangerous.

import pandas as pd

df = pd.DataFrame({"score": [88, 91, 76, 84, 95, 69]}, index=[10, 11, 12, 13, 14, 15])

by_label = df.loc[12:14]     # includes the row labeled 14, three rows back
by_position = df.iloc[2:4]   # stops before position 4, two rows back

That inclusive-versus-exclusive detail is the single most common loc/iloc slip I see in mock interviews. loc includes the label at the end of the range, iloc excludes the position at the end, the same asymmetry Python's own list slicing has always had.

It's split, apply, combine. Pandas splits the DataFrame into groups based on the values in one or more columns, applies a function independently to each group, then combines the results back into a single output. Nothing gets computed until you call an aggregation or transformation on the grouped object, groupby() itself just sets up the split.

merge() combines two DataFrames based on shared column values, with full control over which columns to match on and how. join() is a convenience wrapper around merge() built specifically for combining on the index rather than a column. concat() doesn't match on any key at all, it just stacks DataFrames on top of each other or side by side, along whichever axis you pick.

A rough rule of thumb: reach for concat() when you're stitching together files with the same columns, reach for merge() when you're combining two tables that share a key but don't share row order or row count.

There isn't one. isnull() is an alias for isna(), kept around for backward compatibility with earlier pandas releases, and they behave identically in every case. The underlying missing-value marker is usually NaN, a special floating-point value, though pandas also recognizes None and, for a handful of dtypes, its own pd.NA sentinel as missing.

That last distinction matters more than it sounds like it should. NaN forces an integer column to upcast to float the moment a value goes missing, since NumPy's integer dtype has no way to represent a hole. Nullable dtypes like Int64, capital I, sidestep that by using pd.NA instead, which keeps the column genuinely integer even with missing values present.

Series.map() transforms one column element by element, using a function, a dict lookup, or another Series. DataFrame.apply() runs a function along an axis, one call per row or per column, and hands that function a whole Series each time rather than a single value. DataFrame.map() applies a function to every individual cell across the whole frame, and it's the direct replacement for DataFrame.applymap(), which pandas deprecated back in version 2.1 in favor of the current name.

import pandas as pd

df = pd.DataFrame({"a": [1, 2, 3], "b": [10, 20, 30]})

df["a_label"] = df["a"].map({1: "low", 2: "mid", 3: "high"})
df["total"] = df.apply(lambda row: row["a"] + row["b"], axis=1)
df[["a", "b"]] = df[["a", "b"]].map(lambda x: x * 2)

applymap() doesn't even run anymore. It carried a FutureWarning pointing at DataFrame.map() starting in 2.1, and by 3.0 it's gone outright, calling it now raises a plain AttributeError. That's one of the quieter reasons a pandas 3.0 upgrade can break a CI pipeline without anyone touching a single line of application code.

pivot() reshapes long data to wide without doing any aggregation, and it requires that every combination of your index and column values be unique. Hand it duplicate combinations and it raises an error instead of guessing what to do with them. pivot_table() solves that by aggregating duplicates automatically, using the mean by default, or whatever function you pass to aggfunc.

import pandas as pd

long_df = pd.DataFrame({
    "date": ["2026-01-01", "2026-01-01", "2026-01-02"],
    "city": ["nyc", "sf", "nyc"],
    "temp": [34, 58, 31],
})

wide = long_df.pivot(index="date", columns="city", values="temp")

wide_safe = long_df.pivot_table(index="date", columns="city", values="temp", aggfunc="mean")

If an interviewer asks which one you'd default to, pivot_table() is the safer answer for real data. Production data almost always has at least one duplicate combination hiding somewhere, and pivot() failing loudly on that is arguably a feature, not a bug, but it's not the one you want breaking a scheduled job at 6am.

Pass a list to by, and a matching list to ascending, one direction per column. na_position controls whether missing values sort to the front or the back, independent of the ascending setting for that column.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "dept": ["eng", "eng", "sales", "sales"],
    "salary": [90000, np.nan, 60000, 72000],
})

sorted_df = df.sort_values(
    by=["dept", "salary"],
    ascending=[True, False],
    na_position="last",
)

People forget na_position more than any other argument here, then get confused when a missing value shows up first in a descending sort and throws off whatever "top N" logic they wrote right after it.

read_csv() reads large files in internal chunks for memory efficiency, and if a column's values look like one dtype in an early chunk and a different dtype in a later chunk, say mostly integers with a stray text value near the bottom, pandas raises a DtypeWarning about mixed types and typically falls back to storing that column as object.

import pandas as pd

df = pd.read_csv(
    "orders.csv",
    dtype={"customer_id": "string", "order_id": "int64"},
    parse_dates=["order_date"],
    low_memory=False,
)

Two fixes, and I'd reach for the first one almost every time: specify dtype= up front for any column you know the type of, which skips the chunk-by-chunk guessing entirely. low_memory=False works too, it reads the whole column before committing to a dtype, but it uses more memory to do it and doesn't scale to a genuinely huge file the way an explicit dtype does.

axis=0 means the operation moves down each column, across rows, so df.sum(axis=0) gives you one number per column. axis=1 means the operation moves across each row, across columns, so df.sum(axis=1) gives you one number per row instead. For df.drop(), axis=0 drops rows by index label and axis=1 drops columns by column label.

The naming trips people up because it feels backwards. The easiest way to remember it is that axis=0 refers to the axis that gets collapsed, which is the row axis. If you're ever unsure, test it on a small toy DataFrame first instead of guessing in production code.

The index isn't just a row counter, it's a label pandas uses to align data between two DataFrames or Series in any operation, including arithmetic and joins. Add two Series with different indexes and pandas doesn't add them positionally, it aligns by label first, and any label missing from one side becomes NaN.

This catches people who build a DataFrame, drop some rows for cleaning, then try to combine it with a second DataFrame built before the drop. Even if the row counts happen to match, if the labels drifted, an addition or concat by label inserts NaN in the gaps instead of raising an error. The fix is usually reset_index(drop=True) on both sides before combining, if positional behavior is what you actually want.

df2 = df doesn't create a new object, it just makes df2 a second name pointing at the same DataFrame in memory. Mutating df2, adding a column or changing values in place, also changes df, because there's only one object underneath. df.copy() actually allocates new memory and copies the data, so the two become independent.

The classic bug is someone writes result = df, runs cleanup steps on result thinking it's a scratch copy, and later finds the original df upstream got mutated too. If you need an actual independent copy to experiment on, call .copy() explicitly, don't assume a slice gives you one, since whether a slice returns a view or a copy isn't always guaranteed.

Filtering with a boolean mask keeps the original index labels for whichever rows survived, it doesn't renumber them. So df[df['x'] > 5] might give you an index like 2, 5, 9, 14, not 0, 1, 2, 3. That's fine until you try something positional with it, like assigning a new column from a plain Python list, or concatenating it with another filtered frame and expecting the rows to line up.

Calling df.reset_index(drop=True) after filtering gives you a fresh 0-based index and discards the old one. If you want to keep the old labels as a regular column instead of throwing them away, drop the drop=True and it becomes a column named 'index'.

df.rename(columns={'old': 'new'}) relabels specific columns by name and returns a new DataFrame unless you pass inplace=True. It's safe even if you only want to touch two columns out of twenty, because it works off a mapping, not position.

Setting df.columns = [...] replaces the entire column index in one shot, which means you have to supply a full list matching the current number and order of columns. If you're off by one, say a merge upstream added an extra column, you'll silently mislabel everything instead of getting an error, because Python happily zips a list of names onto whatever columns exist. For anything short of a full rewrite, rename() is safer, and it also accepts a function, like df.rename(columns=str.lower).

df.duplicated() returns a boolean Series flagging every row that's an exact repeat of an earlier row, keeping the first occurrence as False by default. df.drop_duplicates() removes those flagged rows. Passing subset=['col_a', 'col_b'] compares only specific columns instead of the whole row, which matters when an auto-incrementing id column makes every row technically unique even though the actual business data repeats.

The keep parameter controls which copy survives: keep='first' (default), keep='last', or keep=False, which drops every copy of anything that appears more than once. keep=False is useful when you want to isolate and inspect duplicates rather than just discard them.

A bare column label drops a dimension. Pandas treats it as "give me this one column," so the result is a 1-D Series. Wrapping that same label in a list, even a single-item list, tells pandas you're selecting a subset of columns, so the result stays a 2-D DataFrame.

import pandas as pd

df = pd.DataFrame({"age": [25, 41, 33], "salary": [50000, 90000, 70000]})

as_series = df["age"]          # 1-D Series
as_dataframe = df[["age"]]     # 2-D DataFrame with one column

It matters more than it looks like it should. Code that expects a 2-D structure, feeding a column into a scikit-learn estimator's fit method is the classic case, throws a shape error if you hand it the Series form by mistake. More than one candidate has burned a few minutes of a live round hunting for a bug that turned out to be one missing pair of brackets.

Pass a boolean Series back into the DataFrame's brackets and pandas keeps every row where that Series reads True, aligning on the index as it goes. Combine two conditions with & and |, never the Python keywords, and wrap each condition in parentheses because of operator precedence.

import pandas as pd

df = pd.DataFrame({
    "name": ["Ana", "Bo", "Cid", "Dee"],
    "age": [34, 22, 41, 29],
    "dept": ["eng", "sales", "eng", "sales"],
})

senior_eng = df[(df["age"] > 30) & (df["dept"] == "eng")]

and/or fail because Python calls bool() on each side, and bool() on a Series with more than one element raises ValueError: The truth value of a Series is ambiguous. Pandas has no way to collapse an entire array of True/False values into a single Python boolean. & and | are overloaded to compare element-wise instead, which is exactly what filtering needs.

isin() checks membership against a list of values in one call, which reads a lot cleaner than OR-ing together three or four separate equality comparisons once you're past two options. Both approaches build the same kind of boolean Series underneath, so they combine without any friction.

import pandas as pd

df = pd.DataFrame({"dept": ["eng", "sales", "data", "hr"], "age": [34, 22, 41, 29]})

target_depts = df[df["dept"].isin(["eng", "data"]) & (df["age"] > 30)]

The main trap is forgetting that isin() takes a collection, not a single value. Hand it a bare string instead of a list and pandas raises a clear TypeError telling you only list-like objects are allowed, which is a small kindness. Older versions of this same mistake used to fail more quietly in other parts of the ecosystem, so it's worth double-checking the argument type out of habit even when the error message would catch it here.

Fancy indexing means passing a list or array of positions or labels instead of a single value or a contiguous slice. Passing a list of three specific positions pulls exactly those rows, in exactly that order, something a plain contiguous slice can't do, since a slice always grabs a contiguous run.

import pandas as pd

df = pd.DataFrame({"score": [88, 91, 76, 84, 95, 69, 80]})

picked = df.iloc[[0, 3, 6]]        # three specific rows, any order
reordered = df.iloc[[6, 0, 3]]     # same rows, different order, fancy indexing allows that

The trade-off is memory and speed. A plain slice on a NumPy-backed column can often return a view, cheap and fast, while fancy indexing with a list of positions always builds a new array, since the requested positions aren't contiguous.

.agg() collapses each group down to one summary value per column, a group of 40 rows becomes one row. .transform() runs a function per group too, but broadcasts the result back out to the original shape, so a group of 40 rows stays 40 rows, each one now holding, say, its group's average. .apply() is the general-purpose escape hatch: whatever shape your function returns is what you get back, which makes it the most flexible option and also the slowest.

import pandas as pd

df = pd.DataFrame({
    "dept": ["eng", "eng", "sales", "sales"],
    "salary": [90000, 110000, 60000, 72000],
})

avg_by_dept = df.groupby("dept")["salary"].agg("mean")           # 2 rows, one per group

df["dept_avg"] = df.groupby("dept")["salary"].transform("mean")  # 4 rows, same shape as df
df["pct_of_avg"] = df["salary"] / df["dept_avg"]

Reach for .transform() whenever you need a group-level statistic sitting next to the original rows, a running total, a z-score within a category, that sort of thing. Reaching for .apply() when .agg() or .transform() would've done the same job is the single most common performance mistake I see in take-home pandas exercises.

It grows, and often by more than people expect. Every matching row on the left gets paired with every matching row on the right that shares that key, so if a key appears twice on one side, the merged result has twice as many rows for that key as either side had on its own.

import pandas as pd

orders = pd.DataFrame({"order_id": [1, 2, 3], "customer_id": [10, 10, 20]})
customers = pd.DataFrame({"customer_id": [10, 10, 20], "region": ["west", "east", "north"]})

merged = orders.merge(customers, on="customer_id", how="left")
print(len(merged))   # 5 rows, not 3, because customer_id 10 matches twice on the right

This is the pandas equivalent of a SQL fan-out, and it's a genuinely common way real pipelines end up with duplicated revenue numbers after what looked like a routine join (pandas documentation, Merge, join, concatenate and compare).

dropna() removes rows or columns that contain a missing value, which is the right call when a missing field genuinely makes the row unusable for what you're computing. fillna() replaces missing values instead, which is the right call when you have a defensible value to put there, a group mean, a forward-filled last known reading, a hard-coded zero.

import pandas as pd
import numpy as np

df = pd.DataFrame({"score": [88, np.nan, 95, np.nan]})

filled_with_mean = df["score"].fillna(df["score"].mean())
forward_filled = df["score"].ffill()
rows_dropped = df.dropna(subset=["score"])

The actual risk with fillna() is quieter than the risk with dropna(). Dropping rows shrinks your dataset in a way that's easy to notice and audit. Filling a missing salary with the department average, on the other hand, invents a number that looks exactly as real as the ones that were actually measured, and nothing downstream flags it as synthetic unless you track it yourself.

.apply() calls a Python function once per row, or once per column, and every one of those calls carries real interpreter overhead, function call setup, type checking, the works. A vectorized operation runs as a single compiled pass over the underlying NumPy array instead, with no Python-level function call anywhere in the loop.

import numpy as np
import pandas as pd

df = pd.DataFrame({"price": np.random.rand(1_000_000) * 100})

with_tax_apply = df["price"].apply(lambda p: p * 1.0825)   # 1 million Python function calls
with_tax_vectorized = df["price"] * 1.0825                  # one array-level multiply

On something this simple, vectorized wins by somewhere in the range of 50x to 200x, depending on the machine and how much work the lambda is doing. That's a wide range on purpose. I don't have one clean number to give you, because it moves a lot with row count and function complexity, but I've never seen it come out close on a numeric column like this one.

A category column stores two things instead of one: a small array of the distinct values that actually appear, and, for every row, a small integer code pointing into that array. A column with four department names repeated across 250,000 rows stores those four strings exactly once, not 250,000 times, so the memory savings scale with how repetitive the column is.

import pandas as pd

df = pd.DataFrame({"dept": ["eng", "sales", "eng", "sales"] * 250_000})

before = df["dept"].memory_usage(deep=True)     # object dtype, a full Python str per cell

df["dept"] = df["dept"].astype("category")
after = df["dept"].memory_usage(deep=True)      # category dtype, integer codes plus one small lookup table

It stops helping, and can actually hurt, once a column's cardinality gets close to its row count. A column of mostly-unique IDs stored as category still needs an entry in the lookup table for nearly every row, so you're paying for the code array on top of the lookup table with almost none of the deduplication benefit. Category dtype is a low-cardinality tool, department names, status flags, country codes, not a general-purpose string compressor. Worth noting pandas 3.0's new default string dtype trims memory for plain text columns too, but it doesn't deduplicate repeated values the way category does.

melt() turns wide data long, taking a set of columns and stacking them into two columns, a variable name and a value, while keeping one or more identifier columns fixed. It's the reverse operation of pivot().

import pandas as pd

wide = pd.DataFrame({
    "student": ["Ana", "Bo"],
    "math": [88, 91],
    "science": [79, 85],
})

long_df = wide.melt(id_vars="student", var_name="subject", value_name="score")

The real trigger is plotting or modeling. Most charting libraries, and every scikit-learn estimator, expect one observation per row, not one subject per column. A wide table with a column per subject looks readable to a human and is almost unusable to a groupby or a plot call without melting it first.

It parses strings, or numbers under an assumed unit, into pandas's datetime64 dtype, which gives you the .dt accessor for pulling out things like the hour, the weekday name, or the ISO week. Before pandas 3.0, the resulting dtype defaulted to nanosecond resolution, datetime64[ns], no matter what precision the input actually needed.

import pandas as pd

df = pd.DataFrame({"logged_at": ["2026-03-22 11:36", "2026-03-23 09:02"]})
df["logged_at"] = pd.to_datetime(df["logged_at"])

df["hour"] = df["logged_at"].dt.hour
df["weekday"] = df["logged_at"].dt.day_name()

Yes, it changed. As of pandas 3.0, to_datetime() infers the resolution the data actually needs instead of always defaulting to nanoseconds, strings like the ones above land on datetime64[us] now, microsecond resolution, rather than datetime64[ns] (pandas 3.0.0 release notes). It rarely changes your results, but it can break code that checks a column's dtype string against nanosecond resolution exactly instead of just confirming it's some flavor of datetime64.

Parquet is columnar and stores the schema alongside the data, every column's dtype, including things CSV can't represent at all like a proper category dtype, comes back exactly as it went in. CSV is plain text, so every read has to re-infer types from scratch, and anything subtle, a ZIP code that looks numeric, a boolean stored as text instead of a number, is a genuine risk of silently importing wrong on the next read.

import pandas as pd

df = pd.DataFrame({"order_id": [1, 2, 3], "region": ["west", "east", "north"]})

df.to_parquet("orders.parquet", index=False)
df_from_parquet = pd.read_parquet("orders.parquet")

What you give up is human readability and a dependency: Parquet needs pyarrow or fastparquet installed, and you can't open it in a text editor to eyeball a row when something looks off. For anything wide, numeric, and read more than once, Parquet is usually smaller on disk and meaningfully faster to load than the equivalent CSV. For a one-off export someone's going to open in a spreadsheet, CSV still wins.

df.query("age > 30 and salary < 50000") reads like SQL and saves you from repeating df[...] four times, which helps for chained conditions. But it evaluates the expression string at runtime, so a typo in a column name shows up as a runtime error instead of something your editor would catch at write time, and it doesn't handle column names with spaces or special characters without backtick-quoting them. Referencing a Python variable requires an @ prefix, like df.query("age > @min_age").

Boolean indexing (df[df['age'] > 30]) is more verbose but is checked by your editor, works with any column name, and is what most production codebases standardize on. query() earns its keep mainly in interactive analysis, or when you're chaining five or six conditions and the boolean-mask version genuinely gets hard to read.

df['col'].value_counts() counts occurrences of each unique value and sorts descending by count automatically, and normalize=True turns those counts into proportions that sum to 1, handy for checking class balance on a target column. df.groupby('col').size() gives the same counts but as a Series indexed by the group key in group order, not count order, and there's no built-in normalize flag, you'd divide by len(df) yourself.

The real difference shows up with multiple columns. value_counts() only works on a single Series, while groupby(['col_a', 'col_b']).size() is the natural way to get counts across a multi-column combination with a MultiIndex result. Also worth knowing, value_counts() drops NaN by default unless you pass dropna=False, so if you're auditing missing values it silently hides them.

df.where(cond, other) keeps values where cond is True and replaces them with other, default NaN, where cond is False. df.mask(cond, other) is the exact inverse, it replaces values where cond is True and keeps them where it's False. People mix these up constantly because the names don't map obviously to keep versus replace.

A common bug is writing df.where(df['score'] < 0, 0) intending to zero out negative scores, but that actually zeroes out everything that isn't negative, since where() keeps the True side. What most people actually want here is df.mask(df['score'] < 0, 0), or flipping the condition and using where(df['score'] >= 0, 0). Neither mutates in place unless you pass inplace=True, so a bare df.where(...) call with no assignment silently does nothing useful.

pd.cut() bins values into intervals you define by fixed edges, or an integer count of equal-width bins across the data's min and max, so the bins have equal range but not necessarily equal population, like sorting ages into 0-18, 18-35, 35-60, 60+. pd.qcut() bins by quantile instead, so every bin ends up with roughly the same number of rows but the widths vary based on the actual distribution, which is what you want for splitting customers into spend deciles.

A real gotcha with qcut() is that with a lot of repeated values, say a column that's mostly zero, it can fail with a "Bin edges must be unique" error, because two quantile boundaries land on the same value. You either pass duplicates='drop' or fall back to manually defined bins with cut().

The old dict-style aggregation, like df.groupby('team').agg({'sales': 'sum', 'sales': 'mean'}), silently breaks the moment you want two different aggregations on the same column, because dict keys have to be unique, so the second 'sales' just overwrites the first. Named aggregation fixes this by naming each output column explicitly.

df.groupby('team').agg(
    total_sales=('sales', 'sum'),
    avg_sales=('sales', 'mean'),
)

It also flattens the result into normal single-level column names instead of the nested MultiIndex columns from the old style, which used to require a manual columns = ['_'.join(c) for c in df.columns] cleanup step afterward. It's the version worth using in any pipeline code other people will read later, since the output column names are self-documenting.

.at[] and .iat[] are scalar-only accessors, they get or set a single value by label or by integer position respectively, and they skip a lot of the overhead .loc[] and .iloc[] carry for handling slices, boolean masks, and multi-axis selection. In a tight loop pulling or writing one cell at a time, which itself is usually a sign you should be vectorizing instead, .at[] is measurably faster than .loc[] for that single lookup.

The tradeoff is that .at[] and .iat[] raise if you pass anything other than a single row and column pair, no slices, no lists, no boolean arrays. They're not a drop-in replacement for .loc[]/.iloc[] in general code, just a narrow optimization for scalar access in hot paths.

A MultiIndex is an index made of more than one level, typically the result of grouping or pivoting on multiple columns, so a row might be addressed by a tuple like ('north', 'january') instead of a single label. Selecting all rows for 'north' regardless of month is awkward with plain .loc[], since you'd need a value for every level or a tuple with slice(None) placeholders.

df.xs('north', level='region') does exactly that in one call, pulling a cross-section at a specific level and dropping that level from the result by default, unless you pass drop_level=False. The gotcha is that .xs() on a DataFrame is read-only, you can't assign through it the way you can through .loc[], so if you need to both select and mutate a slice of a MultiIndex frame, build the tuple key yourself and use .loc[] instead.

reset_index() takes whatever the current index is and turns it into a regular column, or discards it with drop=True, replacing it with a fresh default 0..n-1 range. reindex() does something unrelated, it takes a completely new set of labels you supply and reshapes the DataFrame to match that set exactly, keeping existing rows whose labels appear in the new set, dropping rows whose labels don't, and inserting NaN rows for any label that wasn't already present.

reindex() is what forces two differently-shaped DataFrames onto a common set of dates or ids before comparing them, for example reindexing a sparse time series onto a full date range to expose missing days as explicit NaN rows instead of just absent rows.

A plain numpy int64 column can't represent a missing value, there's no NaN in the integer type, so the moment a single NaN needs to go into what would otherwise be an all-integer column, pandas upcasts the whole column to float64. Now your ids or counts are technically floats, like 1.0, 2.0, NaN, which trips up anything downstream that assumes integer semantics.

df['user_id'] = df['user_id'].astype('Int64')

The nullable Int64 dtype, capital I, distinct from numpy's lowercase int64, uses pandas's own extension array under the hood and can hold pd.NA alongside actual integers without ever converting to float. The cost is that Int64 columns are slower for heavy numeric computation than plain numpy int64 or float64, because they're not a raw contiguous numpy array. It's a tool for correctness in id-like or count-like columns with real gaps, not something to apply to every numeric column by default.

rolling(window=n) computes a statistic over a fixed-size sliding window, like a 7-day moving average, where the window slides forward one row at a time, dropping the oldest row as it adds the newest. expanding() computes over a window that starts at the beginning of the data and grows by one row each step, so expanding().mean() at row 100 is the average of all 100 rows seen so far, not just a fixed recent slice.

Rolling is what you want for smoothing noisy time series or catching recent trend changes, expanding is what you want for a running total or a cumulative metric like average conversion rate to date. Both respect min_periods to control how many initial rows get NaN before there's enough data to compute the first real value.

The .str accessor lets you call string methods across an entire column, like df['name'].str.lower() or df['email'].str.contains('@'), and it tolerates missing values gracefully, a NaN entry just stays NaN in the output instead of raising an AttributeError. Where this bites people is .str.contains() specifically, because by default it returns NaN for rows where the original value was NaN, and NaN doesn't behave as False in a boolean mask.

df[df['email'].str.contains('@', na=False)]

So df[df['email'].str.contains('@')] can raise a ValueError about the mask containing NaN values, instead of simply excluding those rows. Passing na=False explicitly tells it to treat missing values as non-matches rather than propagating NaN into the filter.

fillna() replaces NaN values in a Series or DataFrame with a single scalar, or with values from another Series aligned by index, but it treats that other Series as pure filler, position by position it just plugs the gap and stops. combine_first() is built specifically for merging two overlapping DataFrames where df1's values should win wherever df1 has data, falling back to df2's value only where df1 is missing, across every column at once, aligned by index and column label.

It's the right tool when you have a primary data source and a backup source with the same schema and overlapping but not identical row coverage, and you want one unified frame that prefers the primary source. fillna(df2) can approximate this for a single column, but combine_first() does it correctly across the whole frame in one call, without looping over columns.

explode() takes a column where each cell holds a list, or any iterable, and turns each element into its own row, duplicating the rest of that row's values across all the new rows, with the index repeated too so you can tell which original row each new row came from. It's the direct fix for data that comes in denormalized, like a column of tags stored as ['python', 'sql', 'aws'] per row, which needs to be exploded before you can run a groupby or value_counts() on individual tags.

A common follow-up mistake is forgetting the index is now duplicated. Reset_index(drop=True) too early and you lose the ability to trace an exploded row back to its source, so it's usually worth keeping the original index, or saving it to a column, until whatever downstream join needs it is done.

That's chained indexing: two separate lookups back to back. The first one, filtering by age, can return either a view into the original data or a fresh copy, and pandas can't always promise which. The second lookup then assigns into whatever that intermediate result happened to be, so the original DataFrame may or may not actually change.

import pandas as pd

df = pd.DataFrame({"age": [25, 41, 33], "salary": [50000, 90000, 70000]})

df[df["age"] > 30]["salary"] = 0        # two chained lookups, unreliable
df.loc[df["age"] > 30, "salary"] = 0    # one call, one target, works

The fix is always the same: do it in one loc call instead of two separate ones. Before pandas 3.0, the chained version usually triggered SettingWithCopyWarning, pandas's way of flagging that it genuinely couldn't tell whether your assignment landed. That warning is gone as of pandas 3.0, released January 21, 2026 (pandas 3.0.0 release notes). Copy-on-write is now always on and can't be disabled, every indexing operation behaves as a copy from the user's point of view, so the ambiguity the old warning used to flag doesn't exist anymore. What replaced it is more direct: run that exact chained line on pandas 3.0 and it raises ChainedAssignmentError, spelling out that chained assignment never updates the original object under copy-on-write and pointing you straight at loc. The assignment still doesn't land, but at least it stops pretending it might have.

Pandas drops them, by default. groupby() has a dropna parameter that defaults to True, which means any row whose group key is null just disappears from the output entirely, silently, with no warning. Set dropna=False and it comes back as its own group.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "dept": ["eng", "eng", "sales", np.nan],
    "salary": [90000, 110000, 60000, 40000],
})

summary = df.groupby("dept").agg(
    avg_salary=("salary", "mean"),
    headcount=("salary", "size"),
)
# the fourth row, dept is NaN, is gone entirely, not shown as its own group

summary_keep_na = df.groupby("dept", dropna=False).agg(
    avg_salary=("salary", "mean"),
    headcount=("salary", "size"),
)

I'd argue this is a bigger real-world risk than any aggregation syntax question. A dashboard that quietly excludes every row with a missing department looks fine until someone asks why the total headcount doesn't match the row count of the source table.

validate= and indicator= are the two arguments built for exactly this. validate checks that the relationship between your keys actually matches what you claim it is, one_to_one, one_to_many, many_to_one, or many_to_many, and raises a MergeError immediately if it doesn't. indicator=True adds a _merge column showing whether each row came from the left table only, the right table only, or matched on both.

import pandas as pd

orders = pd.DataFrame({"order_id": [1, 2, 3], "customer_id": [10, 10, 20]})
customers = pd.DataFrame({"customer_id": [10, 10, 20], "region": ["west", "east", "north"]})

# customer_id isn't actually unique on the right, so this raises MergeError
# instead of silently doubling up rows the way the plain merge above did
try:
    orders.merge(customers, on="customer_id", how="left", validate="many_to_one")
except pd.errors.MergeError as e:
    print(e)

# indicator=True flags which rows matched on both sides versus only one,
# useful even when validate isn't strict enough to catch the whole problem
checked = orders.merge(customers.drop_duplicates("customer_id"), on="customer_id", how="left", indicator=True)
print(checked["_merge"].value_counts())

I always ask candidates whether they'd add validate= to a merge going into a production pipeline versus a one-off notebook. The honest answer is yes, almost every time, and it costs one keyword argument. I don't have a great excuse for why it isn't the default habit more often.

Two separate problems, not one. It's slow because it's a Python-level loop with per-row overhead, same root cause as .apply(). But it also silently upcasts dtypes: iterrows() rebuilds each row as its own Series, and a Series can only hold one dtype, so a row with a float and an int gets coerced to a common dtype, usually object or float64, losing the original per-column types for the duration of that loop.

import pandas as pd

df = pd.DataFrame({"price": [10.5, 20.25, 15.0], "qty": [3, 1, 4]})

# avoid: rebuilds a Series per row, coerces qty's int dtype away
total = 0
for _, row in df.iterrows():
    total += row["price"] * row["qty"]

# better: itertuples() keeps native types, no per-row Series
total = sum(row.price * row.qty for row in df.itertuples())

# best: skip the Python-level loop entirely
total = (df["price"] * df["qty"]).sum()

My honest take: most "use itertuples() instead of iterrows()" advice undersells the fix. itertuples() is a real improvement, it keeps native types and skips building a Series, but it's still a Python-level loop with per-row overhead. If a vectorized version exists, and for arithmetic like this one almost always does, it beats itertuples() by a wide margin too. Reach for itertuples() only when the per-row logic genuinely can't be expressed as array operations.

stack() moves the innermost level of the columns down into the innermost level of the row index, compressing a wide DataFrame into a taller one, often a Series if only one column level existed. unstack() reverses that, pivoting a row index level back out into columns.

import pandas as pd

df = pd.DataFrame(
    {"eng": [90, 85], "sales": [70, 75]},
    index=pd.MultiIndex.from_tuples(
        [("2026-01", "bonus"), ("2026-02", "bonus")], names=["month", "kind"]
    ),
)

stacked = df.stack()            # eng/sales columns fold into the row index
unstacked = stacked.unstack()   # back to the original wide shape

The failure mode shows up with sparse combinations. unstack() on a MultiIndex where most index-and-column combinations never actually co-occur creates a mostly-empty DataFrame full of NaN, since it has to allocate a cell for every combination whether or not the data had one. On a MultiIndex with high cardinality on both levels, that can balloon memory fast for what looks like a small underlying dataset.

resample() requires a DatetimeIndex and produces one row for every calendar bin across the full range you specify, whether or not any data actually falls in that bin. groupby() on a plain date column only ever produces a row for a date that genuinely has at least one matching record, gaps just don't appear at all.

import pandas as pd

ts = pd.DataFrame(
    {"revenue": [120, 80, 95]},
    index=pd.to_datetime(["2026-01-01", "2026-01-01", "2026-01-03"]),
)

daily_resample = ts.resample("D").sum()          # Jan 2 shows up as a row with 0 revenue
daily_groupby = ts.groupby(ts.index.date).sum()  # Jan 2 doesn't appear at all

Which one is "correct" depends entirely on what a missing day means for your data. Zero trades and zero revenue is a real, meaningful zero for a sales table, so resample() is usually right there. For a table of sensor readings where a gap means the sensor was offline rather than reading zero, treating that gap as a real zero would be a genuine data error, and skipping the day with a plain groupby, or filling it with NaN instead of zero, is the more honest choice.

Building a DataFrame by looping and calling pd.concat([df, new_row_df]) on every iteration is quadratic, not linear, because concat allocates a brand new block of memory and copies every row that already existed plus the new one, every single time. Do that a thousand times and you've copied the growing DataFrame roughly a thousand times over, so what looks like O(n) work in the loop is actually O(n^2) in total bytes copied, and it visibly slows down as the loop progresses even though each iteration looks like it's doing the same amount of work.

rows = []
for chunk in source:
    rows.append(process(chunk))
result = pd.concat(rows, ignore_index=True)

The fix is to collect each row, or small DataFrame, into a plain Python list as you go, and call pd.concat(list_of_frames) exactly once at the end, letting pandas do one allocation and one copy instead of thousands. The same principle is why df.append() was deprecated and removed, it encouraged exactly this pattern.

Copy-on-write changes the contract around views versus copies. Under classic pandas behavior, whether df[mask]['col'] = value actually mutated the original DataFrame or a disconnected temporary copy depended on internal details of how the DataFrame's memory was laid out, which is exactly what produced the notorious SettingWithCopyWarning, a warning that fired because pandas genuinely couldn't guarantee which behavior you'd get.

With copy-on-write enabled, opt-in since pandas 2.0 and default since pandas 3.0, every slice or filter behaves as if it returned an independent copy, and pandas only actually copies the underlying data lazily, the moment you try to write to it, not on every slice. The practical upshot for existing codebases is that chained assignment like df[df.x > 0]['y'] = 1, which used to silently mutate the original df, stops working entirely, it just modifies a copy that gets discarded. Upgrading old pipelines means grepping for chained indexing patterns and rewriting them as a single df.loc[mask, 'y'] = 1 call.

groupby().apply() inspects the return type of your function on the first group and builds its output shape based on that. If your function returns a scalar for most groups but occasionally returns a Series or DataFrame, say one group has enough rows for a regression fit while another has a single row so you return an empty frame instead, the combined result can come back with a different index structure than expected, sometimes with an extra level, sometimes flattened, sometimes an error deep in pandas internals that doesn't point at your function at all.

The fix is defensive: always return the same type and shape from the function regardless of the group's size or content, even if that means returning a Series of NaNs with matching index labels for the degenerate case instead of an empty object. It's also worth checking whether what you're doing can be expressed as .agg() or .transform() instead, both of which have a much stricter, more predictable contract about output shape than the fully general .apply().

The first thing to check is dtype, since object columns carry a lot of per-cell overhead for Python string objects, and numeric columns default to int64/float64 even when a smaller type like int32, or a category dtype, would hold the same data in a fraction of the space. df.info(memory_usage='deep') is the command to run first, since the default memory estimate undercounts object columns badly and 'deep' gives the real number.

After that, check for row duplication from an operation like a merge that silently multiplied rows, or an .apply() materializing a large intermediate object per row. If the CSV itself is large, specifying dtype up front instead of letting pandas infer and downcast later, and reading in chunksize batches, avoids ever holding the full inferred, unoptimized frame in memory at once. If none of that is enough and the data genuinely doesn't fit comfortably, that's usually the signal that pandas is the wrong tool for this particular job, and it's time to reach for Polars, Dask, or DuckDB instead of forcing pandas to do it.

Floating point addition isn't associative once you're dealing with binary floating point representation, so summing the same set of numbers in a different order, exactly what happens when you groupby and sum versus computing an expected total some other way, or when you sum again after new rows changed the internal ordering, can produce a result that differs in the last few bits, like 39.99999999999999 instead of 40.0.

import numpy as np
np.isclose(total, 40.0, atol=1e-9)

Comparing that against an expected value with == fails even though the numbers are the same for any practical purpose, and the resulting bug reports look like data corruption when it's actually floating point arithmetic behaving exactly as specified. Never compare floats for exact equality after any aggregation, use numpy.isclose() or pandas's own testing utilities with an explicit tolerance instead, and if the values represent money specifically, consider storing integer cents or using Python's Decimal from the start, since money is exactly the domain where this kind of silent drift causes real damage.

A tz-naive datetime column, no UTC offset attached, and a tz-aware column, with tzinfo whether UTC or a local zone, are genuinely different values as far as pandas is concerned, even if the printed timestamps look identical. Merging or comparing across the two doesn't necessarily raise an error, it can just fail to match anything, or in some operations raise a TypeError about comparing naive and aware datetimes.

The dangerous version of this bug is when the merge doesn't error at all, it just silently produces far fewer matches than expected on a key that includes a timestamp, and unless someone checks the row count against an expectation, it ships. Catching it means checking .dt.tz on both sides before any merge or comparison involving datetime columns, standardizing both to the same tz-awareness, usually converting everything to UTC-aware with tz_localize or tz_convert, up front, and treating an unexpectedly low match rate on a merge as a standing signal to check for exactly this kind of type mismatch rather than assuming the data itself is just sparse.