6. Cleaning Data

df.isna().sum() # count NaNs per column

df.isna().sum()                 # count NaNs per column
df.dropna()                     # drop rows with any NaN
df.dropna(subset=['age'])       # only if 'age' is NaN
df.dropna(axis=1)               # drop columns with NaN
df.fillna(0)                    # fill all NaN with 0
df['age'].fillna(df['age'].median())   # fill with median
df.fillna(method='ffill')       # forward fill (use .ffill() in new pandas)

df.replace({'?': np.nan, 'N/A': np.nan})
df.duplicated().sum()           # count duplicate rows
df.drop_duplicates(subset=['id'], keep='last')
df['age'] = df['age'].astype('int32')
df.rename(columns={'age': 'years'})

df['city'] = df['city'].str.strip()   # trim whitespace
df.columns = df.columns.str.strip().str.lower()

Task	Method
Detect missing	`isna()`, `notna()`
Drop missing	`dropna(subset=, axis=, how=)`
Fill missing	`fillna(value/median/ffill/bfill)`
Replace values	`replace()`
Duplicates	`duplicated()`, `drop_duplicates()`
Convert types	`astype()`, `pd.to_numeric(errors='coerce')`
Rename	`rename(columns=...)`

🚀 Best Practice: For numeric columns with junk strings, pd.to_numeric(df['col'], errors='coerce') turns bad values into NaN so you can handle them cleanly.

⚠️ Common Mistake: df.fillna(0) returns a new DataFrame; it doesn't modify in place unless you reassign (df = df.fillna(0)) — and avoid inplace=True, which is being deprecated.

⭐ Interview Question: How to handle missing values? Understand why they're missing, then choose: drop (if rare), impute (mean/median/mode, ffill/bfill for time series), or flag with an indicator column.