Data Cleansing Patterns and Missing Data Imputation with MICE

Real-world datasets are rarely pristine; missing values, corrupted entries, and incomplete records routinely threaten the reliability of machine learning models.

By understanding missingness mechanisms (MCAR, MAR, MNAR) and implementing Multiple Imputation by Chained Equations (MICE) via Scikit-Learn's IterativeImputer, you can accurately estimate missing attributes and preserve statistical variance.

Core Missingness Mechanisms & MICE Mechanics

Before applying imputation algorithms, data scientists must categorize missing data patterns to avoid introducing severe statistical bias:

• MCAR (Missing Completely at Random): The probability of missingness is entirely independent of any observed or unobserved data (e.g., a sensor failing purely by random chance).
• MAR (Missing at Random): Missingness depends systematically on other observed variables rather than the missing variable itself (e.g., income data missing more frequently for specific age groups).
• MNAR (Missing Not at Random): The missingness is directly related to the unobserved value itself (e.g., high earners refusing to report their salary).
• MICE Algorithm: Imputes missing values iteratively by modeling each feature with missing values as a function of other features in a round-robin regression loop.

Simple Imputation vs. MICE Iterative Imputation

Python Scikit-Learn MICE Implementation Blueprint

Here is how to implement iterative missing data imputation (MICE) in Python using Scikit-Learn's experimental IterativeImputer:

import numpy as np
import pandas as pd
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
from sklearn.linear_model import BayesianRidge

# 1. Create dataset with simulated missing values (NaN)
np.random.seed(42)
n_samples = 100

x1 = np.random.normal(loc=50, scale=10, size=n_samples)
x2 = x1 * 1.5 + np.random.normal(loc=0, scale=5, size=n_samples)
x3 = np.random.choice([0, 1], size=n_samples, p=[0.4, 0.6])

df = pd.DataFrame({'feature_a': x1, 'feature_b': x2, 'feature_c': x3})

# Introduce 20% random missingness (NaN) into feature_b
mask = np.random.rand(n_samples) < 0.2
df.loc[mask, 'feature_b'] = np.nan

print(f"Missing values before imputation: {df['feature_b'].isna().sum()}")

# 2. Initialize MICE Iterative Imputer with Bayesian Ridge estimator
mice_imputer = IterativeImputer(
    estimator=BayesianRidge(),
    max_iter=10,
    random_state=42
)

# 3. Fit and transform dataset
imputed_array = mice_imputer.fit_transform(df)
df_imputed = pd.DataFrame(imputed_array, columns=df.columns)

print(f"Missing values after MICE imputation: {df_imputed['feature_b'].isna().sum()}")
print(f"Imputed DataFrame Head:\n{df_imputed.head()}")

Data Cleansing Best Practices

When executing data cleansing pipelines and handling missing attributes in production machine learning systems, follow these guidelines:

• Fit Imputers on Training Data Only: Always fit your IterativeImputer on training splits and apply transform() to validation and test sets to prevent data leakage.
• Add Missingness Indicators: For features where missingness itself carries predictive weight (e.g., skipped survey questions), append binary indicator columns before imputation.
• Monitor Imputation Convergence: Set appropriate maximum iteration limits and inspect convergence warnings when working with highly collinear multivariate datasets.