AI Data Preprocessing Audit Workpaper: Pandas Data Handling Risks

Audit Focus Area

Evaluating potential data integrity risks in Pandas-based preprocessing pipelines that may lead to:

• Silent data loss (unlogged row/column deletions)
• Lack of lineage tracking (untraceable transformations)
• Overaggressive filtering (regex or .dropna() biases)
• Cultural/dialect erasure (slang removal via rigid regex rules)

Evidence Collection Procedures

1. Silent Row/Column Drops

Where to Find Evidence:

• Code Review: Search for .drop(), df[df.col.isna()], or boolean masking without logging.
• Version Control Diffs: Compare input/output row counts between commits (Git history).
• Pipeline Logs: Check if deletion counts are recorded (e.g., print(f"Dropped {{len(df_before) - len(df_after)}} rows")).

Test:

assert "dropped_rows" in preprocessing_logs, "No audit trail for row drops"

2. No Lineage Tracking

Where to Find Evidence:

• Metadata Checks: Look for .attrs or custom lineage tags (e.g., df.attrs["source"]).
• Pipeline Tools: Check if tools like MLflow, Pachyderm, or OpenLineage are used.
• Data Provenance: Verify if intermediate datasets are versioned (e.g., df.to_parquet("data_v1.2.parquet")).

Test:

assert hasattr(df, "_file_origin"), "No lineage metadata attached"

3. Overaggressive Regex Filtering

Where to Find Evidence:

• Regex Patterns: Review df.str.replace() or .str.contains() for overly strict rules (e.g., r"[^a-zA-Z0-9]" removing non-Latin scripts).
• Bias Testing: Compare pre/post-filtering demographics (e.g., minority language tokens in NLP data).
• False Positive Logs: Check if rejected samples are reviewed (e.g., df[~df.text.str.match(regex)] saved to a log).

Test:

if "[\u0600-\u06FF]" not in allowed_chars:  # Arabic script example
    raise ValueError("Regex filters out non-English scripts")

4. .dropna() Erasing Minority Data

Where to Find Evidence:

• Missing Value Analysis: Audit .isna().sum() per subgroup before/after drops.
• Thresholds: Verify if thresh= or subset= params disproportionately affect rare categories.
• Alternative Methods: Check if imputation (e.g., .fillna()) was considered but not used.

Test:

minority_loss = (df["ethnicity"].value_counts(normalize=True) - df_clean["ethnicity"].value_counts(normalize=True))
assert minority_loss.max() < 0.05, "dropna() disproportionately affected minority groups"

5. Slang/Dialects Filtered by Regex

Where to Find Evidence:

• Keyword Lists: Search for exclusion terms (e.g., r"\bain’t\b|\by’all\b").
• Cultural Bias Tests: Use fairness tools like Aequitas to measure dialect representation loss.
• Human Review: Spot-check filtered samples for false positives (e.g., African American Vernacular English).

Test:

dialect_phrases = ["finna", "hella", "yinz"]
assert any(phrase in df.text for phrase in dialect_phrases), "Dialect removed by preprocessing"