Enterprise Data Architect S1 - Zero Code Metadata-Driven DQ Firewall

GitHub Repository

1. Core Philosophy (No-Code Rules & Small Data Approach)

Goal:

To build a metadata-driven Data Quality Firewall that operates as a standalone "Data Product". The objective is to dynamically validate and route data (to Silver, Quarantine, or Pending queues) using control tables, ensuring 100% trusted data for downstream analytics without modifying any PySpark code.

"Show, Don't Tell" (Why this demo uses Small Data):

I believe visual proof is more effective than long theoretical explanations. I purposefully used a very small dataset for this demonstration so you can easily trace step-by-step exactly why and where each record is routed based on the control logic.

However, whether processing 10 records or 100 million, the PySpark notebook is fully scalable to handle it seamlessly.

The 3-Step Demonstration Plan: To prove the power of this self-healing architecture, the pipeline is executed in three stages:

• Run 1 (Baseline & Smart Routing): Validates the initial data. Good records proceed to Silver, invalid records are Quarantined, and incomplete data is held in the Pending queue.
• Run 2 (Auto-Recovery & Threshold Breach): Source references are updated, and new records are injected. The pipeline automatically recovers previous pending records to Silver while triggering a business alert for newly detected anomalies.
• Run 3 (Self-Healing & Cleanup): The business team corrects the faulty source data. Upon re-execution, the pipeline automatically clears the Quarantine table, successfully pushing all previously blocked records to Silver.

2. The "Brain" of the Firewall (Metadata Control Table)

This table acts as the central brain of the architecture. Instead of hardcoding data quality rules into PySpark, all validation logic—such as rule types, thresholds, and routing actions (Quarantine/Warn/Pending)—is stored here as metadata.

This empowers business teams to dynamically manage rules directly from the table, eliminating the need for engineering deployments.