An enterprise-grade data engineering simulation demonstrating automated raw transaction ingestion, statistical anomaly detection (Z-score cleaning), and secure bulk loading into a production SQL database.
Statistical check flagged values with a Z-score greater than 2.0 (representing extreme duplications or data transmission errors). Choose to clean them below:
| SKU | Order ID | Raw Vol | Z-Score | Type | Action |
|---|
| Timestamp | SKU ID | Store Node | Sales Vol | Integrity Hash |
|---|---|---|---|---|
| Database table empty. Run Pipeline Extract & Load. | ||||
Extract Node: Automatically triggers an API call simulation pulling Shopify Webhook payloads (JSON structure) for 10 consecutive trading days.
Transform Node: Runs Z-score calculation on volume:
Z = (X - μ) / σ. Volume values exceeding a Z-score threshold of 2.0 are flagged as duplicate database errors and cleaned using a rolling median.
Load Node: Executes structured bulk loading. Each record is sealed with an SHA-256 integrity hash before being written to database tables.