FlowLake: Demo

FlowLake is a distributed data processing pipeline built on Apache Spark (Scala + PySpark) and Delta Lake. It ingests raw transaction data, applies business transformations, runs automated data-quality checks (null %, duplicates, completeness, record counts), and writes validated output to a Delta-style data lake, monitored through a FastAPI control-room dashboard. A real Apache Airflow deployment (dags/data_pipeline_dag.py) schedules and runs the pipeline daily, submitting the actual Scala job to the live Spark cluster.

—

Rows processed

—

QC checks passed

—

Pipeline duration

—

Countries covered

—

Uptime

Interactive Pipeline Simulator

Choose a batch size and click Run to simulate a full etl_corporate_registry_pipeline DAG execution: Extract → Transform → Data Quality → Load → Report.

Batch size

Data quality mode

Extract

Parquet source files

—

Transform

total_amount, date, year, month

—

Data Quality

15 checks: nulls, dups, count

—

Load

Delta Lake output

—

Report

JSON + notification

—

Pipeline log

Data Quality Report, Last Run

Total: 15 / 15 passed (100%)

Sample Transformed Record

Architecture

Tech Stack

Apache Spark (Scala) PySpark 3.5 Delta Lake FastAPI pandas Docker Compose Python 3.11 Apache Airflow 2.7