Data Engineering
Insurance Data Engineering & Reporting Pipeline
Null · outlier · distribution-shift checks
Graduate project · Jan – May 2025
- Python
- SQL
- Power BI
Problem
Reports are only as trustworthy as the pipeline behind them. This pipeline needed to move raw structured insurance data to stakeholder-facing Power BI dashboards without letting data-quality problems reach those reports silently.
Architecture
Python- and SQL-based ETL: extraction and transformation in Python, persistence and set-based transformation in SQL, governed by explicit schema design and validation rules. A data-quality layer runs null handling, outlier detection, and distribution-shift checks against the pipeline output. Cleaned, validated datasets feed two consumers: Power BI dashboards for reporting, and downstream modeling layers.
Raw structured insurance data is extracted and transformed in Python, then persisted and further transformed in SQL under explicit schema and validation rules. A data-quality layer applies null handling, outlier detection, and distribution-shift checks. Validated datasets then split into two consumers: Power BI dashboards for reporting, and downstream modeling layers.
Key Decisions
- Schema and validation rules defined explicitly, ahead of load, rather than inferred from the data.
- Distribution-shift checks run alongside null and outlier checks, to catch statistically wrong data as well as structurally invalid data.
- Separate curated datasets produced for reporting and modeling, each shaped for its consumer.
Results
Validated datasets feed both reporting and modeling layers. Power BI dashboards built on those datasets surface trends, anomalies, and performance metrics for stakeholder decision-making.