AWS S3-Backed Medallion Architecture for a Modern Data Lakehouse

A scalable data lakehouse built on AWS S3 using the Medallion Architecture (Bronze → Silver layers) for carbon emissions and energy usage reporting. Leveraging live data from the U.S. EIA API, the system automates ingestion, cleansing, and structured storage of carbon intensity and electricity demand forecasts.

The Bronze layer ingests raw JSON responses from the EIA API via Boto3. The Silver layer applies Pandas-based transformations — type casting, deduplication, Slowly Changing Dimensions (SCD Type 2) — and writes clean Parquet-partitioned data ready for downstream analytics or ML workloads.