Data Engineering🔨 Personal Project
Modern ELT Pipeline with Airflow, Airbyte & dbt
End-to-end ELT pipeline orchestrating data ingestion from HTTP sources to BigQuery with dbt transformations and Airflow workflow management.
Project Overview
🎯
Objective
Design and implement a modern ELT pipeline that automates data ingestion, transformation, and orchestration for building structured DataMarts
💼
My Role
Data Engineer
⏱️
Timeline
2 weeks
🛠️
Tech Stack
Apache Airflow, Airbyte, dbt
📈
Key Results
- ✓Implemented automated data ingestion from public HTTP sources using Airbyte
- ✓Built dbt transformation models for clean, organized analytics-ready data
- ✓Orchestrated entire workflow with Apache Airflow for dependency management
- ✓Created scalable DataMart optimized for business intelligence and reporting
Impact
Automated DataMart creation
Value
End-to-end ELT orchestration
🔗Project Links
GitHub RepositoryPrivate
Tools & Technologies
Apache AirflowAirbytedbtBigQueryPythonDocker