ETLFlow ETL - Data pipeline with Airflow and Spark
End-to-end ETL pipeline orchestrated with Apache Airflow and processed with Apache Spark for scalable data transformation and analytics.
ETLFlow is a production-grade data pipeline that ingests data from multiple sources, performs transformations using Apache Spark, and loads results into analytics databases. Airflow DAGs manage scheduling, monitoring, and error handling. The pipeline supports incremental loads, data quality checks, and is containerized with Docker for easy deployment.