Data Engineering Projects

Context

This repository is part of the class "Data Ingestion" given by Leandro Mendes Ferreira in the specialization Engenharia de Dados & Big Data – EAD – Síncrono/ao Vivo provided by PECE (Programa de Educação Continuada) from Escola Politécnica da USP.

The main idea is store the solution for the proposed problems during the classes using various Data Engineering tools.

Projects

Project 1 | ETL with Drag-And-Drop Orchestration Platform (Low Code)

The main challenge is ingest data from a CSV and API using a Visual/Drag-and-Drop orchestration platform, create a star-schema and display 3 graphs in a dashboard. Check the solution: here

Project 2 | ETL with Python and SQL only

The main challenge is ingest data from a CSV and API using only Python and SQL, create a star-schema and display 3 graphs in a dashboard. Check the solution: here

Project 3 | ETL with Apache Airflow and DBT only

The main challenge is ingest data from a CSV and API using Apache Airflow, DBT to create a star-schema and display 3 graphs in a dashboard. Check the solution: here

Project 4 | ETL with Apache Airflow and EMR

The main challenge is ingest data from a CSV and API using Apache Airflow and EMR to create a star-schema and display 3 graphs in a dashboard. Check the solution: here

Project 5 | ETL with Serverless Services (AWS)

The main challenge is ingest data from a CSV and API using only Serverless Services from AWS. Check the solution: here

Authors

• @joaoluga
• @jondrigues
• @viniabkt
• @andersonmanhaes
• @mariaramim

References:

• Docker
• Docker-compose
• Apache Hop
• Metabase
• Postgres
• Star Schema
• Diagram as code
• GraphViz
• Python 3+
• Jupyter
• Seaborn
• DBT
• Jupyter Kernel Gateway
• yq
• Terraform
• AWS | Elastic Container Service
• AWS | Fargate
• AWS | Application Load Balancer
• AWS | RDS
• AWS | Glue Crawler
• AWS | EMR
• AWS | Athena