Skip to content
This repository has been archived by the owner on Sep 28, 2022. It is now read-only.

jkleinkauff/trips-etl

Folders and files

NameName
Last commit message
Last commit date

Latest commit

c092fea · Feb 4, 2022

History

7 Commits
Jan 24, 2022
Jan 25, 2022
Jan 24, 2022
Jan 24, 2022
Jan 24, 2022
Jan 24, 2022
Jan 25, 2022
Jan 24, 2022
Feb 4, 2022
Jan 24, 2022
Jan 25, 2022
Jan 24, 2022
Jan 24, 2022

Repository files navigation

How to start the project

In your /docker/.env file, add the following variables:

DATA_DIR="/.../<cloned-dir>/src/etl_data"
DAGS_DIR="/.../<cloned-dir>/dags/data"

Execute the following commands:

make build
make build-up

The following services should start:

➜  project git:(master) ✗ docker ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED        STATUS                   PORTS                              NAMES
357b712341f4   apache/airflow:2.2.3         "/usr/bin/dumb-init …"   5 hours ago    Up 5 hours               8080/tcp                           xxx_airflow-scheduler_1
372d440f1b6a   apache/airflow:2.2.3         "/usr/bin/dumb-init …"   5 hours ago    Up 5 hours (healthy)     0.0.0.0:5555->5555/tcp, 8080/tcp   xxx_flower_1
77eadeb3c0aa   apache/airflow:2.2.3         "/usr/bin/dumb-init …"   5 hours ago    Up 5 hours (healthy)     0.0.0.0:8080->8080/tcp             xxx_airflow-webserver_1
7f68845efbc3   apache/airflow:2.2.3         "/usr/bin/dumb-init …"   5 hours ago    Up 5 hours               8080/tcp                           xxx_airflow-worker_1
af9fb24ad6ff   postgres:13                  "docker-entrypoint.s…"   5 hours ago    Up 5 hours (healthy)     5432/tcp                           xxx_postgres_1
4dadbd44de13   postgres                     "docker-entrypoint.s…"   5 hours ago    Up 5 hours               0.0.0.0:5433->5432/tcp             xxx_db_app_1
432837c58485   redis:latest                 "docker-entrypoint.s…"   5 hours ago    Up 5 hours (healthy)     0.0.0.0:6379->6379/tcp             superset_cache
04ee1864109a   apache/superset:latest-dev   "/app/docker/docker-…"   14 hours ago   Up 7 hours (unhealthy)   8088/tcp                           superset_worker
8254b5f3d984   apache/superset:latest-dev   "/app/docker/docker-…"   14 hours ago   Up 7 hours (unhealthy)   8088/tcp                           superset_worker_beat
b35323ed9f27   apache/superset:latest-dev   "/app/docker/docker-…"   14 hours ago   Up 7 hours (healthy)     0.0.0.0:8088->8088/tcp             superset_app
aeefa7a99c4e   postgres:10                  "docker-entrypoint.s…"   14 hours ago   Up 7 hours               5432/tcp                           superset_db

Airflow: localhost:8080 - airflow@airflow

Superset: localhost:8088 - admin@admin

Airflow UI:

etl-dm-ddl-dag: should be executed just one time, then it could be turned off

etl-ingestion-dag:

schedule internal: every 15 minutes
keep on

etl-process-ingested-dag:

schedule internal: every 30 minutes
keep on

etl-load-dm-dag:

schedule internal: every 60 minutes
keep on

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published