Next-gen data orchestration
Dagster is our data orchestrator. Every ETL step takes place here.
Dagster Cloud is a hosted front-end for our Dagster servers where you can observe and run integration jobs.
Dagster hosts multiple "code locations", one for each of our business units, including a separate one for our CMO:
- kippnewark
- kippcamden
- kippmiami
- kipptaf
Each code location hosts and runs the code and configurations for each respective business unit. Behind-the-scenes, these are containers run on Google Cloud Kubernetes. Each code location has it's own respective jobs, schedules, sensors, and assets.
Before you merge:
- Ensure dbt build runs successfully on your branch
- Format your SQL changes in dbt
- Ensure the Dagster build action runs successfully
- Private GKE Autopilot cluster
- Cloud NAT provided static external IP for the cluster
- Google Artifact Registry
- Google Cloud services access prodivded by Workload Identity
- GitHub Actions for CI/CD