Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(pipeline) : Montées de version airflow, sqlfluff, tuning valeurs par défaut #262

Merged
merged 4 commits into from
Jul 31, 2024

Conversation

vperron
Copy link
Contributor

@vperron vperron commented Jul 25, 2024

Tout est dans le titre et les commits.

Rien de spécial ici.

vperron and others added 3 commits July 29, 2024 09:28
The previous version wasn't compatible with DBT 1.8 anymore.
We can use more sensible defaults as Airflow still feels like eating all
of my CPU for no good reason.

We don't have a million DAGs and don't run them all in the same time so
no need for a task pool of 128 units. No more than 4 CPU units is
enough (parallelism is 32 by default)

Increase the DAG parsing processes to 4 to accelerate the parsing time
(default is 2)

https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/scheduler.html
Copy link
Contributor

@vmttn vmttn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️ J'ai vérifié les changelogs pour quelques montées de versions majors dans les dépendances signifacitives, seems good
✂️ j'ai découpé ta PR en deux : la montée de version + les "perfs" d'airflow dans cette PR. Les autres changements dans #263
⏭️ il faudrait check 1 ou 2 dags en staging

@hlecuyer hlecuyer merged commit fb95217 into main Jul 31, 2024
9 checks passed
@hlecuyer hlecuyer deleted the vperron/chores branch July 31, 2024 12:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants