dagster instance migrate - Why not running it all the time?

[AndreaGiardini]

Hi folks 👋

I would like to bring you attention to the command dagster instance migrate that, following your guide ( https://docs.dagster.io/deployment/guides/kubernetes/how-to-migrate-your-instance#step-3-migrate-the-dagster-instance) , needs to be run to add necessary indexes and improvements. I am asking myself... why not running it by default?

We could have a simple kubernetes job that "does nothing" if the change have been applied and applies the changes if there is something to be applied. The current process is quite manual, error-prone and involves a dagster downtime. Other apps like airbyte do this already (see their airbyte-bootstrap job in the helm chart). Am I missing something?

[schrockn]

Hey @AndreaGiardini thanks for your question.

Database migrations are by their nature expensive, dangerous. and destructive operations. We don't know if users have customized their own schema, have written tools that depend on that schema, and so forth. We can't just run this by default for production services.

I could see an argument for running this by default on dagster dev invocations.

[AndreaGiardini]

I could see an argument for running this by default on dagster dev invocations.

Not sure that's a good thing. People could end up having a different DB structure between their local env and their prod env

[AndreaGiardini]

Hi @schrockn and thank you for the answer!

I understand your point of view, but maybe it's worth letting users make the choice? It wouldn't cross my mind to interfere with the dagster db TBH. Currently, we are experiencing performance issues on our database, and we are unsure if the migrations could improve the issue or make it worse (is it adding an index? or what? It's unclear what those extra migrations do). See screenshot attached (this is a CloudSQL query report of 10 minutes). Having every instance "on the same page" could prevent performance skews of this type across instances. Do you have many clients modifying the dagster db directly?

[schrockn]

If you want this behavior could you just just add a "dagster instance migrate" command to your helm chart/deployment process before you start dagster services?

[leejlFG]

We have a shell op in an infrastructure-only code location so we can run dagster instance migrate with the click of a button when necessary. The database itself is locked down tightly - no one has permissions to interact with it directly, but the production deployment service account does. We have thought about moving it elsewhere to Github Actions or something but there is no urgency. We would however love a flag in the release notes that tells us when we should run it, instead of going on vibes only or waiting for new features to fail on account of an old backend.