Should we use CronJob instead of Deployments for sync scripts?

[aodj]

Checks

• I have checked for existing issues.
• I have looked through the FAQ.

Question

Has it ever been discussed to use CronJobs rather than Deployments for the long running scripts? When I'm using all the available features I'm running a full Python pod for db-migrations, sync-connections, sync-pools, sync-users, sync-variables, in addition to any web and scheduler replicas.

Collectively everything uses about 2 GB (not including web and scheduler pods) which is a lot for a bunch of processes which spend the majority of their time asleep.

Using CronJobs would allow the same interval based sync without the cost of running the pods forever.

[thesuperzapper]

@aodj this is an interesting idea, but please realize that sync-connections, sync-pools, sync-users, sync-variables are constantly reverting any changes people make using the Airflow UI, which has the benefit of forcing you to be declarative with your connections/pools/users/variables (and not discover weeks later during your next helm upgrade that someone has updated a connection's password but not updated the values for it).

I worry that including lots of CronJobs is worse for a few reasons:

1. Updates to Secret/ConfigMap referenced in airflow.connectionsTemplates won't be updated as quickly (right now we do this every 10 seconds!)
2. with very frequent schedules (like every 60s) lots of time will be wasted with each CronJobs Pod startup

If you are really concerned about resources, and don't want the benefits that come from automatically updating/reverting the connections/pools/users/variables automatically, you can set:

• airflow.usersUpdate = false
• airflow.connectionsUpdate = false
• airflow.variablesUpdate = false
• airflow.poolsUpdate = false
• airflow.dbMigrations.runAsJob = true

[aodj]

You are correct that with fast jobs that run frequently there would be a cost paid in terms of starting and stopping, and I hadn't dug into the code enough to identify the different frequencies. 10s is very fast, and my gut feel is that it might be too fast for most use cases but as always YMMV.

I did see the one-shot alternative you mention, but I still feel as if there's some middle ground that would allow a lighter footprint without losing the declarative nature of the current solution.

Thanks for your input, I'll put my thinking cap on ;)