[BUG] Database snapshots creation jobs are timing out

[mrchrisadams]

The daily job we have to create a daily snapshot, and upload the compressed version to object storage is currently timing out, which means the actual database snapshot is not being uploaded in any meaningful form.

The data is in our database still, but the snapshot does not contain useful data.

Linked below is an example run:

https://github.com/thegreenwebfoundation/admin-portal/runs/7333125077?check_suite_focus=true

If the link above does not work, TLDR version is that the hosting agent running the job is timing out, and things are not working as they should.

The job running on runner Hosted Agent has exceeded the maximum execution time of 10 minutes.

Fixing this

This naturally is a thing we need to fix.

There are a few solutions from running the job ourselves on our own infrastructure runners again, all the way to simply seeing if we can pay for minutes of github time instead of relying on the free plan.

I'm not sure what is the easiest solution, and I'm not sure if paying for minutes removes the timeout limit that is causing this problem.

We'd need to investigate.

Some links below

https://docs.github.com/en/billing/managing-billing-for-github-actions
https://stackoverflow.com/questions/68187987/github-actions-job-timeout
https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners

[mrchrisadams]

Hey @roald-teunissen - we discussed before doing some work with github actions and periodic jobs. I've assigned this to you for us to talk about on our next call.

[mrchrisadams]

I've replaced this with a regular cronjob for now, and today's snapshot is working as expected again.

There didn't seem to be an obvious way to just pay for compute credits as a way to get around the timeout issue we saw.

@roald-teunissen I'm going to leave this open for now, as I've spoken to Ross about having this one of the next jobs to put into the nomad scheduler.