Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Automate pruning shared docker resources #756

Closed
irq0 opened this issue Oct 12, 2023 · 3 comments
Closed

CI: Automate pruning shared docker resources #756

irq0 opened this issue Oct 12, 2023 · 3 comments
Labels
area/CI Continuous Integration triage/waiting Waiting for triage

Comments

@irq0
Copy link
Contributor

irq0 commented Oct 12, 2023

One of our worker nodes went into disk pressure and took CI with it.

Took CI with it? The actions runner controller scheduled a builder on the node with disk pressure. It went up fine, but was killed shortly before starting the build.

Why disk pressure? The shared docker directory blew up to 642G.

Quick fix: stop scheduling on the node. Attach a dockerd pod and run docker system prune:
Total reclaimed space: 292.2GB

Ideally this is a cronjob and a bit smarter. The prune also deleted shared resources that make the build faster.

@github-project-automation github-project-automation bot moved this to Backlog in S3GW Oct 12, 2023
@irq0 irq0 added the area/CI Continuous Integration label Oct 12, 2023
@github-actions github-actions bot added the triage/waiting Waiting for triage label Oct 12, 2023
@jecluis
Copy link
Contributor

jecluis commented Oct 19, 2023

@Ameenasuhani is this something you can tackle?

@tserong
Copy link
Contributor

tserong commented Oct 19, 2023

Just hit (what I assume is) this again in https://github.com/aquarist-labs/ceph/actions/runs/6572126718/job/17853215667?pr=233

@jecluis
Copy link
Contributor

jecluis commented Mar 20, 2024

there's already a workflow that does this

@jecluis jecluis closed this as completed Mar 20, 2024
@github-project-automation github-project-automation bot moved this from Backlog to Done in S3GW Mar 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/CI Continuous Integration triage/waiting Waiting for triage
Projects
None yet
Development

No branches or pull requests

3 participants