You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
One of our worker nodes went into disk pressure and took CI with it.
Took CI with it? The actions runner controller scheduled a builder on the node with disk pressure. It went up fine, but was killed shortly before starting the build.
Why disk pressure? The shared docker directory blew up to 642G.
Quick fix: stop scheduling on the node. Attach a dockerd pod and run docker system prune:
Total reclaimed space: 292.2GB
Ideally this is a cronjob and a bit smarter. The prune also deleted shared resources that make the build faster.
The text was updated successfully, but these errors were encountered:
One of our worker nodes went into disk pressure and took CI with it.
Took CI with it? The actions runner controller scheduled a builder on the node with disk pressure. It went up fine, but was killed shortly before starting the build.
Why disk pressure? The shared docker directory blew up to 642G.
Quick fix: stop scheduling on the node. Attach a dockerd pod and run
docker system prune
:Total reclaimed space: 292.2GB
Ideally this is a cronjob and a bit smarter. The prune also deleted shared resources that make the build faster.
The text was updated successfully, but these errors were encountered: