Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add explicit queues to buildkite agents #12

Open
avsm opened this issue Jun 12, 2018 · 1 comment
Open

add explicit queues to buildkite agents #12

avsm opened this issue Jun 12, 2018 · 1 comment
Assignees
Labels
buildkite Buildkite infrastructure

Comments

@avsm
Copy link
Member

avsm commented Jun 12, 2018

Right now, we use the default Buildkite queue for every job in the infrastructure. A big bulk build will use up all the slots for an architecture, resulting in everything else being queued up.

To solve this, we need the following queues:

  • bulk: for bulk builds
  • infra: for infrastructure containers
  • default: misc stuff

Unfortunately, there is no way to prioritise queues, so we need to overcommit them in the agents. This might mean that if everything is maxxed out, we are pushing heavy load averages. However, this will be slow, but not a fatal error.

There is some discussion on buildkite/feedback#147 about improving this situation upstream in Buildkite, but for now overcommit should be fine for us.

@avsm avsm added the buildkite Buildkite infrastructure label Jun 12, 2018
@avsm avsm self-assigned this Jun 12, 2018
@petemounce
Copy link

We addressed this by auto-scaling our agent nodes up and down according to metrics from buildkite/buildkite-metrics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
buildkite Buildkite infrastructure
Projects
None yet
Development

No branches or pull requests

2 participants