[Merged by Bors] - Self hosted docker builds #4592

realbigsean · 2023-08-09T20:17:43Z

Issue Addressed

We're OOM'ing on Docker builds on the Deneb branch #3929

Are we ok to self host automated docker builds?

.github/workflows/docker.yml

realbigsean · 2023-08-10T13:53:09Z

I'm going to test this out on the deneb branch #4599

realbigsean · 2023-08-10T16:21:48Z

Ok looks like it's failing, my guess is due to permissions. I'm gonna give this a shot (looks like there's better support for this type of thing nowadays): https://docs.docker.com/build/ci/github-actions/multi-platform/

realbigsean · 2023-08-10T16:23:39Z

We must go deeper: docker/setup-qemu-action#67

* self hosted docker builds attempted fix * fix env var references in docker builds

realbigsean · 2023-08-10T17:50:54Z

This looks promising on the deneb branch following the guidance of the previously linked issue. Here's the action where it's running https://github.com/sigp/lighthouse/actions/runs/5824054179/job/15792273258

One thing to note is that this doesn't currently include --provenance=false, but I can add it if we want it. I'm not sure why it was there, maybe to speed things up? It looks like the build and push step is something like 4 minutes slower without that flag.

realbigsean · 2023-08-10T19:54:27Z

Obviosly jinxed the last one. Looks like it failed on creation of the multiarch image. From what I can gather, we're using an old way to do this: docker manifest, where we can now use docker buildx imagetools create. Testing this again on deneb https://github.com/sigp/lighthouse/actions/runs/5825468485

* self hosted docker builds attempted fix * use imagetools instead of docker manifest

realbigsean · 2023-08-10T21:50:27Z

It worked! 🎉 https://hub.docker.com/layers/sigp/lighthouse/deneb/images/sha256-9f56a6534cb183a4a568b095d8e3ee9b1c558674e5709abef681764f23ed4d5d?context=explore

paulhauner · 2023-08-16T04:45:19Z

This is really cool, but I think we should do the Docker builds on the new release-only runner that @antondlr is working on.

I am concerned about using the testing runners for building binaries (or Docker images) that are expected to run in production.

I've also experienced the ARM OOMs, generally it works after a few tries. It's a crappy solution I know, but I think we should have the release-only boxes soon!

antondlr · 2023-08-16T05:36:01Z

One thing to note is that this doesn't currently include --provenance=false, but I can add it if we want it. I'm not sure why it was there, maybe to speed things up? It looks like the build and push step is something like 4 minutes slower without that flag.

the flag was introduced here; tldr: buildx v0.10.0 added provenance attestations to images but they are packed in a way that's incompatible with docker manifest

antondlr · 2023-08-16T05:40:53Z

This is really cool, but I think we should do the Docker builds on the new release-only runner that @antondlr is working on.

agree; I'll try to get some runners up Soon ™️ , iirc box is basically ready and idling
only the weird windows setup is still pending (I have to copy over a large file from age's network for that)

realbigsean · 2023-08-16T16:19:13Z

This is really cool, but I think we should do the Docker builds on the new release-only runner that @antondlr is working on.

makes sense!

the flag was introduced #3902; tldr: buildx v0.10.0 added provenance attestations to images but they are packed in a way that's incompatible with docker manifest

Since switching to imagetools instead of manifest seems to make it work without that flag, I guess I'll leave it off.

I'll leave this PR open and switch it to the release runner whenever it's ready

jimmygchen

Looks good 👍
It's been working well for Deneb as well (using the dev runners though).

antondlr · 2023-09-20T06:54:55Z

the DinD instance has this buildx builder pre-set as default:
(installed through docker run --privileged tonistiigi/binfmt --install all)

runner$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT             PLATFORMS
default * docker
  default default         running v0.11.6+0a15675913b7 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386

so I think we can skip those 2 steps on self-hosted

paulhauner

Nice work, looks good to me! Keen to see it in action for v4.5.0!

michaelsproul · 2023-09-20T23:50:02Z

bors r+

## Issue Addressed We're OOM'ing on Docker builds on the Deneb branch #3929 Are we ok to self host automated docker builds? Co-authored-by: realbigsean <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: antondlr <[email protected]>

bors · 2023-09-20T23:56:57Z

Build failed (retrying...):

release-tests-ubuntu

## Issue Addressed We're OOM'ing on Docker builds on the Deneb branch #3929 Are we ok to self host automated docker builds? Co-authored-by: realbigsean <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: antondlr <[email protected]>

bors · 2023-09-21T00:04:00Z

Build failed (retrying...):

release-tests-ubuntu

## Issue Addressed We're OOM'ing on Docker builds on the Deneb branch #3929 Are we ok to self host automated docker builds? Co-authored-by: realbigsean <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: antondlr <[email protected]>

michaelsproul · 2023-09-21T00:26:28Z

bors r-
bors r+

bors · 2023-09-21T00:26:40Z

Canceled.

## Issue Addressed We're OOM'ing on Docker builds on the Deneb branch #3929 Are we ok to self host automated docker builds? Co-authored-by: realbigsean <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: antondlr <[email protected]>

bors · 2023-09-21T01:59:03Z

Build failed (retrying...):

release-tests-windows

michaelsproul · 2023-09-21T02:00:40Z

bors r-

bors · 2023-09-21T02:00:43Z

Canceled.

michaelsproul · 2023-09-21T02:01:01Z

bors r+

## Issue Addressed We're OOM'ing on Docker builds on the Deneb branch #3929 Are we ok to self host automated docker builds? Co-authored-by: realbigsean <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: antondlr <[email protected]>

bors · 2023-09-21T02:48:31Z

Build failed (retrying...):

release-tests-ubuntu

## Issue Addressed We're OOM'ing on Docker builds on the Deneb branch #3929 Are we ok to self host automated docker builds? Co-authored-by: realbigsean <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: antondlr <[email protected]>

bors · 2023-09-21T05:17:26Z

Pull request successfully merged into unstable.

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

## Issue Addressed Related to #4676. Deneb-specifc CI code to be removed before merging to `unstable`. Dot not merge until we're ready to merge into `unstable`, as we may need to release deneb docker images before merging. Keep in mind that most of the changes in the below PR (to `unstable`) have already been merged to `deneb-free-blobs`, so merging `deneb-free-blobs` into `unstable` would include those changes - it would be ok if the release runners are ready, otherwise we may want to exclude them before merging. - #4592

## Issue Addressed Related to sigp#4676. Deneb-specifc CI code to be removed before merging to `unstable`. Dot not merge until we're ready to merge into `unstable`, as we may need to release deneb docker images before merging. Keep in mind that most of the changes in the below PR (to `unstable`) have already been merged to `deneb-free-blobs`, so merging `deneb-free-blobs` into `unstable` would include those changes - it would be ok if the release runners are ready, otherwise we may want to exclude them before merging. - sigp#4592

## Issue Addressed We're OOM'ing on Docker builds on the Deneb branch sigp#3929 Are we ok to self host automated docker builds? Co-authored-by: realbigsean <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: antondlr <[email protected]>

self hosted docker builds

2e4ead6

realbigsean added the ready-for-review The code is ready for review label Aug 9, 2023

realbigsean commented Aug 9, 2023

View reviewed changes

.github/workflows/docker.yml Outdated Show resolved Hide resolved

.github/workflows/docker.yml Outdated Show resolved Hide resolved

realbigsean added the infra-ci label Aug 9, 2023

remove self hosted runner check where it might not be needed

0945e4d

realbigsean added 2 commits August 10, 2023 13:46

self hosted docker builds attempted fix (sigp#4603)

815ab8c

Fix env var references (sigp#4604)

70d721b

* self hosted docker builds attempted fix * fix env var references in docker builds

Fix manifest lists (sigp#4605)

1b1fd8c

* self hosted docker builds attempted fix * use imagetools instead of docker manifest

realbigsean added blocked and removed ready-for-review The code is ready for review labels Aug 16, 2023

switch from topaz to production builder

e871f8c

antondlr removed the blocked label Aug 31, 2023

jimmygchen mentioned this pull request Sep 3, 2023

[Merged by Bors] - Deneb review .github (CI cleanup) #4696

Closed

michaelsproul added the v4.5.0 ETA Q4 2023 label Sep 20, 2023

jimmygchen added the ready-for-review The code is ready for review label Sep 20, 2023

jimmygchen approved these changes Sep 20, 2023

View reviewed changes

multiarch buildx instance already present on SH

f3f73cd

correct syntax mistake

eb0b9da

paulhauner approved these changes Sep 20, 2023

View reviewed changes

paulhauner added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Sep 20, 2023

bors bot changed the title ~~Self hosted docker builds~~ [Merged by Bors] - Self hosted docker builds Sep 21, 2023

bors bot closed this Sep 21, 2023

realbigsean deleted the self-hosted-docker-builds branch November 21, 2023 16:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - Self hosted docker builds #4592

[Merged by Bors] - Self hosted docker builds #4592

realbigsean commented Aug 9, 2023

realbigsean commented Aug 10, 2023

realbigsean commented Aug 10, 2023

realbigsean commented Aug 10, 2023

realbigsean commented Aug 10, 2023

realbigsean commented Aug 10, 2023

realbigsean commented Aug 10, 2023

paulhauner commented Aug 16, 2023

antondlr commented Aug 16, 2023

antondlr commented Aug 16, 2023

realbigsean commented Aug 16, 2023

jimmygchen left a comment

antondlr commented Sep 20, 2023

paulhauner left a comment

michaelsproul commented Sep 20, 2023

bors bot commented Sep 20, 2023

bors bot commented Sep 21, 2023

michaelsproul commented Sep 21, 2023

bors bot commented Sep 21, 2023

bors bot commented Sep 21, 2023

michaelsproul commented Sep 21, 2023

bors bot commented Sep 21, 2023

michaelsproul commented Sep 21, 2023

bors bot commented Sep 21, 2023

bors bot commented Sep 21, 2023

[Merged by Bors] - Self hosted docker builds #4592

[Merged by Bors] - Self hosted docker builds #4592

Conversation

realbigsean commented Aug 9, 2023

Issue Addressed

realbigsean commented Aug 10, 2023

realbigsean commented Aug 10, 2023

realbigsean commented Aug 10, 2023

realbigsean commented Aug 10, 2023

realbigsean commented Aug 10, 2023

realbigsean commented Aug 10, 2023

paulhauner commented Aug 16, 2023

antondlr commented Aug 16, 2023

antondlr commented Aug 16, 2023

realbigsean commented Aug 16, 2023

jimmygchen left a comment

Choose a reason for hiding this comment

antondlr commented Sep 20, 2023

paulhauner left a comment

Choose a reason for hiding this comment

michaelsproul commented Sep 20, 2023

bors bot commented Sep 20, 2023

bors bot commented Sep 21, 2023

michaelsproul commented Sep 21, 2023

bors bot commented Sep 21, 2023

bors bot commented Sep 21, 2023

michaelsproul commented Sep 21, 2023

bors bot commented Sep 21, 2023

michaelsproul commented Sep 21, 2023

bors bot commented Sep 21, 2023

bors bot commented Sep 21, 2023