Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pull out common pieces of yaml in the buildkite pipeline #976

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
+93 −104
Open

pull out common pieces of yaml in the buildkite pipeline #976

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d13f854
can these be pulled out into yaml references?
toshok Nov 22, 2023
257a9bc
this may be a bit too far
toshok Nov 22, 2023
d57effb
remove the agents key from the *_agent_queue anchors, so they can be …
toshok Nov 22, 2023
10f2b66
remove the --- that yamllint was convinced we needed
toshok Nov 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
197 changes: 93 additions & 104 deletions .buildkite/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,59 @@
env:
FORCE_COLOR: 1
GIT_TERMINAL_PROMPT: 0
queues:
- &chromium_queue
runtimeType: chromiumbuild
queue: runtime
- &linux_agent_queue
<<: *chromium_queue
os: linux
- &windows_agent_queue
<<: *chromium_queue
os: windows
- &mac_agent_queue
<<: *chromium_queue
os: macos

steps:
- label: "Build chromium - Linux"
key: "build-chromium-linux-x86_64"
timeout_in_minutes: 60
plugins:
- &aws_sm_plugin
seek-oss/aws-sm#v2.3.1:
region: us-east-2
env:
BUILDEVENT_APIKEY: honeycomb-api-key
BUILDEVENT_BUILDKITE_API_TOKEN: buildkite-api-token-honeycomb-build-events
- &buildevents_plugin
replayio/buildevents#adb8a05: ~
Comment on lines +16 to +23
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally I would be able to express this as:

- &buildevents_plugins
  - seek-oss/aws-sm#v2.3.1:
      region: us-east-2
      env:
        BUILDEVENT_APIKEY: honeycomb-api-key
        BUILDEVENT_BUILDKITE_API_TOKEN: buildkite-api-token-honeycomb-build-events
  - replayio/buildevents#adb8a05: ~

and then reference the anchored list below like:

  - &linux_plugins
    plugins:
      - thedyrt/skip-checkout#v0.1.1:
          cd: /home/ubuntu/chromium/src
      <<: *buildevents_plugins

but alas, that isn't valid yaml. instead each plugin needs its own anchor and those get mentioned individually in the os plugins lists.


- &linux_plugins
plugins:
- thedyrt/skip-checkout#v0.1.1:
cd: /home/ubuntu/chromium/src
- seek-oss/aws-sm#v2.3.1:
region: us-east-2
env:
BUILDEVENT_APIKEY: honeycomb-api-key
BUILDEVENT_BUILDKITE_API_TOKEN: buildkite-api-token-honeycomb-build-events
- replayio/buildevents#adb8a05: ~
- *aws_sm_plugin
- *buildevents_plugin
- &mac_plugins
plugins:
- thedyrt/skip-checkout#v0.1.1:
cd: /Users/administrator/chromium/src
- *aws_sm_plugin
- *buildevents_plugin
- &windows_plugins
plugins:
- https://github.com/jazzdan/skip-checkout-buildkite-plugin.git#jazzdan/windows-support: ~

environments:
- &goma_env
GOMA_SERVER_HOST: simpsonite.goma.engflow.com
GOMACTL_USE_PROXY: false
- &linux_env
<<: *goma_env
RECORD_REPLAY_BACKEND_DIR: /home/ubuntu/chromium/backend
- &mac_env
<<: *goma_env
RECORD_REPLAY_BACKEND_DIR: /Users/administrator/chromium/backend
- &windows_env
<<: *goma_env
RECORD_REPLAY_BACKEND_DIR: "C:\\Users\\Administrator\\chromium\\backend"

commands:
- &buck2_build
commands:
- "be_cmd git-fetch-all -- git fetch --all"
- "be_cmd git-reset-branch -- git reset --hard origin/$BUILDKITE_BRANCH"
Expand All @@ -23,28 +62,7 @@ steps:
- "be_cmd buck2-kill -- buck2 kill"
- "be_cmd buck2-build -- buck2 build --console simple //:chromium"
- "popd"
env:
GOMA_SERVER_HOST: simpsonite.goma.engflow.com
GOMACTL_USE_PROXY: false
RECORD_REPLAY_BACKEND_DIR: /home/ubuntu/chromium/backend
agents:
- "runtimeType=chromiumbuild"
- "os=linux"
- "queue=runtime"
artifact_paths:
- "build_id/linux/x86_64/**"
- label: "Build chromium - Mac (x86)"
key: "build-chromium-mac-x86_64"
timeout_in_minutes: 60
plugins:
- thedyrt/skip-checkout#v0.1.1:
cd: /Users/administrator/chromium/src
- seek-oss/aws-sm#v2.3.1:
region: us-east-2
env:
BUILDEVENT_APIKEY: honeycomb-api-key
BUILDEVENT_BUILDKITE_API_TOKEN: buildkite-api-token-honeycomb-build-events
- replayio/buildevents#adb8a05: ~
- &buckle_build
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

q: is there a reason not to use buckle everywhere? that would make it so we can reuse the same build command sequence across linux/macos.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the buck2/buckle build anchors are identical except for two lines. in fact except for two words. This is another place where yaml list merging (and string concatenation) would help a bunch. I didn't really want to create anchors for each shared command here, like I did up in plugins.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eventually we'll be able to use buckle everywhere, but there's currently a bug in the Windows version benbrittain/buckle#21

commands:
- "be_cmd git-fetch-all -- git fetch --all"
- "be_cmd git-reset-branch -- git reset --hard origin/$BUILDKITE_BRANCH"
Expand All @@ -53,86 +71,68 @@ steps:
- "be_cmd buck2-kill -- buckle kill"
- "be_cmd buck2-build -- buckle build --console simple //:chromium"
- "popd"
env:
GOMA_SERVER_HOST: simpsonite.goma.engflow.com
GOMACTL_USE_PROXY: false
RECORD_REPLAY_BACKEND_DIR: /Users/administrator/chromium/backend
agents:
- "runtimeType=chromiumbuild"
- "os=macos"
- "queue=runtime"


env:
FORCE_COLOR: 1
GIT_TERMINAL_PROMPT: 0

steps:
- label: "Build chromium - Linux"
key: "build-chromium-linux-x86_64"
timeout_in_minutes: 60
agents: *linux_agent_queue
env: *linux_env
<<: *linux_plugins
<<: *buck2_build
artifact_paths:
- "build_id/linux/x86_64/**"

- label: "Build chromium - Mac (x86)"
key: "build-chromium-mac-x86_64"
timeout_in_minutes: 60
agents: *mac_agent_queue
env: *mac_env
<<: *mac_plugins
<<: *buckle_build
artifact_paths:
- "build_id/macOS/x86_64/**"

- label: "Build chromium - Mac (arm64)"
key: "build-chromium-mac-arm64"
timeout_in_minutes: 60
plugins:
- thedyrt/skip-checkout#v0.1.1:
cd: /Users/administrator/chromium/src
- seek-oss/aws-sm#v2.3.1:
region: us-east-2
env:
BUILDEVENT_APIKEY: honeycomb-api-key
BUILDEVENT_BUILDKITE_API_TOKEN: buildkite-api-token-honeycomb-build-events
- replayio/buildevents#adb8a05: ~
commands:
- "be_cmd git-fetch-all -- git fetch --all"
- "be_cmd git-reset-branch -- git reset --hard origin/$BUILDKITE_BRANCH"
- "be_cmd update-all-repos -- node replay_build_scripts/update-all-repos.mjs"
- "pushd ../backend"
- "be_cmd buck2-kill -- buckle kill"
- "be_cmd buck2-build -- buckle build --console simple //:chromium"
- "popd"
env:
GOMA_SERVER_HOST: simpsonite.goma.engflow.com
GOMACTL_USE_PROXY: false
agents: *mac_agent_queue
env:
<<: *mac_env
REPLAY_BUILD_ARM: true
RECORD_REPLAY_BACKEND_DIR: /Users/administrator/chromium/backend
agents:
- "runtimeType=chromiumbuild"
- "os=macos"
- "queue=runtime"
<<: *mac_plugins
<<: *buckle_build
artifact_paths:
- "build_id/macOS/arm64/**"

- label: "Build chromium - Windows"
key: "build-chromium-windows"
timeout_in_minutes: 60
plugins:
- https://github.com/jazzdan/skip-checkout-buildkite-plugin.git#jazzdan/windows-support: ~
agents: *windows_agent_queue
env: *windows_env
<<: *windows_plugins
command: "cd C:\\Users\\Administrator\\chromium\\src && git fetch --all && git reset --hard origin/$BUILDKITE_BRANCH && node replay_build_scripts/update-all-repos.mjs && node buildWindows.mjs && node replay_build_scripts/upload_build_artifacts.mjs"
env:
GOMA_SERVER_HOST: simpsonite.goma.engflow.com
GOMACTL_USE_PROXY: false
RECORD_REPLAY_BACKEND_DIR: "C:\\Users\\Administrator\\chromium\\backend"
agents:
- "runtimeType=chromiumbuild"
- "os=windows"
- "queue=runtime"
artifact_paths:
- "build_id/windows/x86_64/**"

- trigger: "testing-runtime-e2e"
label: ":hammer: Trigger Runtime Tests"
depends_on:
- "build-chromium-linux-x86_64"
- "build-chromium-mac-x86_64"
- "build-chromium-mac-arm64"

- label: "Metabase Test Suite"
key: "metabase-test-suite"
agents: *linux_agent_queue
<<: *linux_plugins
depends_on: "build-chromium-linux-x86_64"
if: build.branch == "master"
agents:
- "runtimeType=chromiumbuild"
- "os=linux"
- "queue=runtime"
plugins:
- thedyrt/skip-checkout#v0.1.1: ~
- seek-oss/aws-sm#v2.3.1:
region: us-east-2
env:
GITHUB_AUTH_SECRET: "prod/metabase-github-secret"
BUILDEVENT_APIKEY: honeycomb-api-key
BUILDEVENT_BUILDKITE_API_TOKEN: buildkite-api-token-honeycomb-build-events
- replayio/buildevents#adb8a05: ~
command: "be_cmd metabase-tests -- /home/ubuntu/chromium/src/replay_build_scripts/metabase.sh"

# wait for all steps above, but also continue if they fail
Expand All @@ -141,17 +141,6 @@ steps:

- label: "Buildevents Watch"
key: "buildevents-watch"
plugins:
- thedyrt/skip-checkout#v0.1.1:
cd: /home/ubuntu/chromium/src
- seek-oss/aws-sm#v2.3.1:
region: us-east-2
env:
BUILDEVENT_APIKEY: honeycomb-api-key
BUILDEVENT_BUILDKITE_API_TOKEN: buildkite-api-token-honeycomb-build-events
- replayio/buildevents#adb8a05: ~
agents: *linux_agent_queue
<<: *linux_plugins
command: "be_watch"
agents:
- "runtimeType=chromiumbuild"
- "os=linux"
- "queue=runtime"