From 9bd13255f286e4b7a654617268abe1b2f37c3e0a Mon Sep 17 00:00:00 2001 From: Techassi Date: Fri, 11 Oct 2024 10:29:09 +0200 Subject: [PATCH] fix: Add local scripts (#6) Co-authored-by: Nick <10092581+NickLarsenNZ@users.noreply.github.com> --- .scripts/get_manifest_digest.sh | 11 ++ README.md | 268 +----------------------------- build-container-image/README.md | 36 ++++ build-container-image/action.yml | 6 +- build-product-image/README.md | 54 ++++++ build-product-image/action.yml | 2 +- publish-image/README.md | 40 +++++ publish-image/action.yml | 12 +- publish-index-manifest/README.md | 29 ++++ publish-index-manifest/action.yml | 12 +- run-pre-commit/README.md | 48 ++++++ run-pre-commit/action.yml | 2 +- shard/README.md | 48 ++++++ 13 files changed, 299 insertions(+), 269 deletions(-) create mode 100755 .scripts/get_manifest_digest.sh create mode 100644 build-container-image/README.md create mode 100644 build-product-image/README.md create mode 100644 publish-image/README.md create mode 100644 publish-index-manifest/README.md create mode 100644 run-pre-commit/README.md create mode 100644 shard/README.md diff --git a/.scripts/get_manifest_digest.sh b/.scripts/get_manifest_digest.sh new file mode 100755 index 0000000..3871226 --- /dev/null +++ b/.scripts/get_manifest_digest.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash + +# Get a manifest digest. Example inputs: +# - docker.stackable.tech/stackable/hello-world:0.0.1-SNAPSHOT-stackable0.0.0-dev +# - docker.stackable.tech/stackable/hello-world:0.0.1-SNAPSHOT-stackable0.0.0-dev-amd64 +set -euo pipefail + +# Note: `docker manifest push` currently outputs the same hash, but `manifest` +# is experimental and the STDOUT is more likely to change than the structured +# output. +docker buildx imagetools inspect --format '{{println .Manifest.Digest}}' "$1" diff --git a/README.md b/README.md index 70f79b6..bc1eb44 100644 --- a/README.md +++ b/README.md @@ -17,263 +17,11 @@ particular step in a workflow. | Image Repo Digest | `docker.stackable.tech/stackable/kafka@sha256:917f800259ef4915f976...` | | Digest | `sha256:917f800259ef4915f976e93987b752fd64debf347568610d7f685d2022...` | -## `build-container-image` - -> Manifest: [build-container-image/action.yml][build-container-image] - -This action builds a *single* container image using `docker buildx build`. It does the following work: - -1. Free disk space to avoid running out of disk space during larger builds. -2. Build the image using `docker buildx build`, outputting the architecture specific tag. - -This action is considered to be the **single** source of truth regarding the image manifest tag. -All subsequent tasks must use this value to ensure consistency. - -### Inputs and Outputs - -> [!TIP] -> For descriptions of the inputs and outputs, see the complete [build-container-image] action. - -#### Inputs - -- `image-name` (eg: `kafka`) -- `image-index-manifest-tag` (eg: `3.4.1-stackable0.0.0-dev`) -- `container-file` (defaults to `Dockerfile`) -- `build-context` (defaults to `.`) - - -#### Outputs - -- `image-repository-uri` (eg: `localhost/kafka`) -- `image-manifest-tag` (eg: `3.4.1-stackable0.0.0-dev-amd64`) -- `image-manifest-uri` (eg: `localhost/kafka:3.4.1-stackable0.0.0-dev-amd64`) - -[build-container-image]: ./build-container-image/action.yml - -## `build-product-image` - -> Manifest: [build-product-image/action.yml][build-product-image] - - -> [!NOTE] -> The build step is not concerned with registries, ports, paths to repositories, but still requires -> a name. If the name does not contain a registry, `hub.docker.com` (?) is implied. Therefore, -> `localhost` will be used as the registry so as to avoid accidental interactions with an unintended -> registry. -> -> Ideally, bake should be refactored to use `localhost` as the registry for the previously mentioned -> reason (whether or not that is behind some option). - -This action builds a *single* container image using `bake`. It does the following work: - -1. Free disk space to avoid running out of disk space during larger builds. -2. Build the image using `bake` which internally uses `docker buildx`. -3. Temporarily retag the image to use `localhost` instead of `docker.stackable.tech/stackable`. -4. Produce output values to be used in later steps. - -This action is considered to be the **single** source of truth regarding image index tag and image -manifest tag. All subsequent tasks must use these values to ensure consistency. - -Currently, bake provides the following ouput in the `bake-target-tags` file: - -```plain -docker.stackable.tech/stackable/kafka:3.4.1-stackable0.0.0-dev-amd64 -``` - -Until bake supports the ability to specify the registry, this action will retag the image as: - -```plain -localhost/kafka:3.4.1-stackable0.0.0-dev-amd64 -``` - -### Inputs and Outputs - -> [!TIP] -> For descriptions of the inputs and outputs, see the complete [build-product-image] action. - -#### Inputs - -- `product-name` -- `product-version` -- `image-tools-version` -- `build-cache-username` -- `build-cache-password` - -#### Outputs - -- `image-manifest-tag` - -[build-product-image]: ./build-product-image/action.yml - -## `publish-image` - -> Manifest: [publish-image/action.yml][publish-image] - -This action signs and publishes a *single* container image to the given registry. It does the -following work: - -1. Tag the `source-image-uri` with the specified `image-registry-uti`, `image-repository`, and - `image-repository`. -2. Push the container image to the specified registry. -3. Sign the container image (which pushes the signature to the specified registry). -4. Generate an SBOM via a syft scan. -5. Attest an image with the SBOM as a predicate (which pushes the attestation to the specified - registry). - -### Inputs and Outputs - -> [!TIP] -> For descriptions of the inputs and outputs, see the complete [publish-image] action. - - -> [!IMPORTANT] -> For multi-arch images, the `image-manifest-tag` should have the `-$ARCH` suffix, as the tag -> without it should be reserved for the image index manifest which will refer to container images -> for each architecture we will push images for. - -#### Inputs - -- `image-registry-uri` -- `image-registry-username` -- `image-registry-password` -- `image-repository` -- `image-manifest-tag` -- `source-image-uri` - -#### Outputs - -None - -[publish-image]: ./publish-image/action.yml - -## `publish-index-manifest` - -> Manifest: [publish-index-manifest/action.yml][publish-index-manifest] - -This action creates an image index manifest, publishes it, and signs it. It does the following work: - -1. Create an image index manifest and link to each architecture in `image-architectures`. -2. Push the image index manifest. -3. Sign the image index manifest (which pushes the signature to the specified registry). - -### Inputs and Outputs - -> [!TIP] -> For descriptions of the inputs and outputs, see the complete [publish-index-manifest] action. - -#### Inputs - -- `image-registry-uri` -- `image-registry-username` -- `image-registry-password` -- `image-repository` -- `image-index-manifest-tag` -- `image-architectures` - -#### Outputs - -None - -[publish-index-manifest]: ./publish-index-manifest/action.yml - -## `run-pre-commit` - -> Manifest: [run-pre-commit/action.yml][run-pre-commit] - -This action runs pre-commit by setting up Python and optionally the Rust toolchain and Hadolint in -the requested version. It requires a checkout with depth 0. It does the following work: - -1. Installs Python. The version can be configured via the `python-version` input. -2. Optionally sets up the Rust toolchain and Hadolint. -3. Runs pre-commit on changed files. - -Example usage (workflow): - -```yaml ---- -name: pre-commit - -on: - pull_request: - -jobs: - pre-commit: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout - with: - fetch-depth: 0 - submodules: recursive - - uses: stackabletech/actions/run-pre-commit -``` - -### Inputs and Outputs - -> [!TIP] -> For descriptions of the inputs and outputs, see the complete [run-pre-commit] action. - -#### Inputs - -- `python-version` -- `rust` -- `rust-components` -- `hadolint` - -#### Outputs - -None - -[run-pre-commit]: ./run-pre-commit/action.yml - -## `shard` - -> Manifest: [shard/action.yml][shard] - -This action produces a list of versions for a product. This is to be used as a matrix dimension to -parallelize builds. It does the following work: - -1. Reads the `conf.py`, filtering versions for the product -2. Write the JSON array of version to `$GITHUB_OUTPUT` for use in a matrix. - -Example usage: - -```yaml -jobs: - generate_matrix: - name: Generate Version List - runs-on: ubuntu-latest - steps: - - uses: actions/checkout - - id: shard - uses: stackabletech/actions/shard - with: - product-name: ${{ env.PRODUCT_NAME }} - outputs: - versions: ${{ steps.shard.outputs.versions }} - - actual_matrix: - needs: [generate_matrix] - strategy: - matrix: - versions: ${{ fromJson(needs.generate_matrix.outputs.versions) }} - # ... -``` - -### Inputs and Outputs - -> [!TIP] -> For descriptions of the inputs and outputs, see the complete [shard] action. - -#### Inputs - -- `product-name` - -#### Outputs - -- `versions` - -[shard]: ./publish-index-manifest/action.yml +## Available Actions + +- [build-container-image](./build-container-image/README.md) +- [build-product-image](./build-product-image/README.md) +- [publish-image](./publish-image/README.md) +- [publish-index-manifest](./publish-index-manifest/README.md) +- [run-pre-commit](./run-pre-commit/README.md) +- [shard](./shard/README.md) diff --git a/build-container-image/README.md b/build-container-image/README.md new file mode 100644 index 0000000..879f6e7 --- /dev/null +++ b/build-container-image/README.md @@ -0,0 +1,36 @@ +# `build-container-image` + +> Manifest: [build-container-image/action.yml][build-container-image] + +This action builds a *single* container image using `docker buildx build`. It does the following work: + +1. Free disk space to avoid running out of disk space during larger builds. +2. Build the image using `docker buildx build`, outputting the architecture specific tag. + +This action is considered to be the **single** source of truth regarding the image manifest tag. +All subsequent tasks must use this value to ensure consistency. + +## Inputs and Outputs + +> [!TIP] +> For descriptions of the inputs and outputs, see the complete [build-container-image] action. + +### Inputs + +- `image-name` (eg: `kafka`) +- `image-index-manifest-tag` (eg: `3.4.1-stackable0.0.0-dev`) +- `container-file` (defaults to `Dockerfile`) +- `build-context` (defaults to `.`) + + +### Outputs + +- `image-repository-uri` (eg: `localhost/kafka`) +- `image-manifest-tag` (eg: `3.4.1-stackable0.0.0-dev-amd64`) +- `image-manifest-uri` (eg: `localhost/kafka:3.4.1-stackable0.0.0-dev-amd64`) + +[build-container-image]: ./action.yml diff --git a/build-container-image/action.yml b/build-container-image/action.yml index 5a23245..89ded41 100644 --- a/build-container-image/action.yml +++ b/build-container-image/action.yml @@ -85,13 +85,13 @@ runs: echo "IMAGE_ARCH=${IMAGE_ARCH}" | tee -a "$GITHUB_ENV" IMAGE_MANIFEST_TAG="${IMAGE_INDEX_MANIFEST_TAG}-${IMAGE_ARCH}" - echo "IMAGE_MANIFEST_TAG=${IMAGE_MANIFEST_TAG}" | tee -a $GITHUB_OUTPUT + echo "IMAGE_MANIFEST_TAG=${IMAGE_MANIFEST_TAG}" | tee -a "$GITHUB_OUTPUT" IMAGE_REPOSITORY_URI="localhost/${IMAGE_NAME}" - echo "IMAGE_REPOSITORY_URI=${IMAGE_REPOSITORY_URI}" | tee -a $GITHUB_OUTPUT + echo "IMAGE_REPOSITORY_URI=${IMAGE_REPOSITORY_URI}" | tee -a "$GITHUB_OUTPUT" IMAGE_MANIFEST_URI="${IMAGE_REPOSITORY_URI}:${IMAGE_MANIFEST_TAG}" - echo "IMAGE_MANIFEST_URI=${IMAGE_MANIFEST_URI}" | tee -a $GITHUB_OUTPUT + echo "IMAGE_MANIFEST_URI=${IMAGE_MANIFEST_URI}" | tee -a "$GITHUB_OUTPUT" echo "::group::docker buildx build" # TODO (@NickLarsenNZ): Allow optional buildx cache diff --git a/build-product-image/README.md b/build-product-image/README.md new file mode 100644 index 0000000..22793af --- /dev/null +++ b/build-product-image/README.md @@ -0,0 +1,54 @@ +# `build-product-image` + +> Manifest: [build-product-image/action.yml][build-product-image] + + +> [!NOTE] +> The build step is not concerned with registries, ports, paths to repositories, but still requires +> a name. If the name does not contain a registry, `hub.docker.com` (?) is implied. Therefore, +> `localhost` will be used as the registry so as to avoid accidental interactions with an unintended +> registry. +> +> Ideally, bake should be refactored to use `localhost` as the registry for the previously mentioned +> reason (whether or not that is behind some option). + +This action builds a *single* container image using `bake`. It does the following work: + +1. Free disk space to avoid running out of disk space during larger builds. +2. Build the image using `bake` which internally uses `docker buildx`. +3. Temporarily retag the image to use `localhost` instead of `docker.stackable.tech/stackable`. +4. Produce output values to be used in later steps. + +This action is considered to be the **single** source of truth regarding image index tag and image +manifest tag. All subsequent tasks must use these values to ensure consistency. + +Currently, bake provides the following ouput in the `bake-target-tags` file: + +```plain +docker.stackable.tech/stackable/kafka:3.4.1-stackable0.0.0-dev-amd64 +``` + +Until bake supports the ability to specify the registry, this action will retag the image as: + +```plain +localhost/kafka:3.4.1-stackable0.0.0-dev-amd64 +``` + +## Inputs and Outputs + +> [!TIP] +> For descriptions of the inputs and outputs, see the complete [build-product-image] action. + +### Inputs + +- `product-name` (eg: `kafka`) +- `product-version` (eg: `3.4.1`) +- `image-tools-version` (eg: `0.0.13`) +- `build-cache-username` (required) +- `build-cache-password` (required) + +### Outputs + +- `image-manifest-tag` (eg: `3.4.1-stackable0.0.0-dev-amd64`) + +[build-product-image]: ./action.yml diff --git a/build-product-image/action.yml b/build-product-image/action.yml index 9ad74f5..4c73361 100644 --- a/build-product-image/action.yml +++ b/build-product-image/action.yml @@ -113,4 +113,4 @@ runs: # Add the contents of the env variables to the GitHub output, so that it # can be used as action outputs - echo "IMAGE_MANIFEST_TAG=$IMAGE_MANIFEST_TAG" >> $GITHUB_OUTPUT + echo "IMAGE_MANIFEST_TAG=$IMAGE_MANIFEST_TAG" | tee -a "$GITHUB_OUTPUT" diff --git a/publish-image/README.md b/publish-image/README.md new file mode 100644 index 0000000..f1ff6bb --- /dev/null +++ b/publish-image/README.md @@ -0,0 +1,40 @@ +# `publish-image` + +> Manifest: [publish-image/action.yml][publish-image] + +This action signs and publishes a *single* container image to the given registry. It does the +following work: + +1. Tag the `source-image-uri` with the specified `image-registry-uti`, `image-repository`, and + `image-repository`. +2. Push the container image to the specified registry. +3. Sign the container image (which pushes the signature to the specified registry). +4. Generate an SBOM via a syft scan. +5. Attest an image with the SBOM as a predicate (which pushes the attestation to the specified + registry). + +## Inputs and Outputs + +> [!TIP] +> For descriptions of the inputs and outputs, see the complete [publish-image] action. + + +> [!IMPORTANT] +> For multi-arch images, the `image-manifest-tag` should have the `-$ARCH` suffix, as the tag +> without it should be reserved for the image index manifest which will refer to container images +> for each architecture we will push images for. + +### Inputs + +- `image-registry-uri` (eg: `oci.stackable.tech`) +- `image-registry-username` (required) +- `image-registry-password` (required) +- `image-repository` (eg: `stackable/kafka`) +- `image-manifest-tag` (eg: `3.4.1-stackable0.0.0-dev-amd64`) +- `source-image-uri` (eg: `localhost/kafka:3.4.1-stackable0.0.0-dev-amd64`) + +### Outputs + +None + +[publish-image]: ./action.yml diff --git a/publish-image/action.yml b/publish-image/action.yml index 19ea37d..2444fdf 100644 --- a/publish-image/action.yml +++ b/publish-image/action.yml @@ -58,6 +58,14 @@ runs: username: ${{ inputs.image-registry-username }} password: ${{ inputs.image-registry-password }} + - name: Extract Action Path + shell: bash + env: + GITHUB_ACTION_PATH: ${{ github.action_path }} + run: | + set -euo pipefail + echo "GITHUB_ACTION_PATH=$GITHUB_ACTION_PATH" | tee -a "$GITHUB_ENV" + - name: Re-tag container image shell: bash env: @@ -68,7 +76,7 @@ runs: docker tag "$SOURCE_IMAGE_URI" "$TARGET_IMAGE_URI" # Output for the next step - echo "IMAGE_MANIFEST_URI=$TARGET_IMAGE_URI" >> $GITHUB_ENV + echo "IMAGE_MANIFEST_URI=$TARGET_IMAGE_URI" | tee -a "$GITHUB_ENV" - name: Push the container image to ${{ inputs.image-registry-uri }} shell: bash @@ -77,7 +85,7 @@ runs: docker image push "$IMAGE_MANIFEST_URI" # Output for the next step - echo "IMAGE_REPO_DIGEST=$(.scripts/get_repo_digest.sh $IMAGE_MANIFEST_URI)" >> $GITHUB_ENV + echo "IMAGE_REPO_DIGEST=$($GITHUB_ACTION_PATH/.scripts/get_repo_digest.sh $IMAGE_MANIFEST_URI)" | tee -a "$GITHUB_ENV" - name: Sign the container image (${{ env.IMAGE_REPO_DIGEST }}) shell: bash diff --git a/publish-index-manifest/README.md b/publish-index-manifest/README.md new file mode 100644 index 0000000..478f10e --- /dev/null +++ b/publish-index-manifest/README.md @@ -0,0 +1,29 @@ +# `publish-index-manifest` + +> Manifest: [publish-index-manifest/action.yml][publish-index-manifest] + +This action creates an image index manifest, publishes it, and signs it. It does the following work: + +1. Create an image index manifest and link to each architecture in `image-architectures`. +2. Push the image index manifest. +3. Sign the image index manifest (which pushes the signature to the specified registry). + +## Inputs and Outputs + +> [!TIP] +> For descriptions of the inputs and outputs, see the complete [publish-index-manifest] action. + +### Inputs + +- `image-registry-uri`(eg: `oci.stackable.tech`) +- `image-registry-username` (required) +- `image-registry-password` (required) +- `image-repository` (eg: `stackable/kafka`) +- `image-index-manifest-tag` (eg: `3.4.1-stackable0.0.0-dev`) +- `image-architectures` (defaults to `["amd64", "arm64"]`) + +### Outputs + +None + +[publish-index-manifest]: .//action.yml diff --git a/publish-index-manifest/action.yml b/publish-index-manifest/action.yml index 9797c43..88f7b14 100644 --- a/publish-index-manifest/action.yml +++ b/publish-index-manifest/action.yml @@ -41,6 +41,14 @@ runs: username: ${{ inputs.image-registry-username }} password: ${{ inputs.image-registry-password }} + - name: Extract Action Path + shell: bash + env: + GITHUB_ACTION_PATH: ${{ github.action_path }} + run: | + set -euo pipefail + echo "GITHUB_ACTION_PATH=$GITHUB_ACTION_PATH" | tee -a "$GITHUB_ENV" + - name: Create Image Index Manifest shell: bash env: @@ -54,7 +62,7 @@ runs: # Construct the image index uri, which for example contains: # docker.stackable.tech/stackable/kafka:3.4.1-stackable0.0.0-dev IMAGE_INDEX_URI="$REGISTRY_URI/$IMAGE_REPOSITORY:$IMAGE_INDEX_MANIFEST_TAG" - echo "IMAGE_INDEX_URI=$IMAGE_INDEX_URI" >> $GITHUB_ENV + echo "IMAGE_INDEX_URI=$IMAGE_INDEX_URI" | tee -a "$GITHUB_ENV" AMEND_OPTIONS=$( jq \ @@ -89,7 +97,7 @@ runs: set -euo pipefail # Get the image index manifest digest - DIGEST=$(.scripts/get_manifest_digest.sh "$IMAGE_INDEX_URI") + DIGEST=$($GITHUB_ACTION_PATH/.scripts/get_manifest_digest.sh "$IMAGE_INDEX_URI") # Construct the image repo digest, which for example contains: # docker.stackable.tech/stackable/kafka@sha256:91... diff --git a/run-pre-commit/README.md b/run-pre-commit/README.md new file mode 100644 index 0000000..cc3d1b6 --- /dev/null +++ b/run-pre-commit/README.md @@ -0,0 +1,48 @@ +# `run-pre-commit` + +> Manifest: [run-pre-commit/action.yml][run-pre-commit] + +This action runs pre-commit by setting up Python and optionally the Rust toolchain and Hadolint in +the requested version. It requires a checkout with depth 0. It does the following work: + +1. Installs Python. The version can be configured via the `python-version` input. +2. Optionally sets up the Rust toolchain and Hadolint. +3. Runs pre-commit on changed files. + +Example usage (workflow): + +```yaml +--- +name: pre-commit + +on: + pull_request: + +jobs: + pre-commit: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout + with: + fetch-depth: 0 + submodules: recursive + - uses: stackabletech/actions/run-pre-commit +``` + +## Inputs and Outputs + +> [!TIP] +> For descriptions of the inputs and outputs, see the complete [run-pre-commit] action. + +### Inputs + +- `python-version` (defaults to `3.12`) +- `rust` (eg: `1.80.1`. Disabled if not specified) +- `rust-components` (defaults to `rustfmt,clippy`) +- `hadolint` (eg: `v2.12.0`. Disabled if not specified) + +### Outputs + +None + +[run-pre-commit]: ./action.yml diff --git a/run-pre-commit/action.yml b/run-pre-commit/action.yml index 070a370..70c9968 100644 --- a/run-pre-commit/action.yml +++ b/run-pre-commit/action.yml @@ -48,7 +48,7 @@ runs: curl -sL -o "${LOCATION_BIN}" "https://github.com/hadolint/hadolint/releases/download/${{ inputs.hadolint }}/hadolint-$SYSTEM-$ARCH" chmod 700 "${LOCATION_BIN}" - echo "$LOCATION_DIR" >> "$GITHUB_PATH" + echo "$LOCATION_DIR" | tee -a "$GITHUB_PATH" - uses: pre-commit/action@2c7b3805fd2a0fd8c1884dcaebf91fc102a13ecd # v3.0.1 with: diff --git a/shard/README.md b/shard/README.md new file mode 100644 index 0000000..e8b3c21 --- /dev/null +++ b/shard/README.md @@ -0,0 +1,48 @@ +# `shard` + +> Manifest: [shard/action.yml][shard] + +This action produces a list of versions for a product. This is to be used as a matrix dimension to +parallelize builds. It does the following work: + +1. Reads the `conf.py`, filtering versions for the product +2. Write the JSON array of version to `$GITHUB_OUTPUT` for use in a matrix. + +Example usage: + +```yaml +jobs: + generate_matrix: + name: Generate Version List + runs-on: ubuntu-latest + steps: + - uses: actions/checkout + - id: shard + uses: stackabletech/actions/shard + with: + product-name: ${{ env.PRODUCT_NAME }} + outputs: + versions: ${{ steps.shard.outputs.versions }} + + actual_matrix: + needs: [generate_matrix] + strategy: + matrix: + versions: ${{ fromJson(needs.generate_matrix.outputs.versions) }} + # ... +``` + +## Inputs and Outputs + +> [!TIP] +> For descriptions of the inputs and outputs, see the complete [shard] action. + +### Inputs + +- `product-name` (eg: `kafka`) + +### Outputs + +- `versions` (eg: `["3.7.1", "3.8.0"]`) + +[shard]: ./action.yml