Skip to content

Commit

Permalink
Generate Inventory & Sha7 appended filenames for binaries (#51)
Browse files Browse the repository at this point in the history
* Generate Inventory & Sha7 appended filenames for JRuby

Currently the Ruby buildapack is tightly coupled to the URLs that are generated by this repo. I am in the process of introducing an inventory file that the buildpack can use as a lookup. To start that process this commit introduces generating tgz files with the first 7 SHA256 characters appended. This allows us to update the same version number in the future without worrying that it will break future digest checks of the URL.

With this change the builder will continue to generate the original URLs, but it will also generate "<filename>-<sha256>.tgz" files as well and append this information to a `jruby_inventory.toml` file.

* Use FromStr for converting between arch structs

* Refactor and add inventory_check 

- We now create only one sha named archive for jruby builds since both ARM and AMD point to the same source.
- Binary `inventory_check` takes in a  file and validates that all checksums are valid.

* Use GemVersion

* Refactor atomic calls into their own function

* Refactor: Prefer composable methods

* I did not mean to commit that

* Replace older entries by default

* Refactor: API to mutate the inventory, not the file

* Test logic to preserve/remove artifacts from inventory

* Introduce Ruby manifest file

* Guard against the case of same sha7 but different sha256

* Fix clippy

* Remove accidental inventory addition

Sidenote, the check to validate that the URLs are valid works! https://github.com/heroku/docker-heroku-ruby-builder/actions/runs/10102146736/job/27937182426?pr=51

* Auto PR for ruby_inventory.toml

* Manual global write permissions

Before:

```
drwxr-xr-x 2 root   root       4096 Jul 25 22:34 .
drwxr-xr-x 3 runner docker     4096 Jul 25 22:32 ..
-rw-r--r-- 1 root   root   24061018 Jul 25 22:34 ruby-3.2.3.tgz
```

With these permissions we were unable to write to the directory. After:

After

```
drwxrwxrwx 2 root   root       4096 Jul 27 02:53 .
drwxr-xr-x 3 runner docker     4096 Jul 27 02:51 ..
-rw-r--r-- 1 root   root   24061325 Jul 27 02:53 ruby-3.2.3.tgz
```

Now we can copy the file to a new filename in the same directory.

* Prefer inventory::artifact::Arch

* Fix clippy and doc tests


```
error: the borrowed expression implements the required traits
   --> ruby_executable/src/bin/ruby_build.rs:143:24
    |
143 |           docker_run.arg(&format!(
    |  ________________________^
144 | |             "./make_ruby.sh {} {}",
145 | |             input_tar.display(),
146 | |             output_tar.display()
147 | |         ));
    | |_________^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrows_for_generic_args
    = note: `-D clippy::needless-borrows-for-generic-args` implied by `-D warnings`
    = help: to override `-D warnings` add `#[allow(clippy::needless_borrows_for_generic_args)]`
help: change this to
    |
143 ~         docker_run.arg(format!(
144 +             "./make_ruby.sh {} {}",
145 +             input_tar.display(),
146 +             output_tar.display()
147 ~         ));
    |

error: could not compile `ruby_executable` (bin "ruby_build" test) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
```

---------

Co-authored-by: Rune Soerensen <[email protected]>
  • Loading branch information
schneems and runesoerensen authored Sep 18, 2024
1 parent 2c782ef commit c5f2030
Show file tree
Hide file tree
Showing 20 changed files with 988 additions and 83 deletions.
10 changes: 10 additions & 0 deletions .github/workflows/build_jruby.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,13 @@ jobs:
- name: Upload Ruby runtime archive to S3 production
if: (!inputs.dry_run)
run: aws s3 sync ./output "s3://${S3_BUCKET}"

after-build-and-upload:
needs: build-and-upload
runs-on: pub-hk-ubuntu-24.04-xlarge
steps:
- name: Update Jruby inventory file locally
uses: peter-evans/create-pull-request@v6
with:
path: jruby_inventory.toml
title: "Add JRuby ${{inputs.jruby_version}} to inventory"
12 changes: 11 additions & 1 deletion .github/workflows/build_ruby.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ env:
S3_BUCKET: "heroku-buildpack-ruby"

jobs:
build_ruby:
build-and-upload:
runs-on: ${{ matrix.arch == 'arm64' && 'pub-hk-ubuntu-24.04-arm-xlarge' || 'pub-hk-ubuntu-24.04-xlarge' }}
strategy:
matrix:
Expand Down Expand Up @@ -56,3 +56,13 @@ jobs:
- name: Upload Ruby runtime archive to S3 production
if: (!inputs.dry_run)
run: aws s3 sync ./output "s3://${S3_BUCKET}"

after-build-and-upload:
needs: build-and-upload
runs-on: pub-hk-ubuntu-24.04-xlarge
steps:
- name: Update Ruby inventory file locally
uses: peter-evans/create-pull-request@v6
with:
path: ruby_inventory.toml
title: "Add Ruby ${{inputs.ruby_version}} to inventory"
19 changes: 19 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,3 +102,22 @@ jobs:
run: cargo run --bin jruby_build -- --version ${{matrix.version}} --base-image ${{matrix.base_image}}
- name: Check JRuby
run: cargo run --bin jruby_check -- --version ${{matrix.version}} --base-image ${{matrix.base_image}} --arch ${{matrix.arch}}

check_inventory_urls:
runs-on: ubuntu-24.04
if: (!contains(github.event.pull_request.labels.*.name, 'skip inventory check'))
strategy:
matrix:
inventory: ["jruby_inventory.toml", "ruby_inventory.toml"]
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Grab prior commits
run: |
set -eu
set pipefail
git fetch origin ${{ github.base_ref }} --depth 1 && \
git diff --unified=0 remotes/origin/${{ github.base_ref }} ${{matrix.inventory}} | grep '^+' | grep -v '^+++' | cut -c2- > check_inventory.toml
- name: Check manifest URLs
run: cargo run --bin inventory_check -- check_inventory.toml
Loading

0 comments on commit c5f2030

Please sign in to comment.