diff --git a/.github/ISSUE_TEMPLATE/tracking.md b/.github/ISSUE_TEMPLATE/tracking.md new file mode 100644 index 000000000..045c7c379 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/tracking.md @@ -0,0 +1,35 @@ +--- +name: Tracking Issue +about: Open an issue to track progress toward implementation of a merged RFC +title: '[RFC #] - ' +labels: type/tracking +assignees: '' + +--- + +[RFC #](https://github.com/buildpacks/rfcs/blob/main/text/) - + +Spec: +- [ ] https://github.com/buildpacks/spec/issues/ +- [ ] Released in API version `` + +Lifecycle: +- [ ] https://github.com/buildpacks/lifecycle/issues/ +- [ ] Released in lifecycle version `` + +Libcnb: +- [ ] https://github.com/buildpacks/libcnb/issues/ +- [ ] Released in libcnb version `` + +Pack: +- [ ] https://github.com/buildpacks/pack/issues/ +- [ ] Released in pack version `` + +Documentation: +- [ ] https://github.com/buildpacks/docs/issues/ + +: +- [ ] +- [ ] Released in version `` + +**Maintainers**: when closing this issue as completed, submit a PR to update the `Status` of the RFC to `Implemented`. diff --git a/.github/auto-assign-bat.yml b/.github/auto-assign-bat.yml index 7e6812205..f61b0b51a 100644 --- a/.github/auto-assign-bat.yml +++ b/.github/auto-assign-bat.yml @@ -8,3 +8,4 @@ filterLabels: assignees: - ekcasey - hone +- samj1912 diff --git a/.github/workflows/auto-assign-maintainer.yml b/.github/workflows/auto-assign-maintainer.yml index a247c7174..292bae4e7 100644 --- a/.github/workflows/auto-assign-maintainer.yml +++ b/.github/workflows/auto-assign-maintainer.yml @@ -14,7 +14,7 @@ jobs: runs-on: - ubuntu-latest steps: - - uses: kentaro-m/auto-assign-action@v1.2.1 + - uses: kentaro-m/auto-assign-action@v1.2.5 with: configuration-path: .github/auto-assign-core.yml distribution: @@ -23,7 +23,7 @@ jobs: runs-on: - ubuntu-latest steps: - - uses: kentaro-m/auto-assign-action@v1.2.1 + - uses: kentaro-m/auto-assign-action@v1.2.5 with: configuration-path: .github/auto-assign-distribution.yml implementation: @@ -32,7 +32,7 @@ jobs: runs-on: - ubuntu-latest steps: - - uses: kentaro-m/auto-assign-action@v1.2.1 + - uses: kentaro-m/auto-assign-action@v1.2.5 with: configuration-path: .github/auto-assign-implementation.yml learning: @@ -41,7 +41,7 @@ jobs: runs-on: - ubuntu-latest steps: - - uses: kentaro-m/auto-assign-action@v1.2.1 + - uses: kentaro-m/auto-assign-action@v1.2.5 with: configuration-path: .github/auto-assign-learning.yml platform: @@ -50,7 +50,7 @@ jobs: runs-on: - ubuntu-latest steps: - - uses: kentaro-m/auto-assign-action@v1.2.1 + - uses: kentaro-m/auto-assign-action@v1.2.5 with: configuration-path: .github/auto-assign-platform.yml bat: @@ -59,6 +59,6 @@ jobs: runs-on: - ubuntu-latest steps: - - uses: kentaro-m/auto-assign-action@v1.2.1 + - uses: kentaro-m/auto-assign-action@v1.2.5 with: configuration-path: .github/auto-assign-bat.yml diff --git a/.github/workflows/issues-generation.yml b/.github/workflows/issues-generation.yml index ba2a4828a..092f5af1a 100644 --- a/.github/workflows/issues-generation.yml +++ b/.github/workflows/issues-generation.yml @@ -15,7 +15,7 @@ jobs: if: ${{ github.event.issue.pull_request || github.event.pull_request }} runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Run Issue Generation uses: jromero/issue-generation-action@v1.0.0-beta.4 id: issues-generation diff --git a/0000-template.md b/0000-template.md index fa122548d..cc5d06dc3 100644 --- a/0000-template.md +++ b/0000-template.md @@ -44,6 +44,11 @@ This is the technical portion of the RFC, where you explain the design in suffic The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. +# Migration +[migration]: #migration + +This section should document breaks to public API and breaks in compatibility due to this RFC's proposed changes. In addition, it should document the proposed steps that one would need to take to work through these changes. Care should be give to include all applicable personas, such as platform developers, buildpack developers, buildpack users and consumers of buildpack images. + # Drawbacks [drawbacks]: #drawbacks @@ -73,3 +78,24 @@ Discuss prior art, both the good and bad. Does this RFC entail any proposed changes to the core specifications or extensions? If so, please document changes here. Examples of a spec. change might be new lifecycle flags, new `buildpack.toml` fields, new fields in the buildpackage label, etc. This section is not intended to be binding, but as discussion of an RFC unfolds, if spec changes are necessary, they should be documented here. + +# History +[history]: #history + + \ No newline at end of file diff --git a/README.md b/README.md index 5692aae70..919f58676 100644 --- a/README.md +++ b/README.md @@ -45,12 +45,13 @@ Once an RFC has been accepted, the sub-team maintainers should: Once an `issues-created/` label has been created for each sub-team, the RFC is ready to merge. The team member who merges the pull request should do the following: 1. Assign an id based off the pull request number. -1. Rename the file based off the ID inside `text/`. -1. Fill in the remaining metadata at the top. -1. Commit everything. -1. Update issues with RFC ID and a link to the text file. -1. Update any links in PR description to point at the committed file. -1. Remove the "Final Comment Period" label. +2. Rename the file based off the ID inside `text/`. +3. Fill in the remaining metadata at the top. +4. Commit everything. +5. Update issues with RFC ID and a link to the text file. +6. Update any links in PR description to point at the committed file. +7. Remove the "status/voting" label. +8. Create a [tracking issue](https://github.com/buildpacks/rfcs/issues/new?assignees=&labels=type%2Ftracking&template=tracking.md&title=%5BRFC+%23%3CINSERT+RFC+NUMBER+-+e.g.%2C+0099%3E%5D+-+%3CINSERT+RFC+TITLE%3E). ## Automation @@ -60,4 +61,6 @@ The `merge-rfc.sh` script automates several steps of the merge process for accep ``` Each `` should be of the form `/#` (e.g. `buildpacks/spec#1`). In the rare case that no work must be done in the project as a result of the RFC pass the `-n` flag to explicitly indicate that no issues should be linked. -After running the `merge-rfc.sh` script, manually verify the output before pushing changes. +After running the `merge-rfc.sh` script: +* Manually verify the output before pushing changes. +* Create a [tracking issue](https://github.com/buildpacks/rfcs/issues/new?assignees=&labels=type%2Ftracking&template=tracking.md&title=%5BRFC+%23%3CINSERT+RFC+NUMBER+-+e.g.%2C+0099%3E%5D+-+%3CINSERT+RFC+TITLE%3E). diff --git a/merge-rfc.sh b/merge-rfc.sh index 96b8d1930..8f90848f5 100755 --- a/merge-rfc.sh +++ b/merge-rfc.sh @@ -55,7 +55,7 @@ require_command issues-generation if [[ -z "${GITHUB_TOKEN:-}" ]]; then require_command op echo "> Pulling GitHub token from vault..." - GITHUB_TOKEN=$(op get item 7xorpxvz3je3vozqg3fy3wrcg4 --vault "Shared" --account buildpacks | jq -r '.details.sections[] | select(.fields).fields[] | select(.t == "credential").v') + GITHUB_TOKEN=$(op read op://Shared/7xorpxvz3je3vozqg3fy3wrcg4/credential --account buildpacks) fi #### @@ -146,7 +146,7 @@ if [[ "$OSTYPE" == "darwin"* ]]; then fi sed $SEDOPTION "s|- RFC Pull Request:.*|- RFC Pull Request: [${REPO}#${PR_NUMBER}](https://github.com/${OWNER}/${REPO}/pull/${PR_NUMBER})|" "${SOURCE_DOC}" sed $SEDOPTION "s|- CNB Issue:.*|- CNB Issue: $ISSUES_TEXT|" "${SOURCE_DOC}" -sed $SEDOPTION "s|- State:.*|- State: **Approved**|" "${SOURCE_DOC}" +sed $SEDOPTION "s|- Status:.*|- Status: Approved|" "${SOURCE_DOC}" echo "> Moving ${SOURCE_DOC} to ${TARGET_DOC}..." git mv "${SOURCE_DOC}" "${TARGET_DOC}" diff --git a/text/0001-pack-suggest-stacks.md b/text/0001-pack-suggest-stacks.md index b54f041b9..89f508fca 100644 --- a/text/0001-pack-suggest-stacks.md +++ b/text/0001-pack-suggest-stacks.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Suggest Stacks - Start Date: 2019-04-25 +- Status: Implemented - CNB Pull Request: [rfcs#4](https://github.com/buildpacks/rfcs/pull/4), [pack#190](https://github.com/buildpacks/pack/pull/190) - CNB Issue: [pack#156](https://github.com/buildpacks/pack/issues/156) - Supersedes: N/A diff --git a/text/0002-pack-logging-refactor.md b/text/0002-pack-logging-refactor.md index c7763fa81..73c2eac7f 100644 --- a/text/0002-pack-logging-refactor.md +++ b/text/0002-pack-logging-refactor.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Pack Logging Interface - Start Date: 2019-05-06 +- Status: Implemented - CNB Pull Request: [rfcs#6](https://github.com/buildpacks/rfcs/pull/6), [pack#182](https://github.com/buildpacks/pack/pull/182) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0003-pack-inspect-image.md b/text/0003-pack-inspect-image.md index f46d0ecc2..0d2bc65a5 100644 --- a/text/0003-pack-inspect-image.md +++ b/text/0003-pack-inspect-image.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Inspect Image - Start Date: 2019-04-25 +- Status: Implemented - CNB Pull Request: [rfcs#5](https://github.com/buildpacks/rfcs/pull/5) - CNB Issue: - Supersedes: N/A diff --git a/text/0004-rfc-process.md b/text/0004-rfc-process.md index 0dfc88dab..091387c49 100644 --- a/text/0004-rfc-process.md +++ b/text/0004-rfc-process.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: RFC Process - Start Date: 2019-04-09 +- Status: Superseded - CNB Pull Request: [rfcs#1](https://github.com/buildpacks/rfcs/pull/1), [rfcs#7](https://github.com/buildpacks/rfcs/pull/7) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0005-contractual-build-plan.md b/text/0005-contractual-build-plan.md index 6ac3e456a..c6d1b7a44 100644 --- a/text/0005-contractual-build-plan.md +++ b/text/0005-contractual-build-plan.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Contractual Build Plan - Start Date: 2019-04-12 +- Status: Implemented - CNB Pull Requests: [rfcs#12](https://github.com/buildpacks/rfcs/pull/12), [spec#52](https://github.com/buildpacks/spec/pull/52), [lifecycle#149](https://github.com/buildpacks/lifecycle/pull/149) - CNB Issues: (lifecycle issues to follow) diff --git a/text/0006-stage-specific-mixins.md b/text/0006-stage-specific-mixins.md index 543ce0ec1..ffa762bda 100644 --- a/text/0006-stage-specific-mixins.md +++ b/text/0006-stage-specific-mixins.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Stage-specific Mixins - Start Date: 2019-06-13 +- Status: Superseded - CNB Pull Requests: [rfcs#13](https://github.com/buildpacks/rfcs/pull/13), [spec#54](https://github.com/buildpacks/spec/pull/54) - CNB Issues: (lifecycle issues to follow) diff --git a/text/0007-spec-distribution.md b/text/0007-spec-distribution.md index 41b066862..841c747c2 100644 --- a/text/0007-spec-distribution.md +++ b/text/0007-spec-distribution.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Buildpack Distribution Specification - Start Date: 2019-04-12 +- Status: Implemented - CNB Pull Requests: [rfcs#11](https://github.com/buildpacks/rfcs/pull/11), [spec#53](https://github.com/buildpacks/spec/pull/53), [lifecycle#149](https://github.com/buildpacks/lifecycle/pull/149), [pack#243](https://github.com/buildpacks/pack/issues/243) - CNB Issues: (lifecycle issues to follow) diff --git a/text/0008-buildpack-config-for-dist.md b/text/0008-buildpack-config-for-dist.md index 2f4349a24..85c01b08c 100644 --- a/text/0008-buildpack-config-for-dist.md +++ b/text/0008-buildpack-config-for-dist.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Buildpack Configuration for Distribution - Start Date: July 22, 2019 +- Status: Implemented - CNB Pull Request: [rfcs#15](https://github.com/buildpacks/rfcs/pull/15) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0009-auto-load-user-provided-environment-variables.md b/text/0009-auto-load-user-provided-environment-variables.md index cd97a24d3..2425464f8 100644 --- a/text/0009-auto-load-user-provided-environment-variables.md +++ b/text/0009-auto-load-user-provided-environment-variables.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Auto-load User-provided Environment Variables - Start Date: 2019-06-17 +- Status: Implemented - CNB Pull Requests: [rfcs#14](https://github.com/buildpacks/rfcs/pull/14), [spec#55](https://github.com/buildpacks/spec/pull/55), [lifecycle#163](https://github.com/buildpacks/lifecycle/pull/163) - CNB Issues: (lifecycle issues to follow) diff --git a/text/0010-api-versions.md b/text/0010-api-versions.md index f5d571692..b7506a2f6 100644 --- a/text/0010-api-versions.md +++ b/text/0010-api-versions.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: API Versions - Start Date: 2019-08-05 +- Status: Implemented - CNB Pull Request: [rfcs#19](https://github.com/buildpacks/rfcs/pull/19), [pack#282](https://github.com/buildpacks/pack/pull/282) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0011-lifecycle-descriptor.md b/text/0011-lifecycle-descriptor.md index d2b02744a..5eb4fa3ae 100644 --- a/text/0011-lifecycle-descriptor.md +++ b/text/0011-lifecycle-descriptor.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Lifecycle Descriptor - Start Date: 08/05/2019 +- Status: Superseded - CNB Pull Request: [rfcs#20](https://github.com/buildpacks/rfcs/pull/20), [lifecycle#167](https://github.com/buildpacks/lifecycle/pull/167), [pack#269](https://github.com/buildpacks/pack/pull/269), [pack#294](https://github.com/buildpacks/pack/pull/294) - CNB Issue: [pack#293](https://github.com/buildpacks/pack/issues/293) - Supersedes: N/A diff --git a/text/0012-service-binding.md b/text/0012-service-binding.md index 8bbb3e385..2716f5f17 100644 --- a/text/0012-service-binding.md +++ b/text/0012-service-binding.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Service Binding - Start Date: 2019-08-06 +- Status: Superseded - CNB Pull Request: [rfcs#22](https://github.com/buildpacks/rfcs/pull/22), [spec#57](https://github.com/buildpacks/spec/pull/57) - CNB Issue: - Supersedes: N/A diff --git a/text/0013-app-layer-metadata-source.md b/text/0013-app-layer-metadata-source.md index d683a3bd3..a999d4b0e 100644 --- a/text/0013-app-layer-metadata-source.md +++ b/text/0013-app-layer-metadata-source.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Manifest - App metadata - add source type, version and metadata - Start Date: 2019-06-10 +- Status: Implemented - CNB Pull Request: [rfcs#9](https://github.com/buildpacks/rfcs/pull/9) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0014-combined-restorer-analyzer-phases.md b/text/0014-combined-restorer-analyzer-phases.md index a133e98f8..06f9c2d2e 100644 --- a/text/0014-combined-restorer-analyzer-phases.md +++ b/text/0014-combined-restorer-analyzer-phases.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Lifecycle cache contract changes. - Start Date: 2019-08-02 +- Status: Implemented - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0015-windows-lifecycle.md b/text/0015-windows-lifecycle.md index 65b744288..0ddf93c42 100644 --- a/text/0015-windows-lifecycle.md +++ b/text/0015-windows-lifecycle.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Support for Windows in the Lifecycle components - Start Date: 2019-10-07 +- Status: Implemented - CNB Pull Request: [rfcs#27](https://github.com/buildpacks/rfcs/pull/27) - CNB Issue: - Supersedes: N/A diff --git a/text/0016-use-email-more.md b/text/0016-use-email-more.md index 224ef3c1c..a2b445d00 100644 --- a/text/0016-use-email-more.md +++ b/text/0016-use-email-more.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Use Email More - Start Date: 2019-12-05 +- Status: Implemented - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) diff --git a/text/0017-pack-build-default-process-flag.md b/text/0017-pack-build-default-process-flag.md index 2363d69bd..31036fe8c 100644 --- a/text/0017-pack-build-default-process-flag.md +++ b/text/0017-pack-build-default-process-flag.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Pack Build Default Process Flag - Start Date: 2019-10-29 +- Status: Implemented - CNB Pull Request: [rfcs#28](https://github.com/buildpack/rfcs/pull/28) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0018-remove-pack-run.md b/text/0018-remove-pack-run.md index ca4992736..c8f506bc7 100644 --- a/text/0018-remove-pack-run.md +++ b/text/0018-remove-pack-run.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Remove `pack run` - Start Date: 2019-12-11 +- Status: Implemented - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0019-project-descriptor.md b/text/0019-project-descriptor.md index 280f54488..f3bc5cdf7 100644 --- a/text/0019-project-descriptor.md +++ b/text/0019-project-descriptor.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Minimal Project Descriptor - Start Date: 2019-06-11 +- Status: Implemented - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) - Supersedes: https://github.com/buildpack/rfcs/pull/25 diff --git a/text/0020-landing-page.md b/text/0020-landing-page.md index 4bdff1606..e7a310d9f 100644 --- a/text/0020-landing-page.md +++ b/text/0020-landing-page.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Create a community landing page - Start Date: 1-6-2020 +- Status: Implemented - CNB Pull Request: [rfcs#42](https://github.com/buildpack/rfcs/pull/42) - CNB Issue: 42 - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) diff --git a/text/0021-lifecycle-compat-verification.md b/text/0021-lifecycle-compat-verification.md index 417d502ab..2f8b4c3a8 100644 --- a/text/0021-lifecycle-compat-verification.md +++ b/text/0021-lifecycle-compat-verification.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Lifecycle Compatibility Verification - Start Date: 2019-12-06 +- Status: Implemented - CNB Pull Request: [rfcs#34](https://github.com/buildpacks/rfcs/pull/34) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0022-client-side-buildpack-registry.md b/text/0022-client-side-buildpack-registry.md index 76f69bc60..d8d52cdaf 100644 --- a/text/0022-client-side-buildpack-registry.md +++ b/text/0022-client-side-buildpack-registry.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Client-Side Buildpack Registry - Start Date: 2019-11-21 +- Status: Superseded - CNB Pull Request: [rfcs#35](https://github.com/buildpacks/rfcs/pull/35) - CNB Issue: (leave blank) - Supersedes: https://github.com/buildpack/rfcs/pull/24 diff --git a/text/0023-circleci-orb.md b/text/0023-circleci-orb.md index 4600fe5d1..7f732fb73 100644 --- a/text/0023-circleci-orb.md +++ b/text/0023-circleci-orb.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Support a CircleCI Orb - Start Date: 2019-12-15 +- Status: Implemented - CNB Pull Request: [rfcs#39](https://github.com/buildpacks/rfcs/pull/39) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0024-lifecycle-multicall-binary-for-build.md b/text/0024-lifecycle-multicall-binary-for-build.md index 0b22a1fd1..40e1707ad 100644 --- a/text/0024-lifecycle-multicall-binary-for-build.md +++ b/text/0024-lifecycle-multicall-binary-for-build.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Lifecycle as a multicall binary for build phases - Start Date: 01/21/2020 +- Status: Implemented - CNB Pull Request: [rfcs#45](https://github.com/buildpacks/rfcs/pull/45) - CNB Issue: (leave blank) - Supersedes: "N/A" diff --git a/text/0025-dont-trust-builders.md b/text/0025-dont-trust-builders.md index 5089fbc81..aefa1a5fb 100644 --- a/text/0025-dont-trust-builders.md +++ b/text/0025-dont-trust-builders.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Isolate Registry Credentials from Builder Images - Start Date: 2020-01-14 +- Status: Implemented - CNB Pull Request: [rfcs#43](https://github.com/buildpacks/rfcs/pull/43) - CNB Issue: - Supersedes: N/A diff --git a/text/0026-lifecycle-all.md b/text/0026-lifecycle-all.md index 279924051..adc264cbb 100644 --- a/text/0026-lifecycle-all.md +++ b/text/0026-lifecycle-all.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Lifecycle All Binary - Start Date: 2020-01-21 +- Status: Implemented - CNB Pull Request: [rfcs#46](https://github.com/buildpacks/rfcs/pull/46) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0027-spec-api-branches.md b/text/0027-spec-api-branches.md index 351e3780c..5b6fb5ed0 100644 --- a/text/0027-spec-api-branches.md +++ b/text/0027-spec-api-branches.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Spec API branches - Start Date: 2020-01-27 +- Status: Implemented - CNB Pull Request: [rfcs#49](https://github.com/buildpacks/rfcs/pull/49) - CNB Issue: - Supersedes: N/A diff --git a/text/0028-remove-cred-helpers.md b/text/0028-remove-cred-helpers.md index 32eb14686..56e57a6d0 100644 --- a/text/0028-remove-cred-helpers.md +++ b/text/0028-remove-cred-helpers.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Remove Lifecycle Credential Helpers Integration - Start Date: 2020-01-22 +- Status: Implemented - CNB Pull Request: [rfcs#49](https://github.com/buildpacks/rfcs/pull/49) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0029-template-changes.md b/text/0029-template-changes.md index faef401f7..5c058bdf5 100644 --- a/text/0029-template-changes.md +++ b/text/0029-template-changes.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: RFC template changes - Start Date: 2020-02-03 +- Status: Implemented - CNB Pull Request: [rfcs#52](https://github.com/buildpacks/rfcs/pull/52) - CNB Issue: (leave blank) - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) diff --git a/text/0030-links-for-buildpacks.md b/text/0030-links-for-buildpacks.md index 42dc369fe..161f7d7bf 100644 --- a/text/0030-links-for-buildpacks.md +++ b/text/0030-links-for-buildpacks.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Buildpack homepage - Start Date: 1-23-2020 +- Status: Implemented - CNB Pull Request: [rfcs#55](https://github.com/buildpacks/rfcs/pull/55) - CNB Issue: (leave blank) - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) diff --git a/text/0031-bionic-mixins.md b/text/0031-bionic-mixins.md index 159a88e95..899e03a59 100644 --- a/text/0031-bionic-mixins.md +++ b/text/0031-bionic-mixins.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Defining Mixins for io.buildpacks.stacks.bionic - Start Date: 2019-12-05 +- Status: Superseded - CNB Pull Request: https://github.com/buildpacks/rfcs/pull/40 - CNB Issue: (leave blank) - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) diff --git a/text/0032-update-json-cnb-registry.md b/text/0032-update-json-cnb-registry.md index 9acba4140..f14a4deda 100644 --- a/text/0032-update-json-cnb-registry.md +++ b/text/0032-update-json-cnb-registry.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Update JSON Structure of CNB Registry - Start Date: 2020-03-19 +- Status: Implemented - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) - Supersedes: [RFC-0022](https://github.com/buildpacks/rfcs/blob/main/text/0022-client-side-buildpack-registry.md) diff --git a/text/0033-add-author.md b/text/0033-add-author.md index 6fe705670..df42b0517 100644 --- a/text/0033-add-author.md +++ b/text/0033-add-author.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Add Author to RFC Metadata - Start Date: 2020-03-14 +- Status: Implemented - CNB Pull Request: https://github.com/buildpacks/rfcs/pull/64 - CNB Issue: (leave blank) - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) diff --git a/text/0034-image-labels.md b/text/0034-image-labels.md index 968964637..286e1fcf2 100644 --- a/text/0034-image-labels.md +++ b/text/0034-image-labels.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Image Labels - Start Date: 2020-03-24 +- Status: Implemented - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0035-buildpack-versions.md b/text/0035-buildpack-versions.md index 8a0085deb..dd8230c11 100644 --- a/text/0035-buildpack-versions.md +++ b/text/0035-buildpack-versions.md @@ -2,6 +2,7 @@ [meta]: #meta - **Name:** Semantic Version Formats for Buildpack Versions - **Start Date:** 2020-03-11 +- **Status:** Implemented - **Supersedes:** N/A # Summary diff --git a/text/0036-cnb-buildpack-directory-env-var.md b/text/0036-cnb-buildpack-directory-env-var.md index 51ba4625f..67808c055 100644 --- a/text/0036-cnb-buildpack-directory-env-var.md +++ b/text/0036-cnb-buildpack-directory-env-var.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: CNB\_BUILDPACK\_DIR environment variable - Start Date: 2020-04-02 +- Status: Implemented - CNB Pull Request: [rfcs#49](https://github.com/buildpacks/rfcs/pull/71) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0037-buildpack-uris.md b/text/0037-buildpack-uris.md index 92776495a..12e259263 100644 --- a/text/0037-buildpack-uris.md +++ b/text/0037-buildpack-uris.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Buildpack URIs - Start Date: 02/07/2020 +- Status: Implemented - CNB Pull Request: [rfcs#56](https://github.com/buildpacks/rfcs/pull/56) - CNB Issue: (leave blank) - Supersedes: "N/A" diff --git a/text/0038-gh-repo-labels.md b/text/0038-gh-repo-labels.md index f4bfdd840..00de412e1 100644 --- a/text/0038-gh-repo-labels.md +++ b/text/0038-gh-repo-labels.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: GitHub repo labels - Start Date: 02/03/2020 +- Status: Implemented - CNB Pull Request: [rfcs#53](https://github.com/buildpacks/rfcs/pull/53) - CNB Issue: (leave blank) - Supersedes: N/A diff --git a/text/0039-release-process.md b/text/0039-release-process.md index 3333a02fd..f9a83930f 100644 --- a/text/0039-release-process.md +++ b/text/0039-release-process.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Release Process - Start Date: 2019-11-18 +- Status: Implemented - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) diff --git a/text/0040-export-report.md b/text/0040-export-report.md index 9b4a06b7c..d87c87a76 100644 --- a/text/0040-export-report.md +++ b/text/0040-export-report.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Export Report - Start Date: (fill in today's date: 2020-04-01) +- Status: Approved - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) - Supersedes: "N/A" diff --git a/text/0041-api-version-compat.md b/text/0041-api-version-compat.md index 5d5ff70c3..ae814167f 100644 --- a/text/0041-api-version-compat.md +++ b/text/0041-api-version-compat.md @@ -3,6 +3,7 @@ - Name: Lifecycle API version support changes - Start Date: 2020-05-14 - Author(s): [Emily Casey](https://github.com/ekcasey) +- Status: Implemented - RFC Pull Request: [rfcs#79](https://github.com/buildpacks/rfcs/pull/79) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0042-process-specific-env.md b/text/0042-process-specific-env.md index a455fc2fc..a98e6d2bc 100644 --- a/text/0042-process-specific-env.md +++ b/text/0042-process-specific-env.md @@ -2,6 +2,7 @@ [meta]: #meta - Name: Process Specific Environment - Start Date: 2020-04-03 +- Status: Implemented - CNB Pull Request: (https://github.com/buildpacks/rfcs/pull/72) - CNB Issue: - Supersedes: N/A diff --git a/text/0043-increase-build-plan-flexibility.md b/text/0043-increase-build-plan-flexibility.md index d31670963..f4aa8cd46 100644 --- a/text/0043-increase-build-plan-flexibility.md +++ b/text/0043-increase-build-plan-flexibility.md @@ -3,6 +3,7 @@ - Name: Increase Build Plan Flexibility - Start Date: 2020-05-28 - Author(s): nebhale +- Status: Implemented - RFC Pull Request: [rfcs#82](https://github.com/buildpacks/rfcs/pull/82) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0044-pack-publish-buildpack.md b/text/0044-pack-publish-buildpack.md index 183b1f607..edeb03b0d 100644 --- a/text/0044-pack-publish-buildpack.md +++ b/text/0044-pack-publish-buildpack.md @@ -3,6 +3,7 @@ - Name: Pack Register Buildpack - Start Date: 2020-04-27 - Author(s): [Joe Kutner](https://github.com/jkutner), [Javier Romero](https://github.com/jromero) +- Status: Implemented - RFC Pull Request: [rfcs#75](https://github.com/buildpacks/rfcs/pull/75) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0045-launcher-arguments.md b/text/0045-launcher-arguments.md index 77f6acfba..5a5bbf85d 100644 --- a/text/0045-launcher-arguments.md +++ b/text/0045-launcher-arguments.md @@ -3,6 +3,7 @@ - Name: Launcher Arguments - Start Date: 2020-06-02 - Author(s): nebhale ekcasey +- Status: Superseded - RFC Pull Request: [rfcs#84](https://github.com/buildpacks/rfcs/pull/84) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0046-pack-pull-policy.md b/text/0046-pack-pull-policy.md index 5276e1f06..4b4b76226 100644 --- a/text/0046-pack-pull-policy.md +++ b/text/0046-pack-pull-policy.md @@ -3,6 +3,7 @@ - Name: `pack` Image Pull Policy - Start Date: 2020-05-21 - Author(s): [Javier Romero](https://github.com/jromero) +- Status: Implemented - RFC Pull Request: [rfcs#80](https://github.com/buildpacks/rfcs/pull/80) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0047-danger-zone.md b/text/0047-danger-zone.md index 60515b137..a96e8bc9c 100644 --- a/text/0047-danger-zone.md +++ b/text/0047-danger-zone.md @@ -3,6 +3,7 @@ - Name: Read-Write Volume Mount in Pack - Start Date: 2020-06-02 - Author(s): nebhale +- Status: Implemented - RFC Pull Request: [rfcs#85](https://github.com/buildpacks/rfcs/pull/85) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0048-inline-buildpack.md b/text/0048-inline-buildpack.md index 9785a4d21..8a1da4791 100644 --- a/text/0048-inline-buildpack.md +++ b/text/0048-inline-buildpack.md @@ -3,6 +3,7 @@ - Name: Inline Buildpacks - Start Date: 2020-06-11 - Author(s): [Joe Kutner](https://github.com/jkutner) +- Status: Implemented - RFC Pull Request: [rfcs#86](https://github.com/buildpacks/rfcs/pull/86) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0049-multi-api-lifecycle-descriptor.md b/text/0049-multi-api-lifecycle-descriptor.md index 9ba9427f9..d98902a4c 100644 --- a/text/0049-multi-api-lifecycle-descriptor.md +++ b/text/0049-multi-api-lifecycle-descriptor.md @@ -3,6 +3,7 @@ - Name: Lifecycle Descriptor with Multiple APIs - Start Date: 2020-07-02 - Author(s): ekcasey +- Status: Superseded - RFC Pull Request: [rfcs#92](https://github.com/buildpacks/rfcs/pull/92) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) @@ -75,22 +76,22 @@ A builder or lifecycle image with the above descriptor file should have the foll ### Deprecated APIs Only API versions defined in a spec release can be in the deprecated range. -New `CNB_PLATFORM_DEPRECATION_MODE`, and `CNB_BUILDPACK_DEPRECATION_MODE` environment variables will control deprecation behavior with: +A new `CNB_DEPRECATION_MODE`environment variable will control deprecation behavior with: * allowed values: `warn`, `error`, `silent` * default value: `warn` **When** the `CNB_PLATFROM_API` environment variable is set to an API version in the deprecated platform API, the lifecycle shall: - - **If** `CNB_PLATFORM_DEPRECATION_MODE` is unset, **Then** print a warning and continue - - **If** `CNB_PLATFORM_DEPRECATION_MODE=warn`, **Then** print a warning and continue - - **If** `CNB_PLATFORM_DEPRECATION_MODE=error`, **Then** fail - - **If** `CNB_PLATFORM_DEPRECATION_MODE=silent`, **Then** continue w/o warning + - **If** `CNB_DEPRECATION_MODE` is unset, **Then** print a warning and continue + - **If** `CNB_DEPRECATION_MODE=warn`, **Then** print a warning and continue + - **If** `CNB_DEPRECATION_MODE=error`, **Then** fail + - **If** `CNB_DEPRECATION_MODE=silent`, **Then** continue w/o warning **When** the `api` field in a `buildpack.toml` file is set to an API version in the deprecated buildpack API range the lifecycle shall: - - **If** `CNB_BUILDPACK_DEPRECATION_MODE` is unset, **Then** print a warning and continue - - **If** `CNB_BUILDPACK_DEPRECATION_MODE=warn`, **Then** print a warning and continue - - **If** `CNB_BUILDPACK_DEPRECATION_MODE=error`, **Then** fail - - **If** `CNB_BUILDPACK_DEPRECATION_MODE=silent`, **Then** continue w/o warning + - **If** `CNB_DEPRECATION_MODE` is unset, **Then** print a warning and continue + - **If** `CNB_DEPRECATION_MODE=warn`, **Then** print a warning and continue + - **If** `CNB_DEPRECATION_MODE=error`, **Then** fail + - **If** `CNB_DEPRECATION_MODE=silent`, **Then** continue w/o warning ### Supported APIs Only API versions defined in a spec release can be in the supported range. @@ -156,3 +157,21 @@ Understanding buildpack/platform API support will require more documentation and # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes This RFC will not result in spec changes. The lifecycle descriptor file and builder labels are unspecified features. + +## Amended +### Meta +[meta-1]: #meta-1 +- Name: Variable Rename +- Start Date: 2022-12-07 +- Author(s): natalieparellano +- Amendment Pull Request: (leave blank) + +### Summary + +Instead of `CNB_PLATFORM_DEPRECATION_MODE` and `CNB_BUILDPACK_DEPRECATION_MODE` we have just one variable, `CNB_DEPRECATION_MODE`. + +### Motivation + +Why was this amendment necessary? + +Somewhere along the way, this is what we decided to implement. Updating the RFC to be accurate allows us to point end-users toward this RFC in helping to explain how APIs are deprecated. \ No newline at end of file diff --git a/text/0050-stack-metadata.md b/text/0050-stack-metadata.md index b3bcaa41c..52f40cf16 100644 --- a/text/0050-stack-metadata.md +++ b/text/0050-stack-metadata.md @@ -3,6 +3,7 @@ - Name: Additional Stack Metadata - Start Date: 2020-05-12 - Author(s): kvedurmu +- Status: Implemented - RFC Pull Request: [rfcs#78](https://github.com/buildpacks/rfcs/pull/78) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0051-override-env-by-default.md b/text/0051-override-env-by-default.md index 6fb5b7ffa..bf525f5d0 100644 --- a/text/0051-override-env-by-default.md +++ b/text/0051-override-env-by-default.md @@ -3,6 +3,7 @@ - Name: Override Env Vars by Default - Start Date: 2020-07-26 - Author(s): @sclevine +- Status: Implemented - RFC Pull Request: [rfcs#98](https://github.com/buildpacks/rfcs/pull/98) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0052-opt-in-layer-caching.md b/text/0052-opt-in-layer-caching.md index 32daa2eb6..957e54365 100644 --- a/text/0052-opt-in-layer-caching.md +++ b/text/0052-opt-in-layer-caching.md @@ -3,6 +3,7 @@ - Name: Opt-in Layer Caching - Start Date: 2020-07-26 - Author(s): @sclevine +- Status: Implemented - RFC Pull Request: [rfcs#99](https://github.com/buildpacks/rfcs/pull/99) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0053-decouple-buildpack-plan-and-bom.md b/text/0053-decouple-buildpack-plan-and-bom.md index dc6cc2d4c..44231d8bf 100644 --- a/text/0053-decouple-buildpack-plan-and-bom.md +++ b/text/0053-decouple-buildpack-plan-and-bom.md @@ -3,6 +3,7 @@ - Name: Decouple Buildpack Plan and Bill-of-Materials - Start Date: 2020-07-26 - Author(s): @sclevine +- Status: Superseded - RFC Pull Request: [rfcs#100](https://github.com/buildpacks/rfcs/pull/100) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0054-project-descriptor-schema.md b/text/0054-project-descriptor-schema.md index ac066f5c8..7829c787c 100644 --- a/text/0054-project-descriptor-schema.md +++ b/text/0054-project-descriptor-schema.md @@ -3,6 +3,7 @@ - Name: Project Descriptor (project.toml) Schema - Start Date: 2020-28-07 - Author(s): @joshwlewis +- Status: Implemented - RFC Pull Request: - CNB Pull Request: - CNB Issue: diff --git a/text/0055-deprecate-service-bindings.md b/text/0055-deprecate-service-bindings.md index c36a91177..ce2d5d00e 100644 --- a/text/0055-deprecate-service-bindings.md +++ b/text/0055-deprecate-service-bindings.md @@ -3,6 +3,7 @@ - Name: Deprecate Bindings Extension Specification - Start Date: 2020-08-06 - Author(s): @nebhale +- Status: Implemented - RFC Pull Request: [rfcs#105](https://github.com/buildpacks/rfcs/pull/105) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0056-any-stack-buildpacks.md b/text/0056-any-stack-buildpacks.md index 15d7fd9e2..4d389140b 100644 --- a/text/0056-any-stack-buildpacks.md +++ b/text/0056-any-stack-buildpacks.md @@ -3,6 +3,7 @@ - Name: Stackless Buildpacks - Start Date: 2020-07-26 - Author(s): @sclevine +- Status: Implemented - RFC Pull Request: [rfcs#97](https://github.com/buildpacks/rfcs/pull/97) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0057-exec.d-shell-free-profile-d.md b/text/0057-exec.d-shell-free-profile-d.md index 58e77f1de..aa7c062ad 100644 --- a/text/0057-exec.d-shell-free-profile-d.md +++ b/text/0057-exec.d-shell-free-profile-d.md @@ -3,6 +3,7 @@ - Name: Exec.d - Shell-Free Profile.d - Start Date: 2020-07-26 - Author(s): @sclevine +- Status: Implemented - RFC Pull Request: [rfcs#104](https://github.com/buildpacks/rfcs/pull/104) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0058-pack-subcommands.md b/text/0058-pack-subcommands.md index ce9ef9f16..3b81017be 100644 --- a/text/0058-pack-subcommands.md +++ b/text/0058-pack-subcommands.md @@ -3,6 +3,7 @@ - Name: `pack` subcommands - Start Date: 2020-07-08 - Author(s): @jromero +- Status: Implemented - RFC Pull Request: [rfcs#93](https://github.com/buildpacks/rfcs/pull/93) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0059-label-rfcs.md b/text/0059-label-rfcs.md index 0cdea9ddf..39d6f3903 100644 --- a/text/0059-label-rfcs.md +++ b/text/0059-label-rfcs.md @@ -3,6 +3,7 @@ - Name: Label RFCs With Specification And Target Audience - Start Date: 2020-08-15 - Author(s): ForestEckhardt +- Status: Implemented - RFC Pull Request: [rfcs#108](https://github.com/buildpacks/rfcs/pull/108) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0060-create-repo-issues.md b/text/0060-create-repo-issues.md index ef0e03d9e..92087a85d 100644 --- a/text/0060-create-repo-issues.md +++ b/text/0060-create-repo-issues.md @@ -3,6 +3,7 @@ - Name: Create Repo Issues - Start Date: 2020-08-06 - Author(s): [@dfreilich][https://github.com/dfreilich] +- Status: Superseded - RFC Pull Request: [rfcs#106](https://github.com/buildpacks/rfcs/pull/106) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0061-relax-mixin-contract.md b/text/0061-relax-mixin-contract.md index 3481abbb7..8cdd1635f 100644 --- a/text/0061-relax-mixin-contract.md +++ b/text/0061-relax-mixin-contract.md @@ -3,6 +3,7 @@ - Name: Relax Mixin Contract - Start Date: 2020-08-12 - Author(s): @sclevine +- Status: Superseded - RFC Pull Request: [rfcs#109](https://github.com/buildpacks/rfcs/pull/109) - CNB Pull Request: (leave blank) - CNB Issue: https://github.com/buildpacks/spec/issues/149, https://github.com/buildpacks/pack/issues/868, https://github.com/buildpacks/lifecycle/issues/425 diff --git a/text/0062-distribution-team.md b/text/0062-distribution-team.md index 68c219985..0404a2bfc 100644 --- a/text/0062-distribution-team.md +++ b/text/0062-distribution-team.md @@ -3,6 +3,7 @@ - Name: Create a Distribution Team - Start Date: 2020-09-03 - Author(s): [@jkutner](@jkutner) +- Status: Superseded - RFC Pull Request: [rfcs#113](https://github.com/buildpacks/rfcs/pull/113) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/community#35](https://github.com/buildpacks/community/issues/35) diff --git a/text/0063-stack-images-config-env-path-must-exist.md b/text/0063-stack-images-config-env-path-must-exist.md index 4582cabc4..7693315c0 100644 --- a/text/0063-stack-images-config-env-path-must-exist.md +++ b/text/0063-stack-images-config-env-path-must-exist.md @@ -3,6 +3,7 @@ - Name: Build/Run image configs MUST contain env.PATH - Start Date: 2020-08-21 - Author(s): @aemengo @ameyer-pivotal @mvalliath @TisVictress @micahyoung +- Status: Implemented - RFC Pull Request: [rfcs#114](https://github.com/buildpacks/rfcs/pull/114) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/spec#150](https://github.com/buildpacks/spec/issues/150), [buildpacks/docs#223](https://github.com/buildpacks/docs/issues/223) diff --git a/text/0064-buildpacks-contribute-default-process-type.md b/text/0064-buildpacks-contribute-default-process-type.md index d42e7688b..3466a605d 100644 --- a/text/0064-buildpacks-contribute-default-process-type.md +++ b/text/0064-buildpacks-contribute-default-process-type.md @@ -3,6 +3,7 @@ - Name: Buildpacks should be able to define the default process type for an app - Start Date: 8/20/20 - Author(s): natalieparellano +- Status: Implemented - RFC Pull Request: [rfcs#110](https://github.com/buildpacks/rfcs/pull/110) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/spec#159](https://github.com/buildpacks/spec/issues/159), [buildpacks/spec#160](https://github.com/buildpacks/spec/issues/160), [buildpacks/lifecycle#457](https://github.com/buildpacks/lifecycle/issues/457), [buildpacks/lifecycle#458](https://github.com/buildpacks/lifecycle/issues/458), [buildpacks/pack#946](https://github.com/buildpacks/pack/issues/946), [buildpacks/docs#246](https://github.com/buildpacks/docs/issues/246) diff --git a/text/0065-builder-spec.md b/text/0065-builder-spec.md index 15602b33f..dec3a283b 100644 --- a/text/0065-builder-spec.md +++ b/text/0065-builder-spec.md @@ -3,6 +3,7 @@ - Name: Builder Spec - Start Date: 2020-09-11 - Author(s): [@dfreilich](https://github.com/dfreilich) +- Status: Approved - RFC Pull Request: [rfcs#116](https://github.com/buildpacks/rfcs/pull/116) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/spec#101](https://github.com/buildpacks/spec/issues/101), [buildpacks/pack#945](https://github.com/buildpacks/pack/issues/945) diff --git a/text/0066-lifecycle-prelease-version-and-experimental-section.md b/text/0066-lifecycle-prelease-version-and-experimental-section.md index bd503f1b3..165ad4f41 100644 --- a/text/0066-lifecycle-prelease-version-and-experimental-section.md +++ b/text/0066-lifecycle-prelease-version-and-experimental-section.md @@ -3,6 +3,7 @@ - Name: Prelease APIs and Experimental Features - Start Date: 2020-08-25 - Author(s): @hone +- Status: Approved - RFC Pull Request: [rfcs#115](https://github.com/buildpacks/rfcs/pull/115) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/spec#161](https://github.com/buildpacks/spec/issues/161), [buildpacks/lifecycle#459](https://github.com/buildpacks/lifecycle/issues/459), [buildpacks/lifecycle#460](https://github.com/buildpacks/lifecycle/issues/460), [buildpacks/pack#947](https://github.com/buildpacks/pack/issues/947) @@ -36,7 +37,7 @@ A prerelease API is a non-finalized API but is in a testable state. The the API When Lifecycle supports a prerelease API, it can be treated like any other API version and be used in any mode: `experimental`, `supported`, `deprecated`. ### Experimental Features in the API -For APIs that will need to stabilize over a long time span, they can get added as an "experimental" section inside the API as **Experimental Features**. These will go out into official API releases. Using that part of the API will be experimental and susceptible to change in an upcoming release. This will require a bit more rigor, but will allow us to evolve the experimental sections overtime without as much pressure to "get it right" or "block a release". The downside is that the lifecycle will need to suppert these experimental features since they're part of the API. If the experimental API changes a lot, lifecycle will need to support these differences. +For APIs that will need to stabilize over a long time span, they can get added as an "experimental" section inside the API as **Experimental Features**. These will go out into official API releases. Using that part of the API will be experimental and susceptible to change in an upcoming release. This will require a bit more rigor, but will allow us to evolve the experimental sections overtime without as much pressure to "get it right" or "block a release". The downside is that the lifecycle will need to support these experimental features since they're part of the API. If the experimental API changes a lot, lifecycle will need to support these differences. There will be a new `CNB_PLATFORM_EXPERIMENTAL_FEATURES` environment variable to control ALL experimental features by the platform. This will let a platform decide if they want to turn on experimental features. For now, all features will be turned on/off for simplicity. If the need arises, a list in the future can be explored. diff --git a/text/0067-report-toml-image-manifest-size.md b/text/0067-report-toml-image-manifest-size.md index 3ddf0929d..0f734fee0 100644 --- a/text/0067-report-toml-image-manifest-size.md +++ b/text/0067-report-toml-image-manifest-size.md @@ -3,6 +3,7 @@ - Name: Add Image Manifest Size to report.toml - Start Date: 2020-11-02 - Author(s): djoyahoy +- Status: Implemented - RFC Pull Request: [rfcs#121](https://github.com/buildpacks/rfcs/pull/121) - CNB Pull Request: - CNB Issue: [buildpacks/spec#169](https://github.com/buildpacks/spec/issues/169), [buildpacks/lifecycle#490](https://github.com/buildpacks/lifecycle/issues/490) diff --git a/text/0068-buildpack-registry-search-api.md b/text/0068-buildpack-registry-search-api.md index edce91d72..aa67b1955 100644 --- a/text/0068-buildpack-registry-search-api.md +++ b/text/0068-buildpack-registry-search-api.md @@ -3,6 +3,7 @@ - Name: Buildpack Registry Search API - Start Date: 2020-11-11 - Author(s): @elbandito +- Status: Implemented - RFC Pull Request: [rfcs#125](https://github.com/buildpacks/rfcs/pull/125) - CNB Pull Request: (leave blank) - CNB Issue: [registry-api#3](https://github.com/buildpacks/registry-api/issues/3) diff --git a/text/0069-stack-buildpacks.md b/text/0069-stack-buildpacks.md index bf1c89e8c..fb2d28adb 100644 --- a/text/0069-stack-buildpacks.md +++ b/text/0069-stack-buildpacks.md @@ -3,6 +3,7 @@ - Name: Stack Buildpacks - Start Date: 2020-08-27 - Author(s): [@jkutner](@jkutner) +- Status: Superseded - RFC Pull Request: [rfcs#111](https://github.com/buildpacks/rfcs/pull/111) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/spec#177](https://github.com/buildpacks/spec/issues/177), [buildpacks/spec#178](https://github.com/buildpacks/spec/issues/178), [buildpacks/spec#170](https://github.com/buildpacks/spec/issues/170), [buildpacks/lifecycle#501](https://github.com/buildpacks/lifecycle/issues/501), [buildpacks/pack#1021](https://github.com/buildpacks/pack/issues/1021) diff --git a/text/0070-new-buildpack-toml-keys.md b/text/0070-new-buildpack-toml-keys.md index b5f40f82a..62287f05f 100644 --- a/text/0070-new-buildpack-toml-keys.md +++ b/text/0070-new-buildpack-toml-keys.md @@ -3,6 +3,7 @@ - Name: New Buildpack Descriptor Keys - Start Date: 2020-12-12 - Author(s): [@jkutner](@jkutner) +- Status: Implemented - RFC Pull Request: [rfcs#127](https://github.com/buildpacks/rfcs/pull/127) - CNB Pull Request: [buildpacks/pack#1022](https://github.com/buildpacks/pack/pull/1022) - CNB Issue: [buildpacks/spec#181](https://github.com/buildpacks/spec/issues/181), [buildpacks/docs#288](https://github.com/buildpacks/docs/issues/288) diff --git a/text/0071-cnb-user-research-2021.md b/text/0071-cnb-user-research-2021.md index ebd4a7c8a..c0efc8ba8 100644 --- a/text/0071-cnb-user-research-2021.md +++ b/text/0071-cnb-user-research-2021.md @@ -3,6 +3,7 @@ - Name: CNB User Research 2021 - Start Date: 2020-12-10 - Author(s): [@sampeinado](@sampeinado) +- Status: Implemented - RFC Pull Request: [rfcs#126](https://github.com/buildpacks/rfcs/pull/126) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0072-image-workdir.md b/text/0072-image-workdir.md index 20d265333..b81b4f318 100644 --- a/text/0072-image-workdir.md +++ b/text/0072-image-workdir.md @@ -3,6 +3,7 @@ - Name: Set Image Working Directory to value of CNB_APP_DIR - Start Date: 2021-01-13 - Author(s): @josegonzalez +- Status: Implemented - RFC Pull Request: [rfcs#134](https://github.com/buildpacks/rfcs/pull/134) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/spec#183](https://github.com/buildpacks/spec/issues/183), [buildpacks/lifecycle#516](https://github.com/buildpacks/lifecycle/issues/516), [buildpacks/docs#290](https://github.com/buildpacks/docs/issues/290) diff --git a/text/0073-offline-buildpackages.md b/text/0073-offline-buildpackages.md index 66eb9995b..b875fc907 100644 --- a/text/0073-offline-buildpackages.md +++ b/text/0073-offline-buildpackages.md @@ -3,11 +3,16 @@ - Name: Asset Cache - Start Date: 2020-04-27 - Author(s): dwillist +- Status: On Hold - RFC Pull Request: [rfcs#81](https://github.com/buildpacks/rfcs/pull/81) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/spec#184](https://github.com/buildpacks/spec/issues/184), [buildpacks/pack#1055](https://github.com/buildpacks/pack/issues/1055), [buildpacks/lifecycle#521](https://github.com/buildpacks/lifecycle/issues/521) - Supersedes: NA +# Notes +## On Hold +Since the approval of this RFC, additional concerns around multiple caching strategies within CNB has been raised. Solving this problem is still a priority for the project but the core team has decided to place this RFC on hold pending further discussion of a more holistic caching solution. If you are interested in this problem, please participate in any of our channels. + # Summary [summary]: #summary diff --git a/text/0074-layer-table.md b/text/0074-layer-table.md index 4f5873a32..3adcc0123 100644 --- a/text/0074-layer-table.md +++ b/text/0074-layer-table.md @@ -3,6 +3,7 @@ - Name: Move layer types to new table in layer.toml - Start Date: 2021-01-05 - Author(s): natalieparellano +- Status: Implemented - RFC Pull Request: [rfcs#132](https://github.com/buildpacks/rfcs/pull/132) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/spec#185](https://github.com/buildpacks/spec/issues/185), [buildpacks/lifecycle#522](https://github.com/buildpacks/lifecycle/issues/522), [buildpacks/docs#296](https://github.com/buildpacks/docs/issues/296) diff --git a/text/0075-move-analyze-phase.md b/text/0075-move-analyze-phase.md index 03a4ba9a0..3724619b7 100644 --- a/text/0075-move-analyze-phase.md +++ b/text/0075-move-analyze-phase.md @@ -3,6 +3,7 @@ - Name: Move analyze phase - Start Date: 2021-01-13 - Author(s): [@jkutner](github.com/jkutner/) [@jabrown85](github.com/jabrown85) +- Status: Implemented - RFC Pull Request: [rfcs#135](https://github.com/buildpacks/rfcs/pull/135) - CNB Pull Request: [buildpacks/spec#172](https://github.com/buildpacks/spec/pull/172) - CNB Issue: [buildpacks/spec#194](https://github.com/buildpacks/spec/issues/194), [buildpacks/lifecycle#530](https://github.com/buildpacks/lifecycle/issues/530), [buildpacks/pack#1078](https://github.com/buildpacks/pack/issues/1078), [buildpacks/docs#302](https://github.com/buildpacks/docs/issues/302) diff --git a/text/0076-windows-security-identifiers.md b/text/0076-windows-security-identifiers.md index d3bc19827..c939be4ad 100644 --- a/text/0076-windows-security-identifiers.md +++ b/text/0076-windows-security-identifiers.md @@ -3,6 +3,7 @@ - Name: Use Security Identifiers for Windows User/Group IDs - Start Date: 2021-01-25 - Author(s): micahyoung +- Status: Approved - RFC Pull Request: [rfcs#133](https://github.com/buildpacks/rfcs/pull/133) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/spec#129](https://github.com/buildpacks/spec/issues/129), [buildpacks/lifecycle#343](https://github.com/buildpacks/lifecycle/issues/343), [buildpacks/pack#1079](https://github.com/buildpacks/pack/issues/1079), [buildpacks/docs#303](https://github.com/buildpacks/docs/issues/303) diff --git a/text/0077-pack-buildpack-create.md b/text/0077-pack-buildpack-create.md index f422dd764..3592796b0 100644 --- a/text/0077-pack-buildpack-create.md +++ b/text/0077-pack-buildpack-create.md @@ -3,6 +3,7 @@ - Name: Pack Command to Create a Buildpack Repo - Start Date: 2021-01-19 - Author(s): [jkutner](https://github.com/jkutner) +- Status: Superseded - RFC Pull Request: [rfcs#136](https://github.com/buildpacks/rfcs/pull/136) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/pack#1025](https://github.com/buildpacks/pack/issues/1025) diff --git a/text/0078-group-additions.md b/text/0078-group-additions.md index 91ffc7685..9eed25872 100644 --- a/text/0078-group-additions.md +++ b/text/0078-group-additions.md @@ -3,6 +3,7 @@ - Name: Group additions to Builder order - Start Date: 2020-12-23 - Author(s): [jkutner](@jkutner) +- Status: Approved - RFC Pull Request: [rfcs#129](https://github.com/buildpacks/rfcs/pull/129) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/docs#319](https://github.com/buildpacks/docs/issues/319), [buildpacks/pack#1099](https://github.com/buildpacks/pack/issues/1099), [buildpacks/pack#1100](https://github.com/buildpacks/pack/issues/1100), [buildpacks/spec#195](https://github.com/buildpacks/spec/issues/195) diff --git a/text/0079-create-stackify-repo.md b/text/0079-create-stackify-repo.md index d3d113cae..2c73ab470 100644 --- a/text/0079-create-stackify-repo.md +++ b/text/0079-create-stackify-repo.md @@ -3,6 +3,7 @@ - Name: Create stackify repo - Start Date: 2020-11-06 - Authors: @martyspiewak @mdelillo @dumez-k +- Status: Superseded - RFC Pull Request: [rfcs#123](https://github.com/buildpacks/rfcs/pull/123) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/pack#1101](https://github.com/buildpacks/pack/issues/1101), [buildpacks/docs#321](https://github.com/buildpacks/docs/issues/321) diff --git a/text/0080-builder-key-in-project-descriptor.md b/text/0080-builder-key-in-project-descriptor.md index b71cc6c76..a262cee9d 100644 --- a/text/0080-builder-key-in-project-descriptor.md +++ b/text/0080-builder-key-in-project-descriptor.md @@ -3,6 +3,7 @@ - Name: Builder key in project descriptor - Start Date: 2021-02-11 - Author(s): @wburningham +- Status: Implemented - RFC Pull Request: [rfcs#138](https://github.com/buildpacks/rfcs/pull/138) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/spec#215](https://github.com/buildpacks/spec/issues/215), [buildpacks/pack#1114](https://github.com/buildpacks/pack/issues/1114), [buildpacks/docs#345](https://github.com/buildpacks/docs/issues/345) diff --git a/text/0081-process-specific-working-directory.md b/text/0081-process-specific-working-directory.md index b22fca9ab..60ccd1ae3 100644 --- a/text/0081-process-specific-working-directory.md +++ b/text/0081-process-specific-working-directory.md @@ -3,6 +3,7 @@ - Name: Process Specific Working Directory - Start Date: 2021-03-09 - Author(s): ForestEckhardt, ryanmoran, fg-j +- Status: Implemented - RFC Pull Request: [rfcs#144](https://github.com/buildpacks/rfcs/pull/144) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/spec#212](https://github.com/buildpacks/spec/issues/212), [buildpacks/spec#216](https://github.com/buildpacks/spec/issues/216) diff --git a/text/0082-pack-package-refactor.md b/text/0082-pack-package-refactor.md index 871c6e8b1..2a67c4a8d 100644 --- a/text/0082-pack-package-refactor.md +++ b/text/0082-pack-package-refactor.md @@ -3,6 +3,7 @@ - Name: Refactor Pack go packages - Start Date: 02/24/21 - Author(s): dwillist +- Status: Implemented - RFC Pull Request: [rfcs#139](https://github.com/buildpacks/rfcs/pull/139) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/pack#1128](https://github.com/buildpacks/pack/issues/1128) diff --git a/text/0083-best-practices-and-guidelines.md b/text/0083-best-practices-and-guidelines.md index 6a2f1d283..cfe9bcfe0 100644 --- a/text/0083-best-practices-and-guidelines.md +++ b/text/0083-best-practices-and-guidelines.md @@ -3,6 +3,7 @@ - Name: Best practices and guidelines for Cloud Native Buildpacks - Start Date: 2021-03-29 - Author(s): [@samj1912](https://github.com/samj1912) +- Status: Implemented - RFC Pull Request: [rfcs#150](https://github.com/buildpacks/rfcs/pull/150) - CNB Pull Request: (leave blank) - CNB Issue: N/A diff --git a/text/0084-project-descriptor-domains.md b/text/0084-project-descriptor-domains.md index 3de4815ed..94cdd65cc 100644 --- a/text/0084-project-descriptor-domains.md +++ b/text/0084-project-descriptor-domains.md @@ -3,6 +3,7 @@ - Name: Project Descriptor Domains - Start Date: 2021-02-26 - Author(s): hone +- Status: Implemented - RFC Pull Request: [rfcs#140](https://github.com/buildpacks/rfcs/pull/140) - CNB Pull Request: (leave blank) - CNB Issue: N/A diff --git a/text/0085-run-uid.md b/text/0085-run-uid.md index 0de420daf..3b2ae4237 100644 --- a/text/0085-run-uid.md +++ b/text/0085-run-uid.md @@ -3,6 +3,7 @@ - Name: Recommending different run-time and build-time users - Start Date: 2021-03-15 - Author(s): [@samj1912](https://github.com/samj1912) +- Status: Implemented - RFC Pull Request: [rfcs#146](https://github.com/buildpacks/rfcs/pull/146) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/spec#223](https://github.com/buildpacks/spec/issues/223) diff --git a/text/0086-component-level-contrib.md b/text/0086-component-level-contrib.md index 64e7e6bca..fe88fe3f1 100644 --- a/text/0086-component-level-contrib.md +++ b/text/0086-component-level-contrib.md @@ -3,6 +3,7 @@ - Name: Guidelines for Component-level Contributions - Start Date: 2021-03-08 - Author(s): @sclevine +- Status: Approved - RFC Pull Request: [rfcs#143](https://github.com/buildpacks/rfcs/pull/143) - CNB Pull Request: (leave blank) - CNB Issue: [community#93](https://github.com/buildpacks/community/issues/93) diff --git a/text/0087-bom-in-layer-metadata.md b/text/0087-bom-in-layer-metadata.md index 8beb05d70..aa0f294e7 100644 --- a/text/0087-bom-in-layer-metadata.md +++ b/text/0087-bom-in-layer-metadata.md @@ -3,6 +3,7 @@ - Name: BOM inclusion in layer content metadata - Start Date: 2021-04-01 - Author(s): [@samj1912](https://github.com/samj1912) +- Status: Superseded - RFC Pull Request: [rfcs#151](https://github.com/buildpacks/rfcs/pull/151) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/spec#227](https://github.com/buildpacks/spec/issues/227), [buildpacks/lifecycle#623](https://github.com/buildpacks/lifecycle/issues/623), [buildpacks/docs#375](https://github.com/buildpacks/docs/issues/375) diff --git a/text/0088-minimum-docs-lifecycle.md b/text/0088-minimum-docs-lifecycle.md index 1fbd1811d..2d9ff6c24 100644 --- a/text/0088-minimum-docs-lifecycle.md +++ b/text/0088-minimum-docs-lifecycle.md @@ -3,6 +3,7 @@ - Name: Define a minimum standard for docs in order to ship the lifecycle - Start Date: 4/7/2021 - Author(s): natalieparellano +- Status: Implemented - RFC Pull Request: [rfcs#153](https://github.com/buildpacks/rfcs/pull/153) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/lifecycle#627](https://github.com/buildpacks/lifecycle/issues/627) diff --git a/text/0089-buildpack-authors-tooling-subteam.md b/text/0089-buildpack-authors-tooling-subteam.md index 9128faf0b..887d2f1fe 100644 --- a/text/0089-buildpack-authors-tooling-subteam.md +++ b/text/0089-buildpack-authors-tooling-subteam.md @@ -3,6 +3,7 @@ - Name: Buildpack Authors' Tooling Sub-Team - Start Date: 2021-05-11 - Author(s): [@ekcasey](https://github.com/ekcasey), [@hone](https://github.com/hone), [@samj1912](https://github.com/samj1912) +- Status: Implemented - RFC Pull Request: [rfcs#159](https://github.com/buildpacks/rfcs/pull/159) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/community#102](https://github.com/buildpacks/community/issues/102) diff --git a/text/0090-intro-video-script.md b/text/0090-intro-video-script.md index 2358bb031..f4b2f44b0 100644 --- a/text/0090-intro-video-script.md +++ b/text/0090-intro-video-script.md @@ -3,6 +3,7 @@ - Name: Intro video script - Start Date: 2021-05-18 - Author(s): yaelharel +- Status: Implemented - RFC Pull Request: (leave blank) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) diff --git a/text/0091-pack-cache-options.md b/text/0091-pack-cache-options.md index 37e3d6f4a..b62203af9 100644 --- a/text/0091-pack-cache-options.md +++ b/text/0091-pack-cache-options.md @@ -3,6 +3,7 @@ - Name: Pack cache options - Start Date: 2021-03-25 - Author(s): [@jromero](https://github.com/jromero), [@dwillist](https://github.com/dwillist) +- Status: Implemented - RFC Pull Request: [rfcs#149](https://github.com/buildpacks/rfcs/pull/149) - CNB Pull Request: (leave blank) - CNB Issue: [pack#1077](https://github.com/buildpacks/pack/issues/1077) diff --git a/text/0092-add-visual-pack-build.md b/text/0092-add-visual-pack-build.md index 7d1d271f8..8a56d5ee4 100644 --- a/text/0092-add-visual-pack-build.md +++ b/text/0092-add-visual-pack-build.md @@ -3,6 +3,7 @@ - Name: Add Visual Pack Build - Start Date: 2020-05-14 - Author(s): aemengo +- Status: Implemented - RFC Pull Request: [rfcs#160](https://github.com/buildpacks/rfcs/pull/160) - CNB Pull Request: (leave blank) - CNB Issue: [pack#1200](https://github.com/buildpacks/pack/issues/1200), [pack#1201](https://github.com/buildpacks/pack/issues/1201), [pack#1203](https://github.com/buildpacks/pack/issues/1203), [pack#1204](https://github.com/buildpacks/pack/issues/1204), [pack#1205](https://github.com/buildpacks/pack/issues/1205), [pack#1206](https://github.com/buildpacks/pack/issues/1206) diff --git a/text/0093-remove-shell-processes.md b/text/0093-remove-shell-processes.md index ab54f63f7..9a7f1876f 100644 --- a/text/0093-remove-shell-processes.md +++ b/text/0093-remove-shell-processes.md @@ -3,6 +3,7 @@ - Name: Remove Shell Processes - Start Date: 2021-05-14 - Author(s): @ekcasey +- Status: Implemented - RFC Pull Request: [rfcs#168](https://github.com/buildpacks/rfcs/pull/168) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/spec#244](https://github.com/buildpacks/spec/issues/244), [buildpacks/spec#245](https://github.com/buildpacks/spec/issues/245), [buildpacks/lifecycle#693](https://github.com/buildpacks/lifecycle/issues/693), [buildpacks/pack#1260](https://github.com/buildpacks/pack/issues/1260), [buildpacks/docs#391](https://github.com/buildpacks/docs/issues/391), [buildpacks/libcnb#70](https://github.com/buildpacks/libcnb/issues/70) @@ -87,15 +88,12 @@ The following changes have been made: - **`direct` has been removed** - all processes are executed directly. If `bash` or `cmd.exe` is required it should be included in `command`. No surprises. - **`command` is now an array** - Arguments in `command` *will not* be overwritten if a user provides additional arguments at runtime. `args` are default arguments that *will* be overwritten if a user provides additional arguments at runtime. The makes `command` analogous to `Entrypoint` in the OCI spec and `command` in a Kubernetes PodSpec. `args` is analogous to `cmd` and `args` in docker and Kubernetes respectively. -## Using Environment Variables in a Process - -One upside to our previous execution strategy was that it enable users to include environment variable references in arguments that were later evaluated in the container. To preserve this feature we can instead adopt the Kubernetes strategy for [environment variables interpolation](https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#using-environment-variables-inside-of-your-config). If a buildpack or user includes `$()` in the `command` or `args` and `` is the name of an environment variable set in the launch environment, the launcher will replace this string with the value of the environment variable after apply buildpack-provided env modifications and before launching the process. - # How it Works [how-it-works]: #how-it-works ## Buildpack-provided process types + ### Example 1 - A Shell Process The Paketo .Net Execute Buildpack may generates shell processes similar to the following: ``` @@ -109,12 +107,11 @@ NOTE: the buildpack API used by this buildpack (`0.5`) predates the introduction Using the new API this process could look like: ``` [[processes]] -type = "web" -command = ["dotnet", "my-app.dll", "--urls", "http://0.0.0.0:$(PORT)"] # the default value of PORT would need to be provided in a layer +type = "bash" +command = ["bash", "-c", "dotnet", "my-app.dll", "--urls", "http://0.0.0.0:${PORT:-8080}"] default = true ``` Things to note: -* In the above example I have eliminated the dependency on Bash instead of explicitly adding it to the command, because it is likely unnecessary. * If the buildpack authors believed that `--urls` should be overridable they could set move the last two arguments from `command` to `args`. ### Example 2 - A Script Process @@ -138,39 +135,9 @@ command = ["bash", "-c", "pre-start.sh && nodejs server.js && post-start.sh"] default = false ``` -## User Provided Processes - -Currently if the user can specify a custom process dynamically at runtime by setting the container entrypoint to `launcher` directly rather than using a symlink to the launcher, the providing a custom `cmd`. This custom command is executed directly if `cmd` is an array and the first element is `--`. Otherwise the custom command is assumed to be a shell process. In the interest of removing complexity we should do away with the special `--` argument and execute all custom commands directly. - -### Example 1 - A Direct process -The follow direct commands: -``` -docker run --entrypoint launcher -- env -docker run --entrypoint launcher -- echo hello '$WORLD' -``` -will become the following, using the new platform API -``` -docker run --entrypoint launcher env -docker run --entrypoint launcher echo hello '$(WORLD)' -``` -Previously, in the second command in this example, `$WORLD` would not have been interpolated because this is a direct process; instead the output would include the literal string `$WORLD`. With the changes proposed, `$(WORLD)` will now be evaluated, even though the process is direct. - -### Example 2 - A Shell Process -The follow custom shell command: -``` -docker run --entrypoint launcher echo hello '${WORLD}' -docker run --entrypoint launcher echo hello '${WORLD:-world}' -``` -will become the following, using the new platform API -``` -docker run --entrypoint launcher echo hello '$(WORLD)' -docker run --entrypoint launcher bash -c 'echo hello "${WORLD:-world}"' -``` -The first command in this example needed to adopt the new environment variable syntax to behave as expected with the new API. Previously it was necessary to use a shell process in order to evaluate `${WORLD}`. Now, the shell is unnecessary. - -If the user wishes, they may explicitly invoke a shell and let Bash handle the interpolation, which provides a richer feature set. +## User-provided process types -### Example 3 - A Script Process +### Example 1 - A Script Process The follow custom script command: ``` docker run --entrypoint launcher 'for opt in $JAVA_OPTS; do echo $opt; done' @@ -180,35 +147,6 @@ will become the following, using the new platform API docker run --entrypoint launcher bash -c 'for opt in $JAVA_OPTS; do echo $opt; done' ``` -### Example 4 - A Script Process in Kubernetes - -Because we have adopted the Kubernetes environment variable notation here, users may need to escape some references in their PodSpec in specific situations. This is necessary only if all of the following are true: -* The user is providing a `command` or `args` which contain an environment variable reference. -* The variable is explicitly initialized in the `env` section of the PodSpec. -* The user wishes for the variable to be interpolated **after** build-provided env modifications have been applied. - -``` -apiVersion: v1 -kind: Pod -metadata: - name: env-example -spec: - containers: - - name: env-print-demo - image: bash - env: - - name: IN_CONTAINER_1 - value: "k8s-val" - - name: IN_K8S - value: "val2" - command: ["bash", "-c", "echo $$(IN_CONTAINER_1)) $(IN_CONTAINER_2) $(IN_K8S) ${IN_BASH}"] -``` -In the above example the environment variables will be interpolated as follows: -- `$IN_CONTAINER` - Interpolated by the launcher after buildpack-provided modifications (e.g. `k8s-val:buildpack-appended-val`) -- `$IN_CONTAINER_2` - Interpolated by the launcher after buildpack-provided modifications. No escaping is required here because `$IN_CONTAINER_2` is not set in `env`. -- `$IN_K8S` - Interpolated by Kubernetes before the container runs. Buildpack-provided modifications will not affect the resulting value. -- `$IN_BASH` - Interpolated by Bash. - ## What About Profile Scripts? ### `/profile.d/*` @@ -294,7 +232,6 @@ we would convert it to the following in `metadata.toml` type = "hi" command = ["echo", "hello", "world"] args = [] -api = "0.5" direct = true ``` This spares users from having to learn about the differences between buildpack APIs in order to predict how additional arguments will behave. @@ -318,7 +255,6 @@ It should be converted to the following in `metadata.toml` type = "hi" command = ["echo", "hello", "${WORLD:-world}"] args = [] -api = "0.5" direct = false ``` @@ -338,7 +274,6 @@ When migrating to the new API, buildpack authors should take the following steps 1. Does the buildpack contribute a `direct=false` process? If so, there are two options: 1. Explicitly add `bash` or `cmd` to the process definition, this is the safest path forward. 2. Remove an unnecessary dependency on `bash` or `cmd` by: - * Changing any environment variable references to use the `$())` syntax - -We could use our own explicit syntax like `$(())` or environment variable replacements instead of `$()`. - -pros: -* Extremely explicit -* There is never any need to escape values in k8s -* Eliminates all conflicts or situations where the evaluation order must be explained - -cons: -* A third syntax -* A buildpack-specific syntax that users must learn about through documentation - -## `${}` syntax -We could use Bash-like `${}` syntax for environment variable replacements instead of `$()`. - -pros: -* It might "just work" in a lot of cases -* Familiar - -cons: -* The launcher might evaluate env vars that were truly intended for Bash to evaluate (e.g. in a command like `["bash", "-c", "source foo-setter.sh && echo ${FOO}"]`) -* Users might reasonably expect things like `$FOO` or `${FOO:-dafault-foo}` to work and discover the limitations only via trial and error or documentation. - ## Lifecycle support for `/.exec` When we remove support for `/.profile` we could add support for `/.exec` or similar where `/.exec` must implement the `exec.d` interface. @@ -425,15 +336,7 @@ However, things get very complicated if we explicitly source profiles or shell e The Paketo Java buildpacks have already converted an extensive collection of profile script to exec.d binaries and converted all processes to direct processes in order to support minimal stacks like `io.paketo.stacks.tiny` and to provide users with more intuitive argument handling. -The Procfile interface is supported by Paketo, Heroku, and Google buildpacks, demonstrating that it is possible to have a consistent interface across buildpack ecosystems without building direct support for that interface in the lifecycle. - -# Unresolved Questions -[unresolved-questions]: #unresolved-questions - -Do we need to provide a mechanism for turning `$(env)` interpolation off? E.g. -``` -docker run --env "CNB_INTERPOLATE=false" --entrypoint launcher echo hello '${WORLD}' # prints "hello '$(WORLD)'" -``` +The Procfile interface is supported by Paketo, Heroku, and Google buildpacks, demonstrating that it is possible to have a consistent interface across buildpack ecosystems without building direct support for that interface in the lifecycle. # Spec. Changes @@ -473,3 +376,28 @@ command = [""] args = [""] default = false ``` + +# History +[history]: #history + +## Amended +### Meta +[meta-1]: #meta-1 +- Name: Removed references to custom env templating +- Start Date: 2022-12-02 +- Author(s): natalieparellano +- Amendment Pull Request: (leave blank) + +### Summary + +As this is a breaking change, we decided to do this in a separate (yet to be created) RFC. + +Created issue: #258 + +In addition to the changes described originally in 0093 we'd like some way of versioning the launcher interface, to avoid surprising end-users. + +### Motivation + +Why was this amendment necessary? + +The RFC text should reflect what was actually implemented / agreed upon to avoid confusion. diff --git a/text/0094-add-status-field.md b/text/0094-add-status-field.md index be154b9aa..69347264a 100644 --- a/text/0094-add-status-field.md +++ b/text/0094-add-status-field.md @@ -3,7 +3,7 @@ - Name: Add Status to RFC Metadata - Start Date: 2021-07-14 - Author(s): @aemengo, @ekcasey -- Status: Draft Approved +- Status: Implemented - RFC Pull Request: [rfcs#177](https://github.com/buildpacks/rfcs/pull/177) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/rfcs#183](#https://github.com/buildpacks/rfcs/issues/183) @@ -70,6 +70,12 @@ Since the approval of this RFC additional concerns X, Y, and Z have been raised. If at a later date the core team or responsible sub-team wishes to move the RFC from "On Hold" back to "Approved", this should be done via PR and another note should be added, describing the resolution of the discussion and the rationale for proceeding with the original RFC. +When an RFC is fully implemented (i.e., the associated tracking issue is closed), then change is reflected like so: + +``` +- Status: Implemented +``` + # Drawbacks [drawbacks]: #drawbacks @@ -90,10 +96,33 @@ If at a later date the core team or responsible sub-team wishes to move the RFC [unresolved-questions]: #unresolved-questions - Should **"Implemented"** be one of the accepted values? - - Perhaps, but then what's the process of keeping the _Implemented_ status in sync? + - Yes, the RFC shepherd should create a PR updating the status from `Approved` to `Implemented` when the tracking issue is closed. - How do we retroactively add a **Status:** field to prior RFCs? - Manually? - How do we validate the [#meta](#meta) section of prior RFCs? - Perhaps we make a [../validate-rfcs.sh](../validate-rfcs.sh) script that lints the **#meta** section of all rfcs? + +# History +[history]: #history + +## Amended +### Meta +[meta-1]: #meta-1 +- Name: Add Implemented +- Start Date: 2022-12-01 +- Author(s): natalieparellano +- Amendment Pull Request: (leave blank) + +### Summary + +Added `Implemented` as a valid status. + +### Motivation + +Why was this amendment necessary? + +It can be confusing when reviewing older RFCs to know if they were implemented or not. +This can lead to RFCs being "forgotten". +The introduction of [tracking issues](https://github.com/buildpacks/rfcs/blob/main/.github/ISSUE_TEMPLATE/tracking.md) to our process can help ensure that the status gets updated appropriately. diff --git a/text/0095-sbom.md b/text/0095-sbom.md index c96f43c17..740f81a5b 100644 --- a/text/0095-sbom.md +++ b/text/0095-sbom.md @@ -3,6 +3,7 @@ - Name: Structured SBOMs - Start Date: 2021-06-02 - Author(s): [@samj1912](https://github.com/samj1912), [@sophiewigmore](https://github.com/sophiewigmore), [@ForestEckhardt](https://github.com/ForestEckhardt) +- Status: Implemented - RFC Pull Request: [rfcs#166](https://github.com/buildpacks/rfcs/pull/166) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/lifecycle#732](https://github.com/buildpacks/lifecycle/issues/732), [buildpacks/lifecycle#733](https://github.com/buildpacks/lifecycle/issues/733), [buildpacks/lifecycle#734](https://github.com/buildpacks/lifecycle/issues/734), [buildpacks/lifecycle#735](https://github.com/buildpacks/lifecycle/issues/735), [buildpacks/lifecycle#736](https://github.com/buildpacks/lifecycle/issues/736), [buildpacks/lifecycle#737](https://github.com/buildpacks/lifecycle/issues/737), [buildpacks/lifecycle#738](https://github.com/buildpacks/lifecycle/issues/738) diff --git a/text/0096-remove-stacks-mixins.md b/text/0096-remove-stacks-mixins.md index f2acc4238..34f17ea98 100644 --- a/text/0096-remove-stacks-mixins.md +++ b/text/0096-remove-stacks-mixins.md @@ -3,6 +3,7 @@ - Name: Remove Stacks & Mixins - Start Date: 2021-06-30 - Author(s): sclevine +- Status: Approved - RFC Pull Request: [rfcs#172](https://github.com/buildpacks/rfcs/pull/172) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/pack#1301](https://github.com/buildpacks/pack/issues/1301), [buildpacks/spec#258](https://github.com/buildpacks/spec/issues/258), [buildpacks/samples#111](https://github.com/buildpacks/samples/issues/111), [buildpacks/docs#422](https://github.com/buildpacks/docs/issues/422), [buildpacks/spec#259](https://github.com/buildpacks/spec/issues/259), [buildpacks/spec#260](https://github.com/buildpacks/spec/issues/260), [buildpacks/lifecycle#742](https://github.com/buildpacks/lifecycle/issues/742), [buildpacks/lifecycle#743](https://github.com/buildpacks/lifecycle/issues/743), [buildpacks/lifecycle#744](https://github.com/buildpacks/lifecycle/issues/744), [buildpacks/pack#1302](https://github.com/buildpacks/pack/issues/1302), [buildpacks/pack#1303](https://github.com/buildpacks/pack/issues/1303), [buildpacks/pack#1304](https://github.com/buildpacks/pack/issues/1304) @@ -58,8 +59,8 @@ For Windows-based images, Distribution should be empty. Version should be the [s The `stacks` list in `buildpack.toml` is replaced by a `targets` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS, Architecture, and Variant. -If the `targets` list is empty and `/bin/build` is present, a target with `os = "linux"` and `arch = "x86_64"` is assumed by tools reading `buildpack.toml`. -If the `targets` list is empty and `/bin/build.bat` or `/bin/build.exe` is present, a target with `os = "windows"` and `arch = "x86_64"` is assumed by tools reading `buildpack.toml`. +If the `targets` list is empty and `/bin/build` is present, a target with `os = "linux"` and `arch = "amd64"` is assumed by tools reading `buildpack.toml`. +If the `targets` list is empty and `/bin/build.bat` or `/bin/build.exe` is present, a target with `os = "windows"` and `arch = "amd64"` is assumed by tools reading `buildpack.toml`. App image builds fail if the build image and selected run image have mismatched metadata. We may introduce flags or additional labels to skip this validation (e.g., for cross-compilation or minimal runtime base images). An image without a specified Distribution is compatible with images specifying any Distribution. @@ -72,14 +73,14 @@ When an app image is rebased, `rebaser` must fail if the new run image and previ ```toml [[targets]] os = "linux" -arch = "x86_64" +arch = "amd64" [[targets.distributions]] name = "ubuntu" versions = ["18.04", "20.04"] [[targets]] os = "linux" -arch = "x86_64" +arch = "amd64" [[targets.distributions]] name = "ubuntu" versions = ["14.04", "16.04"] @@ -151,7 +152,7 @@ If the newly-specified field values are missing, the lifecycle and pack may used ``` config.os = "linux" -config.architecture = "x86_64" +config.architecture = "amd64" io.buildpacks.distribution.name = "ubuntu" io.buildpacks.distribution.version = "18.04" ``` @@ -168,7 +169,7 @@ to ```toml [[targets]] os = "linux" -arch = "x86_64" +arch = "amd64" [[targets.distributions]] name = "ubuntu" versions = ["18.04"] @@ -200,3 +201,12 @@ versions = ["18.04"] [spec-changes]: #spec-changes This RFC requires extensive changes to all specifications. + +## Amended +### Summary + +rename x86_64 -> amd64 in keeping with all other usages of arch. descriptors. + +### Motivation + +This is how we do it everywhere else, this is the way. diff --git a/text/0097-official-utility-buildpacks.md b/text/0097-official-utility-buildpacks.md index 4e7477090..e5e6beb75 100644 --- a/text/0097-official-utility-buildpacks.md +++ b/text/0097-official-utility-buildpacks.md @@ -7,6 +7,7 @@ - CNB Pull Request: (leave blank) - CNB Issue: N/A - Supersedes: N/A +- Status: Implemented # Summary [summary]: #summary diff --git a/text/0084-rfc-issue-generation.md b/text/0098-rfc-issue-generation.md similarity index 99% rename from text/0084-rfc-issue-generation.md rename to text/0098-rfc-issue-generation.md index 6a95cde61..4effb3978 100644 --- a/text/0084-rfc-issue-generation.md +++ b/text/0098-rfc-issue-generation.md @@ -3,6 +3,7 @@ - Name: RFC Issue Generation - Start Date: 2020-03-03 - Author(s): @jromero +- Status: Implemented - RFC Pull Request: [rfcs#141](https://github.com/buildpacks/rfcs/pull/141) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/rfcs#158](https://github.com/buildpacks/rfcs/issues/158) diff --git a/text/0099-slack.md b/text/0099-slack.md new file mode 100644 index 000000000..a9e3fb71f --- /dev/null +++ b/text/0099-slack.md @@ -0,0 +1,123 @@ +# Meta +[meta]: #meta +- Name: Migrate to CNCF Slack +- Start Date: 2021-12-07 +- Author(s): [@samj1912](https://github.com/samj1912) +- Status: Implemented +- RFC Pull Request: [rfcs#198](https://github.com/buildpacks/rfcs/pull/198) +- CNB Pull Request: (leave blank) +- CNB Issue: N/A +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +This RFC proposes that we move our independent slack instance to the CNCF slack instance. + +# Definitions +[definitions]: #definitions + +N/A + +# Motivation +[motivation]: #motivation + +CNCF slack is an umbrella slack enterprise instance that quite a few CNCF projects use. As Buildpacks is maturing as a CNCF project, using the CNCF slack will put us closer to our sibling projects and their community. The CNCF slack also comes with CNCF support around spam prevention, moderation and integration with LFX insights. This will help us manage and track the growth of our community. CNCF slack is also an enterprise instance which means that we will have unlimited retention of messages and the ability to create people groups to mention maintainer teams. We will also be able to use private channels if needed. + +As a part of CNCF slack onboarding, we can also migrate all of our data and channels onto the CNCF workspace, preserving any past history of conversations that took place in the Buildpacks slack. For more details please see [CNCF - Migrating workspaces](https://slack.com/intl/en-gb/help/articles/217872578-Import-data-from-one-Slack-workspace-to-another) + +# What it is +[what-it-is]: #what-it-is + +This RFC proposes that we move the Buildpacks slack instance from an independent instance to the CNCF instance. + +# How it Works +[how-it-works]: #how-it-works + +The maintainers will have to request the migration of the Buildpacks slack over to CNCF slack and co-ordinate the announcements/user migration. + +The following are the slack channels that will be migrated over and their proposed names after migrations - + +| Current | Proposed | +| ----------------------- | ------------------------------ | +| announcements | buildpacks-announcements | +| arm | buildpacks-arch-arm | +| buildpacks-authors | buildpacks-authors-team | +| buildpacks-distribution | buildpacks-distribution | +| general | buildpacks | +| implementation | buildpacks-implementation-team | +| implementation-ops | buildpacks-implementation-ops | +| learning | buildpacks-learning-team | +| learning-ops | buildpacks-learning-ops | +| maintainers | buildpacks-maintainers | +| mentoring | buildpacks-mentoring | +| ops | buildpacks-ci-ops | +| pack-cli | buildpacks-pack-cli | +| platform | buildpacks-platform-team | +| registry | buildpacks-registry | +| spec | buildpacks-spec | +| spec-ops | buildpacks-spec-ops | +| tekton | buildpacks-platform-tekton | +| windows | buildpacks-arch-windows | + +The following channels will not be migrated - + +- classic (inactive) +- community (inactive) +- delivery (inactive) +- docs (inactive) +- elixir (inactive) +- governance (inactive) +- java (inactive) +- kaniko (inactive) +- kpack (not part of the official project) +- packfile (inactive) +- python (inactive) +- random (inactive) +- rfcs (inactive) +- ruby (inactive) +- stacks (inactive) +- summit-2021 (inactive) +- user-research (inactive) +- wg-chat (inactive) +## Migration Steps +* [ ] once RFC finalizes, announce migration in slack and mailing list with move over date and a 1 hour maintenance window where chat will be down during the migration. +* [ ] before Feb 2, get slack backup of history +* [ ] on date move over + * [ ] setup final announcement in slack and set all channels as read-only. + * [ ] backup history merge with previous backup, give to CNCF for migration + * [ ] merge docs PR pointing to the new slack. + +### Backups +Slack backups are a zip file that have this layout: +``` +/YYYY-MM-DD.json +... +channels.json +integration_logs.json +users.json +``` + +The plan is to take an initial dump on Feb 1 of all the history up to that point. The second backup will contain the history from Jan 31 up to the the cut off time. The Feb 1->cut off dates will be copied into the original zip and the `channels.json`, `integration_logs.json`, and `users.json` will be copied over as well into the root. + +# Drawbacks +[drawbacks]: #drawbacks + +- All of the current users are not moved over to the new slack instance as a part of the migration. Users who are in the Buildpacks slack but not in the CNCF slack will have to create new accounts on the CNCF slack. +- The migration might also be disruptive to the existing Buildpacks community on slack, as such we should carefully evaluate whether we want to move or not. + +# Alternatives +[alternatives]: #alternatives + +- Keep using the Buildpacks slack. +- Use the Kubernetes slack instance, which is the other alternative that CNCF provides. Kubernetes slack has a larger user base but doesn't come with the ability to create private channels. + +# Prior Art +[prior-art]: #prior-art + +TBD + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +How to handle the migration with no downtime? diff --git a/text/0100-buildpack-input-vars.md b/text/0100-buildpack-input-vars.md new file mode 100644 index 000000000..ee1dee3aa --- /dev/null +++ b/text/0100-buildpack-input-vars.md @@ -0,0 +1,149 @@ +# Meta +[meta]: #meta +- Name: Replace positional args to Buildpack executables with env vars +- Start Date: 2021-11-18 +- Author(s): [jkutner](https://github.com/jkutner) +- Status: Implemented +- RFC Pull Request: [rfcs#190](https://github.com/buildpacks/rfcs/pull/190) +- CNB Pull Request: (leave blank) +- CNB Issue: [buildpacks/lifecycle#806](https://github.com/buildpacks/lifecycle/issues/806), [buildpacks/lifecycle#807](https://github.com/buildpacks/lifecycle/issues/807) +- Supersedes: N/A + +# Summary +[summary]: #summary + +This is a proposal to replace the positional arguments of Buildpack executables with named environment variables. + +# Definitions +[definitions]: #definitions + +- *positional arguments* - values passed into an executable command such that they are accessible as `$1`, `$2`, etc + +# Motivation +[motivation]: #motivation + +Postional arguments to Buildpack executables have been the way for the buildpack execution engine to provide inputs for buildpacks since buildpacks v1 was created. However, positional arguments are limiting, and what they represent is not obvious without reading the spec. + +In the spec today they are defined as: + +* `/bin/detect ` +* `/bin/build ` + +Using env vars helps make these inputs more easily accesible from language bindings like [libcnb.bash](https://github.com/jkutner/libcnb.bash). + +# What it is +[what-it-is]: #what-it-is + +We will deprecate the positional arguments to `bin/detect` and `bin/build` with the following environment variables: + +### bin/detect + +* `CNB_PLATFORM_DIR` - replaces the first positional argument to `bin/detect`. Uses the same env var name as the Platform spec. +* `CNB_BUILD_PLAN_PATH` - replaces the second positional argument to `bin/detect`. Uses the same env var name as the Platform spec. + +### bin/build + +* `CNB_LAYERS_DIR` - replaces the first positional argument to `bin/build`. **Note:** Uses the same env var name as the Platform spec, but refers to a different location. +* `CNB_PLATFORM_DIR` - replaces the second positional argument to `bin/build`. Uses the same env var name as the Platform spec. +* `CNB_BP_PLAN_PATH` - replaces the third positional argument to `bin/build`. + +# How it Works +[how-it-works]: #how-it-works + +Provide the environment variables with the same mechanism used to provide `CNB_BUILDPACK_DIR`. For example in [lifecycle's `build.go`](https://github.com/buildpacks/lifecycle/blob/880a801db2d4bfbb39671a66f7aadd96c0231e37/buildpack/build.go): + +```go +cmd.Env = append(cmd.Env, EnvPlatformDir+"="+b.Dir) +``` + +The positional arguments will be deprecated, but no warnings will be emitted if they are consumed. The lifecycle will continue to provide them to buildpack executable indefinitely, with no plan to remove them. + +## Renaming Platform Env Vars + +Because `CNB_LAYERS_DIR` in the new input vars conflicts with a platform env var name, we intend to rename the platform env var as part of a larger effort to refactor the platform spec. That will be convered in a future RFC. + +The conflict of env var names is strictly an experience problem. Because the env vars are used in different contexts, there is little risk that they will be reused by the internals of lifecycle or pack. + +# Drawbacks +[drawbacks]: #drawbacks + +- People have been using positional arguments to buildpacks for literally a decade +- New env var names conflict with platform env vars + +# Alternatives +[alternatives]: #alternatives + +- Do this but don't deprecate the positional arguments (support both) + +# Prior Art +[prior-art]: #prior-art + +- Buildpack v1, v2a, & v2b + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +N/A + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +## Buildpack spec + +### Detection + +Executable: `/bin/detect`, Working Dir: `` + +| Input | Attributes | Description +|---------------------------|------------|---------------------------------------------- +| `$0` | | Absolute path of `/bin/detect` executable +| `$CNB_BUILD_PLAN_PATH` | E | Absolute path to the build plan +| `$CNB_PLATFORM_DIR` | AR | Absolute path to the platform directory +| `$CNB_PLATFORM_DIR/env/` | | User-provided environment variables for build +| `$CNB_PLATFORM_DIR/#` | | Platform-specific extensions + +| Output | Description +|--------------------|---------------------------------------------- +| [exit status] | Pass (0), fail (100), or error (1-99, 101+) +| Standard output | Logs (info) +| Standard error | Logs (warnings, errors) +| `$CNB_BUILD_PLAN_PATH` | Contributions to the the Build Plan (TOML) + +### Build + +Executable: `/bin/build`, Working Dir: `` + +| Input | Attributes | Description +|---------------------------|------------|---------------------------------- +| `$0` | | Absolute path of `/bin/build` executable +| `$CNB_LAYERS_DIR` | EIC | Absolute path to the buildpack layers directory +| `$CNB_BP_PLAN_PATH` | ER | Relevant [Buildpack Plan entries](#buildpack-plan-toml) from detection (TOML) +| `$CNB_PLATFORM_DIR` | AR | Absolute path to the platform directory +| `$CNB_PLATFORM_DIR/env/` | | User-provided environment variables for build +| `$CNB_PLATFORM_DIR/#` | | Platform-specific extensions + +| Output | Description +|------------------------------------------|-------------------------------------- +| [exit status] | Success (0) or failure (1+) +| Standard output | Logs (info) +| Standard error | Logs (warnings, errors) +| `$CNB_LAYERS_DIR/launch.toml` | App metadata (see [launch.toml](#launchtoml-toml)) +| `$CNB_LAYERS_DIR/launch.sbom.` | Launch Software Bill of Materials (see [Software-Bill-of-Materials](#bill-of-materials)) +| `$CNB_LAYERS_DIR/build.toml` | Build metadata (see [build.toml](#buildtoml-toml)) +| `$CNB_LAYERS_DIR/build.sbom.` | Build Software Bill of Materials (see [Software-Bill-of-Materials](#bill-of-materials)) +| `$CNB_LAYERS_DIR/store.toml` | Persistent metadata (see [store.toml](#storetoml-toml)) +| `$CNB_LAYERS_DIR/.toml` | Layer metadata (see [Layer Content Metadata](#layer-content-metadata-toml)) +| `$CNB_LAYERS_DIR/.sbom.` | Layer Software Bill of Materials (see [Software-Bill-of-Materials](#bill-of-materials)) +| `$CNB_LAYERS_DIR//bin/` | Binaries for launch and/or subsequent buildpacks +| `$CNB_LAYERS_DIR//lib/` | Shared libraries for launch and/or subsequent buildpacks +| `$CNB_LAYERS_DIR//profile.d/` | Scripts sourced by Bash before launch +| `$CNB_LAYERS_DIR//profile.d//` | Scripts sourced by Bash before launch for a particular process type +| `$CNB_LAYERS_DIR//exec.d/` | Executables that provide env vars via the [Exec.d Interface](#execd) before launch +| `$CNB_LAYERS_DIR//exec.d//` | Executables that provide env vars for a particular process type via the [Exec.d Interface](#execd) before launch +| `$CNB_LAYERS_DIR//include/` | C/C++ headers for subsequent buildpacks +| `$CNB_LAYERS_DIR//pkgconfig/` | Search path for pkg-config for subsequent buildpacks +| `$CNB_LAYERS_DIR//env/` | Env vars for launch and/or subsequent buildpacks +| `$CNB_LAYERS_DIR//env.launch/` | Env vars for launch (after `env`, before `profile.d`) +| `$CNB_LAYERS_DIR//env.launch//` | Env vars for launch (after `env`, before `profile.d`) for the launched process +| `$CNB_LAYERS_DIR//env.build/` | Env vars for subsequent buildpacks (after `env`) +| `$CNB_LAYERS_DIR//*` | Other content for launch and/or subsequent buildpacks diff --git a/text/0101-system-buildpacks-in-builder.md b/text/0101-system-buildpacks-in-builder.md new file mode 100644 index 000000000..555bc9e57 --- /dev/null +++ b/text/0101-system-buildpacks-in-builder.md @@ -0,0 +1,142 @@ +# Meta +[meta]: #meta +- Name: System Buildpacks in Builder Images +- Start Date: 2021-07-24 +- Author(s): [@jkutner](https://github.com/jkutner) +- RFC Pull Request: [rfcs#179](https://github.com/buildpacks/rfcs/pull/179) +- CNB Pull Request: (leave blank) +- CNB Issue: [buildpacks/rfcs#209](https://github.com/buildpacks/rfcs/issues/209) +- Supersedes: N/A +- Status: Approved + +# Summary +[summary]: #summary + +This is a proposal for a mechanism that would allow a builder to contain a default set of buildpacks that participate in every detection group, regardless of the buildpack order passed by the platform. + +# Definitions +[definitions]: #definitions + +* _system buildpacks_ - a standard buildpack, conforming to the Buildpack API, which participate in all groups + +# Motivation +[motivation]: #motivation + +Forthcoming changes to the lifecycle (such as [removal of shell-specific logic](https://github.com/buildpacks/rfcs/pull/168)) will remove capabilities that users have come to expect. This includes mechanisms like `.profile`, which allows a buildpack user to customize the environment a process type runs in. We seek to replace these lost mechanisms with buildpacks, in an effort to preserve the capability while still removing complexity from the lifecycle. + +# What it is +[what-it-is]: #what-it-is + +We introduce a `[system]` table in the `builder.toml` schema with the following structure: + +``` +[[system.pre.buildpacks]] + id = "" + version = "" + optional = false + +[[system.post.buildpacks]] + id = "" + version = "" + optional = false +``` + +The fields in the `system.pre.buildpacks` table and `system.post.buildpacks` table match the fields in the existing [`order.group` table](https://buildpacks.io/docs/reference/config/builder-config/#order-_list-required_). + +When a builder includes one or more `system.*.buildpacks` entry, the detect phase will prepend and append all `pre` and `post` buildpacks to each detection group in the provided order, respectively. + +**Note:** A non-`optional` system buildpack creates the possibility that a user provided group with all optional buildpacks could pass detection when it otherwise would not. We leave that up to the platform/builder owner. As long as the platform has a mechanism to disable system buildpacks (and `pack` will), then there is an escape valve for this situation. + +# How it Works +[how-it-works]: #how-it-works + +System buildpacks conform to the [buildpack API](https://github.com/buildpacks/spec/blob/main/buildpack.md). + +The `system.*pre*.buildpacks` will be provided to the lifecycle into a new file, `system.toml`. The `[system]` table in `system.toml` will be processed by the lifecycle, and each `pre`/`post` buildpack will run during the detect phase. Those that pass detection will run during the build phase. + +## Detection + +The exit code of detection by system buildpacks MUST influence the selected buildpack group. If a system buildpack is non-optional and fails detection, then detection MUST for that group fail. If a system buildpack is optional and passes detection, then detection MAY pass for that group. + +System buildpacks may require/provide in the build plan following standard buildpack API specification. + +## Build + +System buildpacks that have passed detection will be added to `group.toml` and treated like any other buildpack for the remainder of the build. + +If a system buildpack exits with a status of `100`, the build will fail. + +# Drawbacks +[drawbacks]: #drawbacks + +- If system buildpacks are hidden from the user before the build, their execution may be unexpected. + +# Alternatives +[alternatives]: #alternatives + +## Do Nothing + +End users would have to add buildpacks like the `profile-buildpack` or other buildpacks that implement system/spec behaviors themselves. + +## Use the `[[order]]` table + +Instead of a new `[system]` table, we could put `pre` and `post` in the `[[order]]` table. However, this could imply that there is a interaction/override/etc between these buildpacks and the `pre`/`post` buildpacks in `project.toml`. But there is not. + +## Use the `[lifecycle]` table + +``` +[lifecycle] + version = "" + + [[lifecycle.pre.buildpacks]] + id = "" + version = "" + optional = false + + [[lifecycle.post.buildpacks]] + id = "" + version = "" + optional = false +``` + +# Prior Art +[prior-art]: #prior-art + +- [RFC-0078: Group additions to Builder order](https://github.com/buildpacks/rfcs/blob/main/text/0078-group-additions.md) + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +## `detector` in Platform specifiction + +This proposal introduces a `--system` option on the `detector`. + +``` +/cnb/lifecycle/detector \ + [--system ]\ +``` + +Where: + +* the lifecycle SHALL merge the `pre` group with each group from `` such that the `pre` buildpacks are placed at the beginning of each order group before running detection. +* SHALL merge the `post` group with each group from `` such that the `post` buildpacks are placed at the end of each order group before running detection. + +#### `system.toml` (TOML) + +```toml +[[system.pre.buildpacks]] + id = "" + version = "" + optional = true + +[[system.post.buildpacks]] + id = "" + version = "" + optional = true +``` + + diff --git a/text/0102-history.md b/text/0102-history.md new file mode 100644 index 000000000..cbcb3dfb3 --- /dev/null +++ b/text/0102-history.md @@ -0,0 +1,85 @@ +# Meta +[meta]: #meta +- Name: Buildpack layer metadata +- Start Date: 2021-12-02 +- Author(s): [@samj1912](https://github.com/samj1912) +- Status: Approved +- RFC Pull Request: [rfcs#194](https://github.com/buildpacks/rfcs/pull/194) +- CNB Pull Request: (leave blank) +- CNB Issue: N/A +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +Add layer history metadata to the output image for buildpack and app layers to improve visualization in tools like `DockerHub`, `dive` etc. + +# Definitions +[definitions]: #definitions + +`history`: The history key in the `OCI` config. See [properties](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) + +# Motivation +[motivation]: #motivation + +This will allow Buildpack built images to be better visualized by common container visualization and debugging tools. + +# What it is +[what-it-is]: #what-it-is + +Various container image introspection tools look at `history.created_by` for each layer to visualize the layer. We should populate this key with a value that describes where a layer came from, for buildpacks this can be - + +- `Layer: {{ buildpack.layer.name }}, Created by: {{ buildpack.id }}@{{ buildpack.version }}` + +for app layers, this can be - + +- `Application Slice: {{ slice_number }} Created by: {{ buildpack.id }}@{{ buildpack.version }}` + +Where `slice_number` is just an integer number starting with `1` and the `buildpack.id` is the id of the buildpack that specified the application slice. + +for config layer, this can be - + +- `Buildpacks Launcher Config` + +for SBOM layer, this can be - + +- `Software Bill-of-Materials` + +for launcher this can be - + +- `Buildpacks Application Launcher` + +for the process types layer this can be - + +- `Buildpacks Process Types` + +For the base image, the lifecycle should copy the existing history to the output image. +# How it Works +[how-it-works]: #how-it-works + +The lifecycle adds the above metadata to the output config blob. + +# Drawbacks +[drawbacks]: #drawbacks + +More complexity? + +# Alternatives +[alternatives]: #alternatives + +Allow buildpacks to create image history metadata. + +# Prior Art +[prior-art]: #prior-art + +N/A + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +N/A + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +Noted above. diff --git a/text/0103-source-date-epoch.md b/text/0103-source-date-epoch.md new file mode 100644 index 000000000..6d55c386a --- /dev/null +++ b/text/0103-source-date-epoch.md @@ -0,0 +1,84 @@ +# Meta +[meta]: #meta +- Name: Use SOURCE_DATE_EPOCH to configure image create time +- Start Date: 2022-02-24 +- Author(s): natalieparellano +- Status: Implemented +- RFC Pull Request: [rfcs#204](https://github.com/buildpacks/rfcs/pull/204) +- CNB Pull Request: (leave blank) +- CNB Issue: [buildpacks/lifecycle#809](https://github.com/buildpacks/lifecycle/issues/809) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +To allow for [build reproducibility](https://github.com/buildpacks/spec/blob/main/platform.md#build-reproducibility), images created by Cloud Native Buildpacks have a hard-coded create time of January 1, 1980. We have received requests from the community (see [here](https://github.com/buildpacks/pack/issues/1281) and [here](https://buildpacks.slack.com/archives/C94UJCNV6/p1643842965830459)) to allow for this value to be configurable. This RFC proposes making the value configurable by setting `SOURCE_DATE_EPOCH` in the lifecycle's execution environment during `export`. + +# Definitions +[definitions]: #definitions + +* build reproducibility: identical inputs should produce identical outputs ([blog post](https://medium.com/buildpacks/time-travel-with-pack-e0efd8bf05db)) +* `SOURCE_DATE_EPOCH`: a "standardised environment variable that distributions can set centrally and have build tools consume this in order to produce reproducible output" (see [here](https://reproducible-builds.org/docs/source-date-epoch/)) + +# Motivation +[motivation]: #motivation + +- Why should we do this? + Users have asked for it +- What use cases does it support? + A meaningful image create time, e.g., the ability to determine which images are most recent +- What is the expected outcome? + Images built with `SOURCE_DATE_EPOCH` set will not be reproducible, as the config blob will change + +# What it is +[what-it-is]: #what-it-is + +- Define the target persona: buildpack user, platform operator + * A buildpack user could pass a flag to `pack` to set `SOURCE_DATE_EPOCH` to the current time in the lifecycle's execution environment during `export`. + * A platform operator could choose to set `SOURCE_DATE_EPOCH` whenever `export` is run. + +- If applicable, describe the differences between teaching this to existing users and new users. + We should mention this feature in the lifecycle release notes and docs. + +# How it Works +[how-it-works]: #how-it-works + +This [PR](https://github.com/buildpacks/imgutil/pull/137) to imgutil would read the environment variable at the point of saving the image. An alternative implementation could be to have the `exporter` read the variable and provide it via a new `SetCreatedAt()` method on the `imgutil.Image` interface. The latter might be safer as it would avoid unintended side effects for other consumers of imgutil (e.g., `pack`). + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +Platforms that choose to set SOURCE_DATE_EPOCH to real creation time will not have 100% reproducible builds. + +# Alternatives +[alternatives]: #alternatives + +- What other designs have been considered? + * Doing nothing + * Allowing layer timestamps to also be configurable, but this would effectively prohibit launch layer re-use and make builds slower +- Why is this proposal the best? + * It is easy to implement, easy to use, and solves for the desired use case +- What is the impact of not doing this? + * The lack of meaningful image create times makes it difficult to automate cleanup of images and could be a deal breaker for some users + +# Prior Art +[prior-art]: #prior-art + +* See under "Reading the variable" [here](https://reproducible-builds.org/docs/source-date-epoch/) +* [ko](https://github.com/google/ko#why-are-my-images-all-created-in-1970) +* [jib](https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#why-is-my-image-created-48-years-ago) + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to be resolved before this gets merged? + * Should we require users to be on platform api 0.9 in order to use this feature? (Opinion: no, because this feature is invisible if the environment variable is not set. The motivation for spec'ing this in the platform api is to make it clear for platform operators how to use it.) +- What parts of the design do you expect to be resolved through implementation of the feature? + * Should we do this in imgutil or the lifecycle? (see above) + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +See [spec PR](https://github.com/buildpacks/spec/pull/292). diff --git a/text/0104-dot-profile-utility-buildpack.md b/text/0104-dot-profile-utility-buildpack.md new file mode 100644 index 000000000..be5f6bea5 --- /dev/null +++ b/text/0104-dot-profile-utility-buildpack.md @@ -0,0 +1,165 @@ +# Meta +[meta]: #meta +- Name: `.profile` Utility Buildpack +- Start Date: 2022-01-25 +- Author(s): mboldt +- Status: Implemented +- RFC Pull Request: [rfcs#200](https://github.com/buildpacks/rfcs/pull/200) +- CNB Pull Request: (leave blank) +- CNB Issue: N/A +- Supersedes: N/A + +# Summary +[summary]: #summary + +As part of RFC 93, [`.profile` scripts will cease to be supported by the platform API](https://github.com/buildpacks/rfcs/blob/main/text/0093-remove-shell-processes.md#appprofile). +This RFC proposes developing a [utility buildpack](https://github.com/buildpacks/rfcs/blob/main/text/0097-official-utility-buildpacks.md) to support `.profile` scripts to prevent regressions after RFC 93 is implemented. + +# Definitions +[definitions]: #definitions + +*utility buildpack*: A buildpack officially supported by the Buildpack Authors' Tooling Team per [RFC 97](https://github.com/buildpacks/rfcs/blob/main/text/0097-official-utility-buildpacks.md) that provides a generic capability. + +# Motivation +[motivation]: #motivation + +[RFC 93](https://github.com/buildpacks/rfcs/blob/main/text/0093-remove-shell-processes.md) resolves to remove shell-specific logic from the CNB Specification. +Part of this includes removing support for `.profile` script in a future version of the Platform API. +RFC 93 recommends supporting the `.profile` script functionality in a utility buildpack to avoid regressions. +This proposal is to develop and support the `.profile` utility buildpack, allowing RFC 93 to be implemented without regression. + +# What it is +[what-it-is]: #what-it-is + +The target persona is a platform operator or implementor who wants to update to the latest platform API, while maintaining the `.profile` functionality for application developers. + +We propose developing and supporting a buildpack to provide an identical interface to the existing `.profile` functionality. +It will: + +- Detect a `.profile` file in the app dir +- Wrap the `.profile` file so that it implements the `exec.d` interface +- Add the `exec.d` executable to the `/exec.d` directory so the launcher will apply it + +We propose the ID `buildpacksio/profile` for this buildpack (cf. [RFC 97](https://github.com/buildpacks/rfcs/blob/main/text/0097-official-utility-buildpacks.md#what-it-is)). + +## Example 1 (environment variables) + +Here is an example of a `.profile` script, inspired by paketo-buildpacks/node-engine: + +``` +memory_in_bytes="$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes)" +MEMORY_AVAILABLE="$(( $memory_in_bytes / ( 1024 * 1024 ) ))" +export MEMORY_AVAILABLE +``` + +The wrapper should ensure that the `MEMORY_AVAILABLE` environment variable is set in the environment with the proper value. + +## Example 2 (file system side effects) + +With this `.profile` script: + +``` +echo 'hello' >> "$HOME/hello" +``` + +The wrapper would not need to set any environment variables, but should maintain the side effect of creating the `$HOME/hello` file. + + +# How it Works +[how-it-works]: #how-it-works + +On Linux, the wrapper will: + +1. Ensure `bash` exists on the run image, and error if it does not. +1. Launch a `bash` shell to: + 1. Source the `.profile` script, which may set environment variables and/or have other side effects. + 1. Write the environment variables to the [`exec.d` output TOML](https://github.com/buildpacks/spec/blob/main/buildpack.md#execd-output-toml). + +Likewise on Windows, the wrapper will: + +1. Ensure `cmd.exe` exists on the run image, and error if it does not. +1. Launch a `cmd.exe` shell to: + 1. Source the `.profile.bat` script, which may set environment variables and/or have other side effects. + 1. Write the environment variables to the [`exec.d` output TOML](https://github.com/buildpacks/spec/blob/main/buildpack.md#execd-output-toml). + +Sourcing the `.profile`/`.profile.bat` script will execute any side effects. +Writing the `exec.d` output TOML will ensure that all environment variables will be set. +So, this will solve for both of the simple examples above. + +Per the [Operating System Conventions in the CNB spec](https://github.com/buildpacks/spec#operating-system-conventions), this buildpack will support scripts compatible with bash version 3 or greater on Linux, and cmd.exe on Windows. + +# Migration +[migration]: #migration + +This buildpack is new, so has no inherent migration considerations. + +For the related migration concerns for removing shell functionality in general, see the [Migration Path section of RFC 93](https://github.com/buildpacks/rfcs/blob/main/text/0093-remove-shell-processes.md#migration-path). + +# Drawbacks +[drawbacks]: #drawbacks + +This is a new component to maintain and support. +That being said, this will enable spec and lifecycle simplifications noted in [RFC 93](https://github.com/buildpacks/rfcs/blob/main/text/0093-remove-shell-processes.md) (where this approach was first suggested). +Also, [RFC 97](https://github.com/buildpacks/rfcs/blob/main/text/0097-official-utility-buildpacks.md) resolves to support and maintain utility buildpacks. + +# Alternatives +[alternatives]: #alternatives + +If we do nothing, we introduce a regression in functionality, and force application developers to rework their `.profile` scripts. + + +# Prior Art +[prior-art]: #prior-art + +- Current CNB spec: + - `.profile` scripts are part of the buildpack app interface ([spec](https://github.com/buildpacks/spec/blob/main/buildpack.md#app-interface)) + - `.profile.d/` scripts are part of the buildpack build phase output ([spec](https://github.com/buildpacks/spec/blob/main/buildpack.md#build)). + +- Profile scripts: + - Heroku supports [`.profile` scripts](https://devcenter.heroku.com/articles/dynos#the-profile-file) and [`.profile.d/` scripts](https://devcenter.heroku.com/articles/buildpack-api#profile-d-scripts). + - Cloud Foundry supports [`.profile and .profile.d/ scripts](https://docs.cloudfoundry.org/devguide/deploy-apps/deploy-app.html#profile). + +- Prior related RFCs: + - [Officially Supported Utility Buildpacks](https://github.com/buildpacks/rfcs/blob/main/text/0097-official-utility-buildpacks.md) + - [Remove Shell Processes](https://github.com/buildpacks/rfcs/blob/main/text/0093-remove-shell-processes.md) + - [Exec.d - Shell-Free Profile.d](https://github.com/buildpacks/rfcs/blob/main/text/0057-exec.d-shell-free-profile-d.md) + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- **Are there things that `.profile` scripts do that will not be covered by the exec.d interface?** + For example, defining environment variables and side effects like writing files should be supported. + But, something like defining a bash function will not be supported. + + This is definitely an edge case. + The app profile runs after the buildpack provided profiles and just before the process type is executed so the only thing that could depend on one of the functions in the process. + This will almost certainly never happen with a buildpack-provided process (the buildpack depending on the app to define a bash function would be very weird and brittle). + + If the user provides the process definition (e.g. with a procfile) this is slightly more plausible. + But in this case there is a very easy workaround - source a script containing the functions as part of the command provided in the procfile. + +- **What if the run image does not contain bash (e.g., `FROM scratch`)?** + Should it error, no-op, should this buildpack add a layer with bash? + + Note that the [utility buildpacks RFC](https://github.com/buildpacks/rfcs/blob/main/text/0097-official-utility-buildpacks.md#how-it-works) states: + > The buildpack MUST NOT install any dependencies in the output image + which precludes this buildpack (or any CNB utility buildpack) from adding bash to the run image. + + While one can imagine a buildpack to add bash, we would like to see the community move to more secure run images that don't include bash. + + We will document that this buildpack requires bash to be in the run image, and this buildpack will not add bash. + If bash is not in the run image, the exec.d wrapper will error out. + +- **How should we implement the buildpack?** + Should it be a bash script? + A Go binary (or two)? + What should we use for CI, frameworks, etc.? + + We will leave these implementation details to the Buildpack Authors' Tooling sub-team. + We can finalize these details after the RFC is approved. + We have captured some initial ideas for the [project plumbing](https://github.com/mboldt/utility-buildpack-plumbing) and had some discussion about [implementation details](https://github.com/buildpacks/rfcs/pull/200#discussion_r828063918) to seed those conversations. + +# Spec. Changes +[spec-changes]: #spec-changes + +None. diff --git a/text/0105-dockerfiles.md b/text/0105-dockerfiles.md new file mode 100644 index 000000000..6af6023a7 --- /dev/null +++ b/text/0105-dockerfiles.md @@ -0,0 +1,224 @@ +# Meta +[meta]: #meta +- Name: Support Dockerfiles +- Start Date: 2021-06-30 +- Author(s): sclevine +- RFC Pull Request: [rfcs#173](https://github.com/buildpacks/rfcs/pull/173) +- CNB Pull Request: (leave blank) +- CNB Issue: [buildpacks/lifecycle#890](https://github.com/buildpacks/lifecycle/issues/890), [buildpacks/lifecycle#891](https://github.com/buildpacks/lifecycle/issues/891), [buildpacks/lifecycle#892](https://github.com/buildpacks/lifecycle/issues/892), [buildpacks/lifecycle#893](https://github.com/buildpacks/lifecycle/issues/893), [buildpacks/lifecycle#894](https://github.com/buildpacks/lifecycle/issues/894) +- Supersedes: [RFC0069](https://github.com/buildpacks/rfcs/blob/main/text/0069-stack-buildpacks.md), [RFC#167](https://github.com/buildpacks/rfcs/pull/167) +- Depends on: [RFC#172](https://github.com/buildpacks/rfcs/pull/172) +- Status: Approved + +# Summary +[summary]: #summary + +This RFC introduces functionality for customizing base images, as an alternative to stackpacks ([RFC0069](https://github.com/buildpacks/rfcs/blob/main/text/0069-stack-buildpacks.md)). + +# Motivation +[motivation]: #motivation + +[RFC0069](https://github.com/buildpacks/rfcs/blob/main/text/0069-stack-buildpacks.md) introduces complexity by defining an API that allows buildpacks to modify base images. To avoid this complexity, we could rely on generated Dockerfiles for base image manipulation. This would simplify the original proposal by, e.g., only requiring a copy of the extension on the build-time base image. + +# What it is +[what-it-is]: #what-it-is + +This RFC proposes that we replace stackpacks with dynamically-generated build-time and runtime Dockerfiles that act as pre-build base image extensions. +These extensions would participate in detection and execute before the buildpack build process. + +For a given application, a build that uses extensions could be optimized by creating a more narrowly-scoped builder that does not contain extensions. + +# How it Works +[how-it-works]: #how-it-works + +Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to apply Dockerfiles at the platform's discretion. The order of operations would be something like the following: +* `analyze` +* `detect` - after standard detection detect will also run extensions' bin/generate, output Dockerfiles are written to a volume +* `extend` - applies run.Dockerfiles to run image (could run in parallel with builder image extension) +* `extend` - applies build.Dockerfiles to builder image +* `restore` +* `build` +* `export` + +When Dockerfiles are used to update the run image, care should be taken to ensure registry access prior to the `build` phase, to avoid long builds that fail due to incorrect credentials. + +### Dynamically-applied Dockerfiles + +A builder image may include any number of "extensions" directories in `/cnb/extensions/`. + +Extensions are similar to buildpacks: they have two executables: `./bin/detect` and `./bin/generate`. The interface for these executables is similar to a buildpack's `./bin/detect` and `./bin/build`. +However, instead of a `buildpack.toml` file, extensions have a `extension.toml` file: +```toml +api = "" + +[extension] +id = "" +name = "" +version = "" +homepage = "" +description = "" +keywords = [ "" ] + +[[extension.licenses]] +type = "" +uri = "" +``` + +Extensions may be packaged and examined similar to buildpacks, but with analogous `pack extension` subcommands. + +Other pack CLI commands, such as `pack builder create`, will be extended to include support for extensions. + +Unlike buildpacks, +- Extensions must not be included in a meta-buildpacks +- Extensions must not have `order`/`group` definitions in `extension.toml` + +Extensions participate in the buildpack detection process, with the same UID, GID, and interface for `./bin/detect`. +However, +- `./bin/detect` is optional for extensions, and they are assumed to pass detection when it is not present. Just like with buildpacks, a `./bin/detect` that exits with a 0 exit code passes detection, and fails otherwise. +- If an extension is missing `./bin/detect`, the extension root `./detect` directory is treated as a pre-populated output directory (i.e., extensions can include a static build plan). +- Extensions may only output `provides` entries to the build plan. They must not output `requires`. +- Extensions are not included in `order` definitions (e.g., in `builder.toml`); instead, a separate `order-extensions` table should be used. The `order-extensions` table will be prepended to each group in the provided `order` (as if `order-extensions` were a composite buildpack). +- Extensions are always `optional`. + +Extensions generate Dockerfiles before the regular buildpack build phase. +To generate these Dockerfiles, the lifecycle executes the extension's `./bin/generate` executable with the same UID, GID, and interface as regular buildpacks. +However, +- Extensions `./bin/generate` must not write to the app directory. +- Extensions `` directory is replaced by an `` directory. +- If an extension is missing `./bin/generate`, the extension root `./generate` directory is treated as a pre-populated `` directory. + +After `./bin/generate` executes, the `` directory may contain +- `build.toml`, + - With an `args` table array with `name` and `value` fields that are provided as build args to `build.Dockerfile` +- `run.toml`, + - With an `args` table array with `name` and `value` fields that are provided as build args to `run.Dockerfile` +- Either or both of `build.Dockerfile` and `run.Dockerfile` + +Support for other instruction formats, e.g., LLB JSON files, could be added in the future. + +`build.Dockerfile` and `run.Dockerfile`target the builder image or runtime base image, respectively. + +If no Dockerfiles are present, `./bin/generate` may still consume build plan entries. Unlike buildpacks, extensions must consume all entries in the provided plan (they cannot designate any entries as "unmet"). + +Dockerfiles are applied to their corresponding base images after all extensions are executed and before any regular buildpacks are executed. +Dockerfiles are applied in the order determined during buildpack detection. When multiple Dockerfiles are applied, the intermediate image generated from the application of the current Dockerfile will be provided as the `base_image` ARG to the next Dockerfile. Dockerfiles that target the run image (only) may ignore the provided `base_image` (e.g., `FROM some-other-image`). Dockerfiles that change the runtime base image may still use `COPY --from=${base_image}`. + +All Dockerfiles are provided with `base_image` and `build_id` args. +The `base_image` arg allows the Dockerfile to reference the original base image. +The `build_id` arg allows the Dockerfile to invalidate the cache after a certain layer and should be defaulted to `0`. The executor of the Dockerfile will provide the `build_id` as a UUID (this eliminates the need to track this variable). +When the `$build_id` arg is referenced in a `RUN` instruction, all subsequent layers will be rebuilt on the next build (as the value will change). + +Build args specified in `build.toml` are provided to `build.Dockerfile` (when applied to the build-time base image). +Build args specified in `run.toml` are provided to `run.Dockerfile` (when applied to the runtime base image). + +A runtime base image may indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. In the case of builder-specified Dockerfiles, `io.buildpacks.rebasable=false` is set automatically on the base image before a runtime Dockerfile is applied and must be explicitly set to `true` if desired. If multiple Dockerfiles are applied, all must set `io.buildpacks.rebasable=true` for the final value to be `true`. Rebasing an app without this label set to `true` requires passing a new `--force` flag to `pack rebase`. When the run image is extended and `io.buildpacks.rebasable=true`, the `extend` phase will communicate to the `export` phase the top layer of the run image (prior to extension) so that the exporter can set the appropriate value of `io.buildpacks.lifecycle.metadata` `runImage.topLayer`. + +#### Example: App-specified Dockerfile Extension + +This example extension would allow an app to provide runtime and build-time base image extensions as "run.Dockerfile" and "build.Dockerfile." +The app developer can decide whether the extensions are rebasable. + +##### `/cnb/ext/com.example.appext/bin/generate` +``` +#!/bin/sh +[ -f build.Dockerfile ] && cp build.Dockerfile "$1/" +[ -f run.Dockerfile ] && cp run.Dockerfile "$1/" +``` + +#### Example: RPM Dockerfile Extension (app-based) + +This example extension would allow a builder to install RPMs for each language runtime, based on the app directory. + +Note: The Dockerfiles referenced must disable rebasing, and build times will be slower compared to buildpack-provided runtimes. + +##### `/cnb/ext/com.example.rpmext/bin/generate` +``` +#!/bin/sh +[ -f Gemfile.lock ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-ruby" "$1/build.Dockerfile" +[ -f package.json ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-node" "$1/build.Dockerfile" +``` + +### Dockerfiles for Base Images + +The same Dockerfile format may be used to create new base images or modify existing base images outside of the app build process (e.g., before creating a builder). Any specified labels override existing values. + +The project will provide tooling that can be used to scan the extended run image. For more information, see https://github.com/buildpacks/rfcs/pull/195. + +### Example Dockerfiles + +Dockerfile used to create a runtime base image: + +``` +ARG base_image +FROM ${base_image} +ARG build_id=0 + +LABEL io.buildpacks.image.distro=ubuntu +LABEL io.buildpacks.image.version=18.04 +LABEL io.buildpacks.rebasable=true + +ENV CNB_USER_ID=1234 +ENV CNB_GROUP_ID=1235 + +RUN groupadd cnb --gid ${CNB_GROUP_ID} && \ + useradd --uid ${CNB_USER_ID} --gid ${CNB_GROUP_ID} -m -s /bin/bash cnb + +USER ${CNB_USER_ID}:${CNB_GROUP_ID} +``` + +`run.Dockerfile` that always installs the latest version of curl: +``` +ARG base_image +FROM ${base_image} +ARG build_id=0 + +RUN echo ${build_id} + +RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/* +``` +(note: this Dockerfile disables rebasing, as OS package installation is not rebasable) + +`run.Dockerfile` that installs a special package to /opt: +``` +ARG base_image +FROM ${base_image} +ARG build_id=0 + +LABEL io.buildpacks.rebasable=true + +RUN curl -L https://example.com/mypkg-install | sh # installs to /opt +``` +(note: rebasing is explicitly allowed because only a single directory in /opt is created) + + +# Drawbacks +[drawbacks]: #drawbacks + +- Involves breaking changes. + +# Alternatives +[alternatives]: #alternatives + +- Stackpacks +- Previous versions of this RFC that don't interact with the buildpack API. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- Should we allow base images to provide build plan entries so that extensions aren't required to satisfy buildpacks? Opinion: Not yet, no-op extension can be used for now. + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +This RFC requires extensive changes to all specifications. + +To deliver incremental value and gather feedback as we implement this large feature, a phased implementation is proposed: +* Phase 1: run.Dockerfiles can be used to switch the runtime base image +* Phase 2: build.Dockerfiles can be used to extend the build time base image + * 2a: `pack` applies the build.Dockerfiles + * 2b: the lifecycle applies the build.Dockerfiles using kaniko +* Phase 3: run.Dockerfiles can be used to extend the runtime base image + * 3a: `pack` applies the run.Dockerfiles + * 3b: the lifecycle applies the run.Dockerfiles using kaniko + +https://github.com/buildpacks/spec/pull/307 and https://github.com/buildpacks/spec/pull/308 describe the spec changes needed for phase 1. https://github.com/buildpacks/spec/pull/298 approximately describes the spec changes needed for all phases. diff --git a/text/0106-rfc-process.md b/text/0106-rfc-process.md new file mode 100644 index 000000000..7b36509fa --- /dev/null +++ b/text/0106-rfc-process.md @@ -0,0 +1,237 @@ +# Meta +[meta]: #meta +- Name: RFC Process +- Start Date: 2021-05-10 +- Author(s): [@ekcasey](https://github.com/ekcasey) +- Status: Implemented +- RFC Pull Request: [rfcs#233](https://github.com/buildpacks/rfcs/pull/233) +- CNB Pull Request: (leave blank) +- CNB Issue: N/A +- Supersedes: [RFC 0004](0004-rfc-process.md) + +# Summary +[summary]: #summary + +The "RFC" (request for comments) process provides a consistent and controlled path for new features and other substantive changes to enter the project. + +The RFC process: +* provides a set of checkpoints that ensure changes align with the overall technical and strategic vision for the process. +* ensures the motivation for a change is clear. +* ensures the impact of a change on users is clear and migration path and backwards compatibility are considered. +* aligns stakeholders on any changes to the user interface(s) and/or APIs (e.g. pack user interface, platform API, buildpack API). +* aligns stakeholders on any substantive architectural changes. +* aligns stakeholders on any processes or workflows adopted by the project. +* provides visibility to the community regarding ongoing work and upcoming changes. +* provides a mechanism by which any interested party can provide early feedback on an upcoming change. +* provides a version controlled record of our decisions and the thought-process behind them. +* is open to anyone! We enthusiastically welcome RFCs from authors that have no formal role in project governance (yet ;p). + +The RFC process **is not**: +* a replacement for high-quality user-facing documentation(although high-quality RFCs enable the creation of high-quality documentation). +* a transaction log of changes. Readers should be able to understand the change proposed without undue reference to previous RFCs. +* a feature request process. RFCs require a level of design and implementation detail that goes beyond a feature request. Pure feature requests should instead be initiated as discussions on the community repo, issues on this repo, or issues on component repos. These requests may serve as the motivation for future RFCs. + + +# Definitions +[definitions]: #definitions + +**Technical Oversight Committee**: The governing body responsible for overseeing the project as described in our governance document. + +**Team**: A group responsible for a particular area of the project. + +**Team Lead**: A team maintainer with special responsibilities for representing the concerns of a team to the project, including casting binding votes on project RFCs. + +**RFC**: A document proposing a substantive change to the project as described in this document. + +**Project RFC**: An RFC with cross-cutting implications, requiring collaboration between multiple teams or affecting multiple personas. + +**Team RFC**: An RFC with narrower implication in comparison to a project RFC, with work scoped to a single team and implications for a narrower set of personas. + +**Author**: The author or authors of an RFC are responsible for producing the draft RFC and updating it to incorporate feedback. Changes should not be made to a draft RFC without the author's consent. + +**Steward**: The steward of an RFC is responsible for shepherding an RFC through the process, including working with the author to ensure RFC completeness and quality, and building consensus among stakeholders. + +**Call for Votes**: When an RFC is deemed ready by its steward, the steward initiates the voting process with a call for votes. At this point the RFC is closed to modification. + +**End Date**: When a call for votes is initiated, an end date for voting is set. Any person wishing to vote on an RFC must do so by the end date. + +**Voting Window**: The time period between a call for votes and the voting end date is referred to as the voting window. + +**Lazy Consensus**: Voting on RFCs is done by lazy consensus. Any project member with a binding vote who has not voted by the end date is assumed to assent to the RFC. + +**Binding Vote**: Binding Votes have formal power within the RFC process. A single no vote by a project member with a binding vote is sufficient to reject an RFC. + +**Non-Binding Vote**: Anyone from the project or community may cast a non-binding vote on an RFC to express their position. While these votes do not have formal power they are taken seriously and may affect the votes of members with binding votes. + + +# Motivation +[motivation]: #motivation + +This RFC process is an evolution of our [previous process](0004-rfc-process.md). + +As the project grows our process needs to grow with it. Compared to the previous iteration, this new process should: +1. **Re-balance responsibilities**. We wish to empower team leads to take ownership of the future of the project and transition the TOC into an oversight role. +2. **Favor action**. By moving from a super-majority vote to a lazy-consensus we remove the ability to block change through inaction. It is now the responsibility of those with concerns to raise them in a timely fashion. +3. **Manage our bandwidth**. Sometimes we need to say "not now" to a good idea to manage the number of large changes proceeding simultaneously. This new process should help the project focus on shipping high-quality enhancements by managing the number of in-flight RFCs, and ensuring each approved RFC has an implementer. +4. **Improve Readability**. By allowing amendments to RFCs we make life easier for anyone wishing to understand a change by ensuring they can find the full plan in a single document. + +# What it is +[what-it-is]: #what-it-is + +### What Types of Changes Require an RFC? + +Any "substantive" change to the project should require an RFC. Substantive includes but is not limited to: +* changes to the specification. +* the adoption, creation, or deprecation of a component (e.g. a new platform implementation, a new shared library, a new system buildpack). +* new features (e.g. a new pack command, a new flag on an existing pack command, an addition to the buildpack API) +* any major refactor that affects consumers of our libraries or materially impacts contribution. +* any major re-architecture especially if it has noteworthy implications for security or performance. +* introduction of new processes or changes to our existing processes including the RFC process. + +If there is any doubt, maintainers should prefer opening an "unnecessary" RFC over surprising users with unexpectedly impactful changes. + +#### Project vs Team RFCs + +An RFC should be a project RFC if: +* it impacts the spec. +* it introduces a new component. +* its implementation necessitates coordination across multiple teams. +* it meaningfully impacts multiple personas (e.g. buildpack authors and platform authors). +* the TOC requests that it be a project RFC. + +Given the nature of our project many RFCs will happen at the project level. However, some types of changes are more appropriately scoped to the team level. Some examples include: +* Platform example: additions to or modification of the pack CLI interface (e.g. [pack pull policy](0046-pack-pull-policy.md)) or library interface (e.g. [pack logging refactor](0002-pack-logging-refactor.md)). +* BAT example: a new major version of the libcnb API. +* Implementation example: [Layer history](0102-history.md) or the lifecycle [multicall binary](0024-lifecycle-multicall-binary-for-build.md). +* Distribution example: [Update CNB Registry JSON Schema](https://github.com/buildpacks/rfcs/pull/45). +* Learning example: [intro video](0090-intro-video-script.md). + +### Process + +#### Requesting an RFC +If you believe an RFC should be written but are either not prepared to author it personally or not prepared to do it _now_, please open an issue on this repo with a high level description of the desired change. This helps us keep track of good ideas and match make ambitious contributors with appropriately sized challenges. + +#### Drafting an RFC +All RFCs begin life as a pull request. Anyone wishing to propose a change to the project should create an RFC pull request by: + +- Fork the RFC repo: +- Copy `0000-template.md` to `text/0000-my-feature.md` (where 'my-feature' is descriptive. don't assign an RFC number yet). +- Fill in RFC. Any section can be marked as "N/A" if not applicable. +- Submit a pull request. + +#### Finding a Steward +All RFCs, even project RFCs "belong" to a team. For project RFCs, The team lead of the responsible team is the steward of the RFC. For team RFCs any maintainer may be the steward. The author and the steward of an RFC may, at times, be the same person. + +In many but not all cases the appropriate team to own an RFC will be obvious. When the appropriate team is not obvious, the author should work with the community to find a home within one of the teams. Factors to consider when finding home include: +* Which team has the most relevant technical context? +* Which team has the deepest understanding of the use-case and the needs of impacted personas? +* Which team is responsible for the components most impacted by the proposed change? +* Which team is enthusiastic about supporting the change? + +If a home cannot be found for a draft RFC it remains in draft until one can be found. This does not necessarily mean that the RFC is not a good idea or isn't something the project will take up eventually. It may simply be that the project does not have the bandwidth to prioritize this particular change at this particular time. + +#### Stewardship + +The steward and their team should: + * work with the author of the RFC to ensure that the RFC is complete and implementable contingent upon approval. This can happen synchronously at team working groups or asynchronously via github and slack. + * raise visibility to and solicit feedback from other stakeholders including the TOC, other teams, and the community at large. This can happen synchronously at the project working group and asynchronously via github and slack. + * drive consensus for the RFC by incorporating feedback from stakeholders so that the RFC has the best possible chance of approval during the voting process. + * ensure there is a plan in place to implement the RFC in a reasonable time frame, contingent upon approval. The team itself need not implement the RFC but we should not approve RFCs for which we have no concrete plan to implement. + * in the case of complicated or risky RFCs, a POC should be developed at this stage to validate and de-risk the proposed design. + +#### Voting + +Ideally the voting should be a formality and not a moment to discover new disagreement, consensus already haven been driven by the steward. + +When the steward deems the RFC ready and likely to be accepted they should formally call for votes and set an end date for voting. Stewards should make a good faith effort to ensure that all interested parties, and in particular those with binding votes, have adequate opportunity to review the finalized RFC and cast votes. A minimum time period of a week (unless all of the binding votes are cast) is a good guideline. + +The RFC may not be edited during the voting window. + +For project RFCs, all TOC members and all team leads are given a binding vote. + +Additionally, for team RFCs, all team maintainers are given a binding vote. + +Votes are cast via reviews on the RFC PR. Accepting the PR signifies a yes vote while a request for changes signifies a no vote. If all members with a binding vote vote in the affirmative, the voting window may close early. In order for a RFC to be approved, at least 1 yes vote. In the rare case there are no votes, the Calling for Votes process may be followed again or the RFC may be closed. + +##### Acceptance + +If the end date of the vote arrives without a no vote from a member with binding vote, the RFC is accepted. It will be merged into the repo and assigned a number. The steward should create a tracking issue to coordinate implementation and link the tracking issue in the RFC header. When this is complete, implementation can begin. + +##### Rejection + +If a single no vote from a member with binding vote is cast before the arrival of the end date the RFC is immediately rejected and the PR should be closed. + +The same RFC may be re-opened and brought to a vote again in the future assuming the concerns that lead to the no vote are addressed. + +##### Implementation + +After the RFC has been accepted, any team or individual may begin implementing the changes it defines. + +##### Amending an RFC + +While we should strive to get the details right the first time, by doing our due diligence including a proof of concept implementation for larger/riskier RFCs, there will be times when we discover during the process of implementation that something about the original plan was incomplete or needs adjustment. Small changes may be PR'ed by the implementer and merged upon approval by the steward. It is the responsibility of the Steward to determine what changes are minor enough to qualify as an amendment and which are fundamental and/or controversial enough to require a new superseding RFC. + +PRs that amend an RFC should add an amendment to the [History](../0000-template.md#history) section of the RFC document with: +1. Amendment Metadata +2. A summary of the changes +3. The motivation for the changes + + +### Amending the RFC Process + +The RFC process should be amended through the RFC process. However, the TOC reserves the right to change the process via a super-majority vote in the unlikely even that the process proves so irreparably flawed as to preclude its amendment via the process. + +# How it Works +[how-it-works]: #how-it-works + +### Labels + +| name | purpose +|---------------------------|-------- +| `type/idea` | An Issues requesting an RFC +| `type/rfc` | A PR containing a new RFC +| `type/tracking` | An issue tracking the implementation of an RFC +| `type/ammendment` | A PR containing an ammendment to an existing RFC +| `scope/project` | Applied to project-level RFCs +| `scope/team` | Applied to team-level RFCs. +| `team/` | E.g `team/platform`. Designates the team that "owns" an RFC. Applies to both team-level and project-level RFCs. For project-level RFCs the lead of this team is the steward. +| `status/needs-steward` | Applied to all new RFCs. Removed once the RFC is accepted by a steward or closed. +| `status/ready-for-review` | The Steward applies this to an RFC when they deem it ready for review by a broader audience. +| `status/voting` | The Steward applies this label when the voting window opens +| `status/steward-assigned` | The Steward applies this label when adopting the RFC along with self-assigning the PR +| `status/accepted` | The Steward applies this label to accepted RFCs. +| `status/rejected` | The Steward applies this label to rejected RFCs. + +The `Final Comment Period` label will no longer be used. + +# Migration +[migration]: #migration + +This RFC should be accepted via the existing RFC process. The new process will take effect immediately after its ratification, superseding in its entirety the previous process. + +# Drawbacks +[drawbacks]: #drawbacks + +* Lazy consensus favors velocity and moving things forward, over rigor and thoroughness. This could result in a higher volume of changes that have some unforeseen consequences. This is a tradeoff we are consciously making. + +# Alternatives +[alternatives]: #alternatives + +None + +# Prior Art +[prior-art]: #prior-art + + +* [Tekton Enhancement Proposal Process (TEP)](https://github.com/tektoncd/community/blob/main/teps/0001-tekton-enhancement-proposal-process.md) +* [Kubernetes Enhancement Proposals (KEP)](https://github.com/kubernetes/enhancements/blob/master/keps/README.md) + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +None + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +N/A diff --git a/text/0107-pack-buildpack-new-alternatives.md b/text/0107-pack-buildpack-new-alternatives.md new file mode 100644 index 000000000..266e43f17 --- /dev/null +++ b/text/0107-pack-buildpack-new-alternatives.md @@ -0,0 +1,301 @@ +# Meta +[meta]: #meta +- Name: Pack Buildpack New to Support Alternatives +- Start Date: 2022-03-10 +- Author(s): aidandelaney +- Status: Approved +- RFC Pull Request: [rfcs#212](https://github.com/buildpacks/rfcs/pull/212) +- CNB Pull Request: (leave blank) +- CNB Issue: N/A +- Supersedes: https://github.com/buildpacks/rfcs/blob/main/text/0077-pack-buildpack-create.md + +# Summary +The `pack buildpack new` subcommand creates a new buildpack based on a shell-script template. This proposal replaces `pack buildpack new` with `pack buildpack create` to allow alternatives to the shell-script template. Invoking `pack buildpack create` creates a new buildpack using the existing `bash` scaffolding. The `--template URL` option allows buildpack creation from an arbitrary 3rd party URL. + +We will implement the project scaffolding logic in a Go module separate from `pack`. This decouples `pack` from the project scaffolding implementation. + +# Definitions +[definitions]: #definitions + +* **project scaffolding**: minimal code and file system structure used to start a project. + +# Motivation +[motivation]: #motivation + +The creation of buildpacks in languages other than `bash` is undocumented in the [Buildpack Author Guide](https://buildpacks.io/docs/buildpack-author-guide). In particular, creating a buildpack in Go using `libcnb` is undocumented because there is no mechanism to generate a scaffolded project. This proposal adds a mechanism to `pack` that would allow the buildpacks project to provide `libcnb` buildpack project scaffolding. + +`pack buildpack create` supports end-users who wish to scaffold a new buildpack. The operation of `pack buildpack create` requires Internet access to succeed. `pack buildpack create --template URL` allows projects (such as Paketo, Heroku and others) to define skeleton projects for new buildpacks. + +Replacing `pack buildpack new` with `pack buildpack create` allows buildpack authors to more-easily create new buildpacks in their chosen language and framework. + +# What it is +[what-it-is]: #what-it-is + +Project scaffolding is [very](https://cookiecutter.readthedocs.io/) [popular](https://yeoman.io/) [in](https://github.com/kowainik/summoner) [other](https://crates.io/crates/cargo-scaffold) [ecosystems](https://github.com/golang-standards/project-layout). Scaffolding systems are employed to ease onboarding of new developers. Within `pack` this feature is targeted at onboarding buildpack authors. + +`pack buildpack create` creates the same buildpack scaffolding as `pack buildpack new`. However, `pack buildpack create --template URL` can be used to access alternative project scaffolding. + +# How it Works +[how-it-works]: #how-it-works + +The design is modeled on [cookiecutter](https://cookiecutter.readthedocs.io/en/1.7.2/) with reference to [springerle](https://github.com/carlmjohnson/springerle) -- a similar implementation in golang -- and [create-go-app](https://github.com/create-go-app/). We do not want to `os.Exec` a python subprocess to run cookiecutter as this would require availability of a python runtime environment. Instead we propose to borrow heavily from create-go-app, and generate the default shell scaffolding from a cloned git repoistory. + +The `pack buildpack new` project scaffolding accepts `--api`, `--path` and `--stacks` command line flags and the buildpack id as a positional argument. These command line flags and positional arguments will be replaced with user prompts `pack buildpack create`. + +A full session includes the terminal prompts that the project scaffolding tool asks of the end user: + +```command +$ pack buildpack create +✔ Enter an ID for this buildpack: example/bash +✔ Enter a directory in which to scaffold the project: bash_buildpack +Use the arrow keys to navigate: ↓ ↑ → ← +? Choose the buildpack API version (use the default if you are unsure): + ▸ 0.7 + 0.8 +✔ Enter a stack this buildpack will support by default: io.buildpacks.samples.stacks.bionic + +Created project in bash_buildpack +``` + +The user can skip prompts by providing `--arg key=value` as command line flags. Each of the prompts from the previous interactive invocation can be skipped by providing appropriate pairs of key and value. + +```command +$ pack buildpack create --arg ProjectDirectory=bash_buildpack --arg BuildpackApi=0.8 --arg BuildpackID=example/bash --arg BuildpakStacks=io.buildpacks.samples.stacks.bionic + +Created project in bash_buildpack +``` + +Templates are provided as a file tree. A root-level `prompts.toml` file contains prompts for the end-user. For example the current `bash` scaffolding could be structured as follows, a full specification of `prompts.toml` is provided in the following subsection. + +```bash +. +├── prompts.toml +└── {{.ProjectDirectory}} + └── bin + ├── build + └── detect +``` + +Where `prompts.toml` is of the form: + +```toml +[[prompt]] +name="ProjectDirectory" +prompt="Enter a directory in which to scaffold the project" +default="bash_buildpack" +required=true + +[[prompt]] +name="BuildpackApi" +prompt="Choose the buildpack API version (use the default if you are unsure)" +choices=["0.7", "0.8"] + +... +``` + +A specification of `prompts.toml` is provided below. + +Multiple templates can be managed in a single repository, referred to as a **template colleciton**. + +## Template Collections + +We expect project templates to be managed in `git` repositories. Projects may wish to manage multiple templates in a single repository. We refer to this as a template collection. More formally, a template collection is a `git` repository where top-level directories are templates. A template collection does not contain a top-level `prompts.toml`, but top-level subdirectories are expected to contain `prompts.toml` files. As an example, a template collection might have the following structure where the `bash` sub-directory is a template and the `Go` sub-directory is a template. + +```bash +. +├── bash +└── Go +``` + +A user may use a template collection as an argument to `--template`. A template collection passed as `--template` first prompts the end user to choose between the available templates in the repository. Once a template is chosen, the user is prompted using the `prompts.toml` from the chosen template. + +```command +$ pack buildpack create --template https://github.com/AidanDelaney/cnb-buildpack-templates +Use the arrow keys to navigate: ↓ ↑ → ← +? choose a project template: + ▸ Go + bash +Enter a directory in which to scaffold the project: go_buildpack +? Choose the buildpack API version (use the default if you are unsure): + 0.7 + ▸ 0.8 +✔ 0.8 +Enter an identifier for the buildpack: example/golang +Enter a stack this buildpack will support by default: io.buildpacks.samples.stacks.bionic +✔ Enter a valid Go module name for this buildpack: github.com/user/buildpack +Created project in go_buildpack +``` + +A user may specify the choice of template from a template collection via a `--sub-path` flag. A user may choose a `bash` template from a template collection and provide the api, output directory and stacks as command line flags: + +```command +$ pack buildpack create --template https://github.com/AidanDelaney/cnb-buildpack-templates \ + --sub-path Go \ + --arg BuildpackID=example/golang \ + --arg BuildpackAPI=0.8 \ + --arg ProjectDirectory=go_buildpack \ + --arg BuildpackStacks=io.buildpacks.samples.stacks.bionic +Created project in go_buildpack +``` + +Having provided examples of the use of `pack buildpack create`. We now specify the format of `prompts.toml`. + +## prompts.toml + +The prompts are contained in a TOML file. The `prompts.toml` file contains an array of tables. Each table specifies a single `prompt`. Each prompt defines a variable identified by a provided `name`. + +| field | required | description | type | +|---------------|----------|------------------------------------------------------------------------------------------------------------------------------------------|----------------| +| name | Yes | The variable identifier. Must be unique within the `prompts.toml` file. | string | +| prompt | Yes | The default prompt question to ask the user | string | +| required | No | Evaluates to `false` if not provided. Specifies whether the user **must** provide a value for the variable identified by `name`. | boolean | +| default | No | A default value for the variable identified by `name`. Mutually exclusive to `choices` | string | +| choices | No | A list of default values from which the user may choose a value for the variable identified by `name`. Mutually exclusive to `default`. | list of stings | + +For example, the following is a valid `prompts.toml` file: + +```toml +[[prompt]] +name="ProjectDirectory" +prompt="Enter a directory in which to scaffold the project" +``` + +The following `prompts.toml` is invalid as it does not contain the required `name` field: + +```toml +[[prompt]] +prompt="Enter a directory in which to scaffold the project" +``` + +The following `prompts.toml` is invalid as value of the `name` field is not unique within the document: + +```toml +[[prompt]] +name="ProjectDirectory" +prompt="Enter a directory in which to scaffold the project" + +[[prompt]] +name="ProjectDirectory" +prompt="Enter a directory" +``` + +The following `prompts.toml` is invalid as the `default` and `choices` fields are mutually exclusive: + +```toml +[[prompt]] +name="ProjectDirectory" +prompt="Enter a directory in which to scaffold the project" +default="/tmp" +choices=["/tmp", "/home"] +``` + +## Arguments + +Arguments may be provided as command line arguments. For example `pack buildpack create --arg ProjectDirectory=/tmp` provides a value, `/tmp`, for the variable identified by the name `ProjectDirectory`. + +When an argument, `key=value`, is provided then `pack buildpack create` must not prompt the user with the prompt identified by `name=key`. + +Where an `--arg key=value` is provided and no prompt identified by `name=key` exists in `prompts.toml`, then the provided argument is ignored. + +Choices constrain values for a given variable. Where a provided `value` does not match one of the `choices` then an error is returned. That is to say, given + +``` +[[prompt]] +name="BuildpackApi" +prompt="Choose the buildpack API version (use the default if you are unsure)" +choices=["0.7", "0.8"] +``` + +and given a CLI invocation `pack buildpack create --arg BuildpackApi=0.6` the choices are restricted to `["0.7", "0.8"]`. Therefore, `0.6` is an invalid argument and an error is returned to the end-user. + +Arguments for buildpack creation need to be discoverable. We intend to support this via the `--help` flag to `pack buildpack create`. Executing + +``` +pack buildpack create --help + +pack buildpack create +arguments offered by template + ProjectName (default: pyexample) + BuildpackApi 0.7, 0.8 (default 0.7) +``` + +shows a default help message. Invoking `--help` with a template url queries the prompts provided by the template: + +``` +pack buildpack create --help --template= +arguments offered by template + ProjectName (default: pyexample) + PythonVersion=python3.10, python3.9, python3.8 (default: python3.10) + NumDigits (default: 3) +``` +The possible values of a template collection can similarly be queried: + +``` +pack buildpack create --help --template= +templates available in collection + Go + bash +``` + +## `pack buildpack create` Substitutions + +The operation of variable substitution follows the operation of Go [`text/template`](https://pkg.go.dev/text/template). Prompts defined in `prompts.toml` are interpreted, the user may be prompted and set of variables is generated. The identifiers of variables are replaced with the value of the variable. + +Variables may be used in files, for example where a `prompts.toml` defines a variable with identifier `ProjectDirectory` then the expression `{{.ProjectDirectory}}` in files is replaced with the value of the variable identified by `ProjectDirectory`. In addition, the expression `{{.ProjectDirectory}}` may be used in template file paths. The generated project substitutes file paths with variable expressions with the value of the variable. + +When a source file contains a Go `text/template` style expression and the variable name does not appear in `prompts.toml`, then the `text/template` style expression is not replaced. For example, if a `README.md` file contains the expression `{{.Example}}` and `Example` is not a variable defined in `prompts.toml`, then the string `{{.Example}}` is not replaced in `README.md`. + +# Migration +[migration]: #migration + +The current `bash` project scaffolding can be re-used in the default project scaffolding. + +We intend to maintain `pack buildpack new` in parallel with `pack buildpack create` for two `pack` releases. After two `pack` releases, `pack buildpack new` will be replaced with an error instructing the user to run `pack buildpack create`. We will remove `pack buildpack new` after three `pack` releases. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +* Project structure is straightforward and it could be better to document a `libcnb` project structure. However, given that we already support `pack builpack new`, we feel it important that `pack` also creates generation of `libcnb`-style projects. +* Project scaffolding could be delegated to a 3rd party tool. The current `pack buildpack new` functionality could be extracted from `pack` and, instead, we could suggest another tool for end users to use for project scaffolding eg: `bare use buildpacks/bash my_buildpack`. +* Including generalized project scaffolding in `pack` will increase size of `pack` binary. The size of `pack` will increase by the size of an appropriate prompting package (such as [survey](https://pkg.go.dev/github.com/AlecAivazis/survey/v2)) and an appropriate package allowing `git clone` (such as [go-git](https://github.com/go-git/go-git) at ). +* This proposal commits to support a specific project scaffolding format. A migration path should be established if and when a de-facto standard golang template library becomes available. + +# Alternatives +[alternatives]: #alternatives + +- What other designs have been considered? + +We have also considered [springerle](https://github.com/carlmjohnson/springerle) which uses a single txtar file to describe skeleton project structure. The use of a single txtar file requires template providers to write and maintain this format. + +[bare](https://github.com/bare-cli/bare) is a new project. It assumes that all templates are stored on github.com. + +- Why is this proposal the best? +To integrate with `pack` we want a pure-golang project scaffolding tool. This proposal advocates an approach that integrates future cnb-provided project scaffolding with `pack` and allows `pack` to clone 3rd party project scaffolding from a location chosen by the end-user. + +- What is the impact of not doing this? + +Omitting support for generalized project scaffolding requires new buildpack authors to consult our documentation about project structure. Moreover, as `pack` supports scaffolding of shell-script buildpacks the impression is given that the buildpacks project _prefers_ shell implementations. + +# Prior Art +[prior-art]: #prior-art + +There are many, many competing implementations of project scaffolding tools. + +* Python's [Cookiecutter](https://cookiecutter.readthedocs.io/): widely used to scaffold projects in many languages, including golang. Requires a Python runtime to be available. +* [springerle](https://github.com/carlmjohnson/springerle): golang re-think of cookiecutter. Springerle uses a single txtar file augmented with a header containing user prompts. This proposal prefers using a filesystem rather than a single txtar file as the filesystem approach extends to cloning repositories. +* [cgapp](https://github.com/create-go-app/cli): This proposal heavily borrows from cgapp. +* JavaScript's [Yeoman](https://yeoman.io/): widely used in the web ecosystem +* [boilr](https://github.com/tmrts/boilr/): golang cookiecutter. Moribund project. +* [bare](https://github.com/bare-cli/bare): golang cookiecutter. Possible successor to boilr. Both boilr and bare assume the templates are stored on github.com and use the zip downloading functionality of that specific service. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions +* What team maintains template repositories? + * Buildpack Author Tooling team + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +This proposal does not require any spec changes. diff --git a/text/0108-governance-component-maintainer-role.md b/text/0108-governance-component-maintainer-role.md new file mode 100644 index 000000000..e5d50fdf7 --- /dev/null +++ b/text/0108-governance-component-maintainer-role.md @@ -0,0 +1,223 @@ +# Meta +[meta]: #meta +- Name: add component maintainer role +- Start Date: 2022-09-39 +- Author(s): @jjbustamante +- Status: Approved +- RFC Pull Request: [rfcs#234](https://github.com/buildpacks/rfcs/pull/234) +- CNB Pull Request: (leave blank) +- CNB Issue: [buildpacks/community#196](https://github.com/buildpacks/community/issues/196) +- Supersedes: [#228](https://github.com/buildpacks/rfcs/pull/228) + +# Summary +[summary]: #summary + +This RFC proposes the creation of a new role **component maintainer** in the project governance, responsible of executing maintainer duties on the repositories in a specific component under a CNB team. + +# Definitions +[definitions]: #definitions + +- **maintainers**: individual that maintain specific code repositories. +- **contributor**: individual who make regular contributions to a team (documentation, code reviews, responding to issues, participation in proposal discussions, contributing code, etc.) +- **teams**: Group of individuals focused on narrower sets of concerns related to specific aspects of the project. Example: implementation team, platform team. +- **team lead**: is a maintainer who has special responsibilities for representing team concerns at the project level +- **component maintainer**: the proposed role in this RFC. Individual that maintain specific code repositories of a software-component inside a CNB team. +- **software component**: is a software unit with a well-defined interface and explicitly specified dependencies. +- **github team**: groups of organization members that reflects a company or group's structure with cascading access permissions. + +# Motivation +[motivation]: #motivation + +### Why should we do this? + +Our current governance process defines [teams](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md#teams) and each team is responsible for maintaining some number of software components, these teams are organized internally with 3 different roles: + - Team Leads + - Maintainers + - Contributors + +The process does not take into consideration the sizes of the software components that a maintainer must take accountability for or the expectations of the community contributors who made want to specialize in certain pieces of the projects, as the number and size of all the software components in a team increases we may need to distribute the responsibilities in a different way. + +Compared to the model based on 3 roles, this new role should: +- **re-balance responsibilities inside a team** empower contributors to take ownership of key components align with their technical skills or career path expectations +- **support maintainers on their role's goals** one responsibility of a maintainer is `growing the team by mentoring aspiring contributors and maintainers`, this new role offers a growing path for some contributors what would like to become maintainers. + +### What use cases does it support? + +It provides a mechanism to handle the increases of complexity of the source code of each component maintain by a team, when a team lead or maintainer determines a software component is big enough to be delegated to a contributor that desires to specialize and has the know-how to be responsible of a component they can nominate him/her to become **component maintainer** and delegates some of the day to day activities to this person. + +### What is the expected outcome? + +A new role and guideline on how to nominate individuals to this role will be include it in our governance process. + +# What it is +[what-it-is]: #what-it-is + +CNB maintainer contributor responsibility can be describe as: `is in charge of the day to day maintenance of the team's projects` + +![](https://i.imgur.com/yXsLK6N.png) + +As we can see in diagram above, a CNB team takes care of `N` number of [software components](#definitions) and the project contributors from the community make their contributions to any of those components. + +Depending on the team, these components can increase in size or complexity, or there could be someone from the community that wants to specialize their contributions on certain components without taking the responsibility of become a **team maintainer**. + +### Examples + +#### Platform Team + +Let's take for example the [Platform Team](https://github.com/buildpacks/community/blob/main/TEAMS.md#platform-team), which right now have 2 maintainers, and let's use the LOC (lines of code) metric, for each of the components maintains by this team, to dimension the size of them. + +| Component | LOC | +|--------------------------|--------| +| [Pack]() | +58000 | +| [Tekton Tasks + Pipelines]() | +2150 | +| [CircleCI Pack Orb]() | +400 | + +**Note**: [Tokei](https://github.com/XAMPPRocky/tokei) tool was used to calculate the LOC of the repositories. + +##### Integration with the Cloud Native Ecosystem + +As part of the [CNB roadmap](https://github.com/buildpacks/community/blob/main/ROADMAP.md#integration-with-the-cloud-native-ecosystem) a `better out-of-the box Kubernetes and Docker integration` is a goal of the project and in order to do that, the [Platform Team](https://github.com/buildpacks/community/blob/main/TEAMS.md#platform-team) will have to include another software component that accomplish this goal. + +In case of this scenario, then an hypothetical update of the component table if kpack project is donated into CNB (see [RFC](https://github.com/buildpacks/rfcs/pull/235) for more details) will be: + + +| Component | LOC | +|--------------------------|--------| +| [Pack]() | +58000 | +| [Tekton Tasks + Pipelines]() | +2150 | +| [CircleCI Pack Orb]() | +400 | +| [Kpack]() | +57000 | + +As we can see, a new implementation of the [Platform Interface Specification](https://github.com/buildpacks/spec/blob/main/platform.md) could be as big as [pack]() but most important: +- It requires a specific knowledge in [Kubernetes](https://kubernetes.io/) + +##### Adding Cosign integration + +Another example of the problem presented in this RFC is: [adding support to cosign RFC](https://github.com/buildpacks/rfcs/pull/195). In this RFC a new phase executable must be developed and maintain by the [Platform team](https://github.com/buildpacks/community/blob/main/TEAMS.md#platform-team), but this implementation requires knowledge and expertise on technologies like [sigstore](https://www.sigstore.dev/). + +# How it Works +[how-it-works]: #how-it-works + +The proposal is to incorporate a **component maintainer** role in our governance process. + +The **component maintainer** will take under his/her responsibility a well defined software component in a CNB team and for each repository will be allow to: + +- reviewing, approving, and merging pull requests. +- planning release milestones, and releasing components versions +- edit, label, assign issues + +An updated version of the previous diagram shows graphically this new role. + +![](https://i.imgur.com/rWElkCw.png) + +As we can see a new orange box was drawn representing a component, or group of software components, the new **component maintainer** role is taking responsibilities for. When a team lead or maintainer wants to use the role in their team, they must follow these steps: + +- In case it doesn't exist, a new **github team** must be created. A recommended name for this new team must follows the format `[cnb-team]-[component]-maintainers`, where: + - **cnb-team**: is the CNB team responsible for the software component, for example: `Platform Team`. + - **component**: is the software component name. for example: `kpack`. + + Some examples are: `platform-kpack-maintainers` or `platform-cosign-maintainers` + +- For each repository related to the component or group of components: + - In case it doesn't exist, a new **CODEOWNERS** file must be added. + - Add the team `[cnb-team]-[component]-maintainers` created into the **CODEOWNERS** file. + - Members of the team should have maintainers permissions. + - The branch protection should require **CODEOWNERS** to approve or merge a pull request. + +- Add the new component maintainer's github account into the `[cnb-team]-[component]-maintainers` **github team**. + +### Examples + +Let's come back to our previous examples. + +#### Platform Team + +In these cases, the existence of the **component maintainer** role will provide the rules to the platform team lead or platform maintainers to nominate (following the guidelines describe in the next section) a **component maintainer**. + +##### Integration with the Cloud Native Ecosystem + +In case [kpack](https://github.com/pivotal/kpack) is donated to CNB, [kpack](https://github.com/pivotal/kpack) maintainers could become **component maintainers** of this component and keep doing all the activities required for maintaining the lights on in the project without having to assume the whole set of responsibilities of a Platform team maintainer. Also, platform maintainers will not be overwhelm being the sole reviewers/approvers for [kpack's](https://github.com/pivotal/kpack) pull requests if they are not familiar with the project. + +##### Adding Cosign integration + +The existence of the **component maintainer** will open the door to the community, in particular, those volunteers with experience on [sigstore](https://www.sigstore.dev/) to help on with the contributions and maintenance of the new `signer` binary proposed. + +This case, is an example of two different areas of interest or knowledge where having the separation of responsibilities is relevant. + +## Guidelines nominate component maintainer + +Follow this guideline to nominate a **component maintainer** for a software component inside a team + +- The software component must be defined under the [GOVERNANCE](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md) team section, for example: Platform Team -> kpack +- New **component maintainers** must already be contributors of the team +- New **component maintainers** must be nominate by a **team lead** or a **maintainer** of the team under the following scenarios: + - A software component developed outside CNB project was [accepted](https://github.com/buildpacks/community/blob/main/contributors/guide.md#component) under their team and they do not have the know-how or experience to handle it. + - A community **contributor** have explicitly manifest the desire to become a **component maintainer** and the **team lead** or **maintainer** consider he/she has the skills and knowledge to take the responsibility and accountability of the component. +- New **component maintainers** must be elected by [super-majority](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md#supermajority) of the team’s maintainers + +# Migration +[migration]: #migration + +None + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? +- Yet another role +- We decide to wait until the problem arises + +# Alternatives +[alternatives]: #alternatives + +### Do Nothing + +If no new role is created, `maintainer` will continue to be accountable and responsible of all the software components inside a team. + +### Repository level ownership RFC + +The first attempt to addressed the problem was proposing the **repository level ownership** [RFC](https://github.com/buildpacks/rfcs/pull/228) but abandoned this idea because: +- A new role is more consistent with current governance structure +- A new role will provide more visibility to the community to identify individuals responsible for each repository. + +# Prior Art +[prior-art]: #prior-art + +Discuss prior art, both the good and bad. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + + + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +None + +# History +[history]: #history + +None + + diff --git a/text/0109-build-config.md b/text/0109-build-config.md new file mode 100644 index 000000000..fde9e5e8f --- /dev/null +++ b/text/0109-build-config.md @@ -0,0 +1,122 @@ +# Meta +[meta]: #meta +- Name: Build config +- Start Date: 2022-08-29 +- Author(s): [samj1912](https://github.com/samj1912) +- Status: Approved +- RFC Pull Request: [rfcs#230](https://github.com/buildpacks/rfcs/pull/230) +- CNB Pull Request: (leave blank) +- CNB Issue: [buildpacks/lifecycle#956](https://github.com/buildpacks/lifecycle/issues/956), [buildpacks/spec#330](https://github.com/buildpacks/lifecycle/issues/330), [buildpacks/docs#531](https://github.com/buildpacks/lifecycle/issues/531) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +This RFC proposes an easy way to configure build images to allow specifying a `/cnb/build-config` CNB environment directory that allows updating the Buildpack `detect` and `build` environment based on the directory. + + +# Definitions +[definitions]: #definitions + +- CNB environment directory: A directory that follows the conventions as defined [here](https://github.com/buildpacks/spec/blob/main/buildpack.md#provided-by-the-buildpacks) +- Operator: Owner of builders and stacks. See [CNB Operator guide](https://buildpacks.io/docs/operator-guide/). + + +# Motivation +[motivation]: #motivation + +Often times, especially in enterprise settings, organizations often have to update the buildpacks to use internal mirrors, proxies and other settings which can be configured easily with environment variables. + +Some examples include - +- `GOPROXY` +- `PIP_INDEX_URL` +- `npm_config_registry` +- `BUNDLE_MIRROR__URL` + +The buildpack logic in the Buildpacks largely remains the same, except these environment variables might need to be injected during the `build` and `detect` phases. + +The environment variables may ideally also take precendence over any user provided values to ensure that the operators have full control over their builders. + +# What it is +[what-it-is]: #what-it-is + +The RFC proposes the introduction of the following directory `/cnb/build-config/env` in build images. The directory follows the same convention as a `CNB environment directory`. The notable difference is that the environment variables sourced from this directory are applied **AFTER** processing the user-provided platform environment variables i.e. they should have the highest precedence. These variables should be available during both `detect` and `build` phases (and the `generate` phase in the future). + + +The operator can define this directory in the build image under `/cnb/build-config` or `CNB_BUILD_CONFIG_DIR` if defined. + +# How it Works +[how-it-works]: #how-it-works + +Reference implementation available at https://github.com/buildpacks/lifecycle/pull/899/files + +Examples - + +Buildpack value: `FOO=test` +Build config: `FOO.default=another-value` +Final value: `FOO=test` + + +Buildpack value: `FOO=test` +Build config: `FOO.append=another-value, FOO.delim=:` +Final value: `FOO=test:another-value` + +Buildpack value: `FOO=test` +Build config: `FOO.override=another-value` +Final value: `FOO=another-value` + +Buildpack value: `FOO=test` +Platform Enviroment varaible: `FOO=value` +Build config: `FOO=another-value` +Final value: `FOO=value` + +Platform Enviroment varaible: `FOO=value` +Build config: `FOO.override=another-value` +Final value: `FOO=another-value` + +Platform Enviroment varaible: `FOO=value` +Build config: `FOO.prepend=another-value, FOO.delim=:` +Final value: `FOO=another-value:value` + +Platform Enviroment varaible: `FOO=value` +Build config: `FOO.append=another-value, FOO.delim=:` +Final value: `FOO=value:another-value` + +In case a buildpack uses `clear-env=true` then it is up to the buildpack will not see user provided platform values unless it looks in the platform directory and resolve the user provided platform values against the values set by previous buildpacks and the build config. + +# Migration +[migration]: #migration + +N/A + +# Drawbacks +[drawbacks]: #drawbacks + +- Increased complexity in sourcing of environment variables. + +# Alternatives +[alternatives]: #alternatives + + +## Implementing at a Buildpack level + +See https://github.com/paketo-buildpacks/rfcs/pull/241 + +# Prior Art +[prior-art]: #prior-art + +- See https://github.com/paketo-buildpacks/rfcs/pull/241 and +- See the CNB BAT meeting. https://youtu.be/e8FgLwVN5VQ?t=1153 + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +N/A + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +Addition of the definition of the above directory in the Platform specification i.e. - + +- `CNB_BUILD_CONFIG_DIR` +- `/cnb/build-config/` diff --git a/text/0110-deprecate-apis.md b/text/0110-deprecate-apis.md new file mode 100644 index 000000000..a44ac3bfc --- /dev/null +++ b/text/0110-deprecate-apis.md @@ -0,0 +1,189 @@ +# Meta + +[meta]: #meta + +- Name: Deprecate old buildpack and platform APIs +- Start Date: 2022-06-28 +- Author(s): natalieparellano +- Status: Approved +- RFC Pull Request: [rfcs#226](https://github.com/buildpacks/rfcs/pull/226) +- CNB Pull Request: (leave blank) +- CNB Issue: N/A +- Supersedes: N/A + +# Summary + +[summary]: #summary + +Today, the CNB project [defines](https://github.com/buildpacks/spec/releases) 7 buildpack APIs and 7 platform APIs. The +CNB lifecycle, for backwards compatibility and in order to enable buildpacks and platforms to upgrade their respective +APIs independently, currently supports all 14 APIs and all 49 combinations of buildpack and platform APIs. This is a +maintenance burden. + +Additionally, as we've talked about adding new features to the project, it has become difficult to determine how these +additions might be compatible with older APIs. This has slowed down progress within the project. + +As we progress toward 1.0, it seems prudent to start deprecating APIs from which we have made breaking changes. + +This RFC proposes: + +* Marking as deprecated the following APIs: + * buildpack APIs 0.6 and older + * platform APIs 0.6 and older +* A general policy of: + * Removing support for deprecated APIs 6 months after those APIs are marked as deprecated + +# Definitions + +[definitions]: #definitions + +[**lifecycle**](https://github.com/buildpacks/lifecycle): software that orchestrates a CNB build; it executes in a series of phases that each have a distinct +responsibility + +The CNB project distributes the lifecycle in two ways: + +* As a **lifecycle image**: a minimal [image](https://hub.docker.com/r/buildpacksio/lifecycle) containing the lifecycle + binaries +* As a **lifecycle tarball**: a [`.tgz` file](https://github.com/buildpacks/lifecycle/releases) containing the lifecycle + binaries + +The [**lifecycle +descriptor**](https://github.com/buildpacks/rfcs/blob/main/text/0049-multi-api-lifecycle-descriptor.md#lifecycle-descriptor) +is data describing the lifecycle and the APIs that it supports. It is contained in the `io.buildpacks.lifecycle.apis` +and `io.buildpacks.lifecycle.version` labels on lifecycle images, and the `lifecycle.toml` file in lifecycle tarballs. + +**platform**: system or software that orchestrates the lifecycle by invoking each lifecycle phase in order + +[**Buildpack API**](https://github.com/buildpacks/spec/blob/main/buildpack.md): specifies the interface between a lifecycle program and one or more buildpacks + +[**Platform API**](https://github.com/buildpacks/spec/blob/main/platform.md): specifies the interface between a lifecycle program and a platform + +# Motivation + +[motivation]: #motivation + +- Why should we do this? Lower maintenance burden, enable faster development of new APIs. + - We should draw the boundary at the 0.6 APIs because: + - They are both over a year old (March 2021) + - For buildpacks, the 0.6 API is the last API with unstandardized SBOMs. + - For platforms, the 0.6 API is the last API where `detect` happens before `analyze`. + +- What use cases does it support? As a lifecycle maintainer, I only want to support the latest buildpack and platform + APIs. As a CNB contributor, when thinking about new features, I want a smaller set of supported APIs to care about. + +- What is the expected outcome? Hopefully, not too much inconvenience for buildpack authors and platform operators, + because: + - Hopefully they are already on newer versions of the APIs ( + our [user survey](https://docs.google.com/presentation/d/10CBBld2VV0iCfrYPMbFI4--kh9UZ9mJVXhE1p1Cjqls/edit#slide=id.p) + supports this), but if not... + - We will have socialized this change appropriately (e.g., in Slack, through the mailing list, GitHub, etc.) + - The `CNB_DEPRECATION_MODE` setting will have alerted them that upgrading is necessary + - We will have published [migration guides](https://buildpacks.io/docs/reference/spec/migration/) to help them upgrade + - 6 months is a reasonable amount of time to complete this process + +# What it is + +[what-it-is]: #what-it-is + +`io.buildpacks.lifecycle.apis` would contain the following: + +``` +{ + "buildpack": { + "deprecated": [ "0.2", "0.3", "0.4", "0.5", "0.6" ], + "supported": [ "0.2", "0.3", "0.4", "0.5", "0.6", "0.7", "0.8" ] + }, + "platform": { + "deprecated": [ "0.3", "0.4", "0.5", "0.6" ], + "supported": [ "0.3", "0.4", "0.5", "0.6", "0.7", "0.8", "0.9" ] + } +} +``` + +As described +in [RFC 0049](https://github.com/buildpacks/rfcs/blob/main/text/0049-multi-api-lifecycle-descriptor.md#lifecycle-descriptor) +, if a buildpack or platform tries to use a deprecated API: + +* If `CNB_DEPRECATION_MODE` is unset, the lifecycle will print a warning and continue +* If `CNB_DEPRECATION_MODE`=`warn`, the lifecycle will print a warning and continue +* If `CNB_DEPRECATION_MODE`=`error`, the lifecycle will fail +* If `CNB_DEPRECATION_MODE`=`silent`, the lifecycle will continue w/o warning + +After 6 months, the deprecated APIs would become unsupported and `io.buildpacks.lifecycle.apis` would contain the +following: + +``` +{ + "buildpack": { + "deprecated": [ ], + "supported": [ "0.7", "0.8", "" ] + }, + "platform": { + "deprecated": [ ], + "supported": [ "0.7", "0.8", "0.9", "" ] + } +} +``` + +* If a buildpack tries to use an unsupported API, the lifecycle will fail with a message such + as: `buildpack API version '0.6' is incompatible with the lifecycle`. +* If a platform tries to use an unsupported API, the lifecycle will fail with a message such + as: `platform API version '0.6' is incompatible with the lifecycle`. + +# Migration + +[migration]: #migration + +### Buildpack API + +Buildpack authors will need to update buildpacks using the old APIs to a newer API. These buildpacks will need to be +re-published and re-discovered. + +Platform authors / builder authors will need to re-create builders using the newer buildpacks. + +End-users will need to consumer the newer buildpacks and / or builders. + +### Platform API + +Platform authors will need to update their platform implementations to use the newer platform API. The appropriate value +of `CNB_PLATFORM_API` must be set in the lifecycle's execution environment. + +End-users will need to update their usage of the newer images if they were formerly using a platform +API < [0.4](https://github.com/buildpacks/spec/releases/tag/platform%2Fv0.4) (see +[RFC 0045](https://github.com/buildpacks/rfcs/blob/main/text/0045-launcher-arguments.md)). + +# Drawbacks + +[drawbacks]: #drawbacks + +Why should we *not* do this? + +* This will inevitably cause extra work for a fraction of buildpacks users. + +# Alternatives + +[alternatives]: #alternatives + +- What other designs have been considered? Doing nothing. +- Why is this proposal the best? If we want to make progress toward a 1.0 version of the spec, we need to start dropping + older APIs. +- What is the impact of not doing this? Slower progress in the project due to the maintenance burden and difficulty + conceptualizing compatibility concerns for so many APIs. + +# Prior Art + +[prior-art]: #prior-art + +* [Ubuntu release cycle](https://ubuntu.com/about/release-cycle) +* [Golang release policy](https://go.dev/doc/devel/release#policy) +* [Docker Community Edition release cadence](https://www.serverwatch.com/server-news/docker-18-06-ce-debuts-alongside-new-release-cadence/) +* [Kubernetes support window](https://kubernetes.io/blog/2020/08/31/kubernetes-1-19-feature-one-year-support/) +* CNB RFC that was put "on pause" due to difficulties managing API + complexities: https://github.com/buildpacks/rfcs/pull/145 + +# Spec. Changes (OPTIONAL) + +[spec-changes]: #spec-changes +Does this RFC entail any proposed changes to the core specifications or extensions? If so, please document changes here. + +We should clearly mark the deprecated specs as deprecated on their respective branches. diff --git a/text/0111-support-insecure-registries.md b/text/0111-support-insecure-registries.md new file mode 100644 index 000000000..cbc7f3ba4 --- /dev/null +++ b/text/0111-support-insecure-registries.md @@ -0,0 +1,74 @@ +# Meta +[meta]: #meta +- Name: Support for pull images from or push images to insecure image registries. +- Start Date: 2022-08-17 +- Author(s): wanjunlei +- Status: Approved +- RFC Pull Request: [rfcs#229](https://github.com/buildpacks/rfcs/pull/229) +- CNB Pull Request: (leave blank) +- CNB Issue: N/A +- Supersedes: N/A + +# Summary +[summary]: #summary + +This RFC describes how to pull images from or push images to insecure image registries when building images using buildpacks. + +# Definitions +[definitions]: #definitions + +Insecure image registry - The insecure image registry mentioned in this RFC refers to the image registry using the http protocol, and with a non-internal ip or a domain name. Because currently buildpacks can access insecure image registries using internal ip. + +# Motivation +[motivation]: #motivation + +To fix issue [Support insecure registries in non-daemon case](https://github.com/buildpacks/lifecycle/issues/524). + +To pull images from insecure image registries when building images using buildpacks, and push target images to insecure image registries after the build is complete. + +# What it is +[what-it-is]: #what-it-is + +With this RFC, user can use images in insecure image registries to build image, and push the compiled image to the insecure image registries like this. + +```shell +creator --run-image=develoment-registry.com/run-java:v16 --insecure-registry=develoment-registry.com --insecure-registry=testing-registry.com testing-registry.com/java-sample:latest +``` + +The flag `--insecure-registry` will be added to analyzer, export, restorer, rebaser and image tool too. + +# How it Works +[how-it-works]: #how-it-works + +With this [PR](https://github.com/buildpacks/imgutil/pull/154), the component that buildpacks used to operate images, already supports pulling images from or pushing images to insecure registries. +We should create a image with insecure registry by calling [NewImage](https://github.com/buildpacks/imgutil/blob/main/remote/remote.go#L119) like this + +```shell +remote.NewImage(imageName, authn.DefaultKeychain, withRegistrySetting("mydomain.registry.com:1080", true, false) +``` +> The second parameter specify whether the registry is insecure. +> The third parameter can be always false. + +It is necessary to judge whether the registry where this image is located is insecure. If so, it needs to set the insecure registry through `WithRegistrySetting`. + +The `NewImage` can set base image and prev image using `FromBaseImage` and `WithPreviousImage`, so it may need to call `WithRegistrySetting` multiple times to set up multiple insecure registries. + +# Drawbacks +[drawbacks]: #drawbacks + +N/A + +# Alternatives +[alternatives]: #alternatives + +N/A + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +N/A + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +A new flag `--insecure-registry` will add to analyzer, creator, export, restorer, rebaser and image tool. \ No newline at end of file diff --git a/text/0112-launcher-sbom.md b/text/0112-launcher-sbom.md new file mode 100644 index 000000000..bb2080381 --- /dev/null +++ b/text/0112-launcher-sbom.md @@ -0,0 +1,163 @@ +# Meta + +[meta]: #meta + +- Name: SBOM for lifecycle / launcher +- Start Date: 2022-06-27 +- Author(s): natalieparellano +- Status: Approved +- RFC Pull Request: [rfcs#225](https://github.com/buildpacks/rfcs/pull/225) +- CNB Pull Request: (leave blank) +- CNB Issue: N/A +- Supersedes: N/A + +# Summary + +[summary]: #summary + +Today, SBOMs produced for CNB-built images do not include information describing the CNB lifecycle itself - including +the launcher, a component in the final image. This RFC proposes a mechanism by which the lifecycle can provide an SBOM +describing itself (as a build-time dependency) and the launcher (as a runtime dependency). + +# Definitions + +[definitions]: #definitions + +SBOM: A list of components in a piece of software. Software vendors often create products by assembling open source and +commercial software components. The SBOM describes the components in a product. + +lifecycle: software that orchestrates a CNB build. + +launcher: an executable that is the `ENTRYPOINT` of the exported OCI image. It is used to start processes at runtime. + +# Motivation + +[motivation]: #motivation + +- Why should we do this? A more complete SBOM for CNB-built images. +- What use cases does it support? As an end user, I want SBOM information for every component of my software supply + chain. + +# What it is + +[what-it-is]: #what-it-is + +The SBOM for a CNB-built image could be broken down as follows: + +* **Base image dependencies** + * The platform is responsible for providing SBOM information (see the closure + of https://github.com/buildpacks/rfcs/pull/211 in favor of https://github.com/buildpacks/rfcs/pull/195). Out of + scope for this RFC. +* **Buildpack-provided dependencies** + * SBOM files output by buildpacks are copied to `/sbom/build/` + and `/sbom/run/`. `/sbom/run` is a layer in the final image. + See [RFC 0095](https://github.com/buildpacks/rfcs/blob/main/text/0095-sbom.md). +* **CNB lifecycle** + * Currently there is no SBOM information associated to the image. A CycloneDX SBOM is included in lifecycle + [releases](https://github.com/buildpacks/lifecycle/releases), but it is left up to individual end users to locate + this information. The subject of this RFC. + +## Proposed Changes: + +* The lifecycle will ship with SBOM files in CycloneDX, SPDX, and Syft formats. + * [Lifecycle images](https://hub.docker.com/r/buildpacksio/lifecycle) will include the following SBOM files: + * `/cnb/lifecycle/lifecycle.sbom.cdx.json` + * `/cnb/lifecycle/lifecycle.sbom.spdx.json` + * `/cnb/lifecycle/lifecycle.sbom.syft.json` + * `/cnb/lifecycle/launcher.sbom.cdx.json` + * `/cnb/lifecycle/launcher.sbom.spdx.json` + * `/cnb/lifecycle/launcher.sbom.syft.json` + * Analogously, [lifecycle tarballs](https://github.com/buildpacks/lifecycle/releases) will include these files + rooted at `./lifecycle` - e.g., `/cnb/lifecycle/lifecycle.sbom.cdx.json` above will + be `./lifecycle/lifecycle.sbom.cdx.json` in the tarball. + +* Builder authors should include the new SBOM files in builder images. + * There shouldn't be any changes required to `pack builder create` as `pack` takes a lifecycle tarball as an input, + and the files will be there in the tarball. + +* Before `export`, the lifecycle will copy the above files to: + * `/cnb/lifecycle/lifecycle.sbom.cdx.json` -> `/sbom/build/buildpacksio_lifecycle/sbom.cdx.json` + * `/cnb/lifecycle/lifecycle.sbom.spdx.json` -> `/sbom/build/buildpacksio_lifecycle/sbom.spdx.json` + * `/cnb/lifecycle/lifecycle.sbom.syft.json` -> `/sbom/build/buildpacksio_lifecycle/sbom.syft.json` + * `/cnb/lifecycle/launcher.sbom.cdx.json` -> `/sbom/launch/buildpacksio_lifecycle/launcher/sbom.cdx.json` + * `/cnb/lifecycle/launcher.sbom.spdx.json` -> `/sbom/launch/buildpacksio_lifecycle/launcher/sbom.spdx.json` + * `/cnb/lifecycle/launcher.sbom.syft.json` -> `/sbom/launch/buildpacksio_lifecycle/launcher/sbom.syft.json` + +* `buildpacksio/lifecycle` will become a reserved buildpack ID + +* The `exporter` when adding the launcher layer should use the layer ID `buildpacksio/lifecycle:launcher`, instead + of `launcher` (what it is today). This provides a clear mapping between layer IDs and SBOM paths. + * For consistency, we may also wish to update the IDs for the `launch.sbom`, `config`, and `process-types` layers to + have the prefix `buildpacksio/lifecycle:`, even though these layers have no associated SBOM. + +* Because they are a part of `/sbom/launch`, SBOM files describing the launcher will be exported in the final + image. + * SBOM files in `/sbom/build` may be saved off by the platform before the build container exits (no changes + to the existing workflow). + +* If no SBOM files are found in `/cnb/lifecycle` (e.g., if the builder author did not include them), the lifecycle will warn and + continue. Alternatively, the lifecycle could generate the files on the fly, but this would increase build times. + * If there is a previous image, `/sbom/launch/buildpacksio_lifecycle` may contain SBOM files - but these + should be deleted before `export` as there is no guarantee that the lifecycle that created the previous image is + the same as the current lifecycle. + +* In theory, there should be no changes needed for end-users to consume the new SBOM files, as the files will be placed + in the same directory, with the same structure, as SBOM files for buildpack-provided dependencies. + +* To expose the SBOM formats that it supports, a new field `[lifecycle.sbom-formats]` will be added to lifecycle.toml, + where `sbom-formats = [ "" ]` must be supported SBOM media types (just like for buildpack.toml). + +# Drawbacks + +[drawbacks]: #drawbacks + +Why should we *not* do this? More work for the lifecycle. + +# Alternatives + +[alternatives]: #alternatives + +- What other designs have been considered? + - This could all be implemented with a buildpack - e.g., a `buildpacksio/lifecycle` utility buildpack whose sole + responsibility would be to copy the SBOM files from `/cnb/lifecycle` to `/sbom`. However, this would + introduce quite a bit of complexity and overhead for what is ultimately a very simple operation. A potential + benefit is that platforms using Platform API `0.8` and higher wouldn't need to upgrade - but they would need to + add the new utility buildpack to builders, which is probably just as much work. + - In theory, we don't actually need the files to be in `/sbom//buildpacksio_lifecycle` - + because any files restored from a previous build will be ignored (see above). They could be annotations or + attestations on the image instead (see https://github.com/buildpacks/rfcs/pull/195) - but this would require + platforms to keep track of the files as inputs to the signer binary. + +- Why is this proposal the best? It is easy and straightforward for the lifecycle to copy pre-generated files + to `/sbom//buildpacksio_lifecycle`. + +- What is the impact of not doing this? A less complete SBOM for CNB-built images. + +# Prior Art + +[prior-art]: #prior-art + +Discuss prior art, both the good and bad. + +* RFCs: + * Base image dependencies: https://github.com/buildpacks/rfcs/pull/211, https://github.com/buildpacks/rfcs/pull/195 + * Buildpack-provided dependencies: [RFC 0095](https://github.com/buildpacks/rfcs/blob/main/text/0095-sbom.md) + +# Unresolved Questions + +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to be resolved through implementation of the feature? + - Tooling or libraries used by the lifecycle to generate SBOM files describing itself. + +# Spec. Changes (OPTIONAL) + +[spec-changes]: #spec-changes +Does this RFC entail any proposed changes to the core specifications or extensions? If so, please document changes here. + +The platform spec should make mention of the new SBOM files. The buildpack spec should note +that `buildpacksio_lifecycle` is a reserved buildpack ID. + +The `exporter` and the `creator` will accept a new flag, `-launcher-sbom`, for a directory containing SBOM files for a +provided `-launcher`. The lifecycle will warn if `-launcher` is provided without `-launcher-sbom`. The lifecycle will +ignore `-launcher-sbom` if no `-launcher` is provided. diff --git a/text/0113-additonal-oci-artifacts.md b/text/0113-additonal-oci-artifacts.md new file mode 100644 index 000000000..0954b032a --- /dev/null +++ b/text/0113-additonal-oci-artifacts.md @@ -0,0 +1,90 @@ +# Meta +[meta]: #meta +- Name: Additional OCI Artifacts from Buildpacks +- Start Date: 2022-04-24 +- Author(s): agracey +- Status: Approved +- RFC Pull Request: https://github.com/buildpacks/rfcs/pull/223 +- CNB Pull Request: n/a +- CNB Issue: n/a +- Supersedes: n/a + +# Summary +[summary]: #summary + +Allow buildpack authors to specify additional artifacts from their buildpacks to be uploaded to a registry along with the standard set of layers. + +# Definitions +[definitions]: #definitions + + +- WebAssembly (WASM) -- Portable binaries to be run with a WebAssembly virtual machine. +- [OCI Artifact](https://github.com/opencontainers/artifacts/blob/main/definitions-terms.md#media-type) -- Any blob of data that can be stored in an OCI registry. From the spec: +> An artifact has a type, which has a collection of layers. The Artifact is defined as unique by its `manifest.config.mediaType`. Layers are defined by their `layer.config.mediaType`. + +# Motivation +[motivation]: #motivation + +Cloud Native Buildpacks are awesome for building container images but come with a draw back that they can only produce a single output. This is limiting to platform builders because there are many cases where you might want to have multiple outputs from the same build process. + +# What it is +[what-it-is]: #what-it-is + +This change allows for additional flexibility in what buildpack authors and platforms can build on top of buildpacks. Three initial use cases are WebAssembly, test output, and Helm charts. + +For the first, a buildpack would be able to also generate a wasm bundle that can be used by a downstream platform (such as the Krustlet or runwasi). + +Similarly, by allowing the buildpack to produce a helm chart, you could integrate in logic about how to set up an application into the build pipeline where you are able to do bits of code analysis. (NetworkPolicies that lock down an app, sidecars that need to be run, ConfigMaps to pass around out-of-band, etc...) + +Lastly, it may be advantageous to allow a buildpack to produce the results of unit tests and publish alongside the container itself. Then a tool (such as Kubewarden) could gate running of the container based on the passing tests. + +# How it Works +[how-it-works]: #how-it-works + +Add an optional file per layer called `artifacts.toml` that tells lifecycle to publish the specified file(s) in the layer to an OCI registry in the publish step. It also needs to allow for setting labels and the tag to publish with. + +For the WASM use-case, a buildpack author looking to also publish a node.js WASM bundle would write the build logic like any other buildpack. Create a directory in $LAYERS_DIR with the following `artifacts.toml`: + +``` +[[artifact]] +tag = "wasm" +file = "mywasmprogram.wasm" # Should be a file relative to the current $LAYERS_DIR + +[[artifact.labels]] +ConfigMediaType = "application/vnd.wasm.config.v1+json" +ContentLayerMediaType = "application/vnd.wasm.content.layer.v1+wasm" +``` + +As a note: to keep this generic to any potential OCI artifact, it seems best to provide a labelling mechanism instead of prescribing certain fields. + +# Migration +[migration]: #migration + +This functionality should be purely additive and not break existing buildpacks. + +# Drawbacks +[drawbacks]: #drawbacks + +This does bring in additional scope to the project and could potentially lead to confusion around what OCI image that got published should be used. This can be mitigated with good docs and platform output. Adding a change to make the standard image output optional seems like a larger change than is appropriate at the moment. + +# Alternatives +[alternatives]: #alternatives + +Other ways of doing this would be to add the artifacts to the primary OCI image directly then pull back out in a future stage. This has two obvious issues: artifact size and escalation in tooling needed in pipeline. + +# Prior Art +[prior-art]: #prior-art + +None that I'm aware of. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- [] What to do when there are collisions in tags. +- [] Could the [Bindle](https://github.com/deislabs/bindle) project help? + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +The only addition to the spec is the addition of the artifacts.toml as well as the changes to lifecycle to parse and act on the new config. + diff --git a/text/0114-distribution-team-merge-platform.md b/text/0114-distribution-team-merge-platform.md new file mode 100644 index 000000000..22e99e51e --- /dev/null +++ b/text/0114-distribution-team-merge-platform.md @@ -0,0 +1,104 @@ +# Meta +[meta]: #meta +- Name: Merge Distribution Team with Platform Team +- Start Date: 2022-12-01 +- Author(s): [@jkutner](https://github.com/jkutner) +- Status: Draft +- RFC Pull Request: https://github.com/buildpacks/rfcs/pull/239 +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: [RFC-0062](https://github.com/buildpacks/rfcs/blob/main/text/0062-distribution-team.md) + +# Summary +[summary]: #summary + +This is a proposal to consolidate two existing teams, Distribution and Platform, into a single team. + +# Definitions +[definitions]: #definitions + +* Platform - pack and any other platforms or platform components maintained by the [Platform Team](https://github.com/buildpacks/community/blob/main/TEAMS.md#platform-team) +* Distribution - tools and services that support the distribution, discovery, and integration of buildpacks. Owned by the [Distribution Team](https://github.com/buildpacks/community/blob/main/TEAMS.md#distribution-team) + +# Motivation +[motivation]: #motivation + +The distribution team only has one maintainer, which is not sustainable and present a redunancy risk. However, the distribution team's workload is fairly small, consisting mostly of security patches and simple releases. At the same time, the Platform team has lost key resources including a maintainer and team lead. The components owned by the Distribution team align well with the Platform team's mission. For these reasons, we aim to consolidate leadership energy by merging these teams. + +# What it is +[what-it-is]: #what-it-is + +The active maintainers and contributors of the Distribution Team will move into the Platform team with the same role. The components owned by the Distribution Team will henceforth be owned by the Platform Team. A single Team Lead will continue to represent Platform Team. + +The Distribution Team will be removed from the CNB [Governance model](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md). + +# How it Works +[how-it-works]: #how-it-works + +The Platform Team will have the following members: + +* Team Lead: Terence Lee +* Maintainers: Terence Lee, Joe Kutner, David Freilich +* Contributors: All active contributors from both existing teams + +The Platform Team will own the following components: +* [pack](https://github.com/buildpacks/pack) +* [Tekton Tasks + Pipelines](https://github.com/buildpacks/tekton-integration) +* [CircleCI Pack Orb](https://github.com/buildpacks/pack-orb) +* [Buildpack Registry API](https://github.com/buildpacks/registry-api) +* [Buildpack Registry Index](https://github.com/buildpacks/registry-index) +* [Buildpack Registry Namespace Owners](https://github.com/buildpacks/registry-namespaces) +* [Github Actions](https://github.com/buildpacks/github-actions) + +# Migration +[migration]: #migration + +N/A + +# Drawbacks +[drawbacks]: #drawbacks + +* This increases the surface of the Platform Team, which already suffered from attrition. It is also one the team that owns one of the most critical CNB components: pack. This could put additional strain on its members. + +# Alternatives +[alternatives]: #alternatives + +* Do nothing - keep the same team structure. Both the Platform team and Distribution team would lack redundancy +* Disolve the Distribution Team - sunset the components it owns and find alternatives to Buildpack Registry and github-actions + +# Prior Art +[prior-art]: #prior-art + +* [Create a Distribution Team](https://github.com/buildpacks/rfcs/blob/main/text/0062-distribution-team.md) +* [Announcing OpenTelemetry: the merger of OpenCensus and OpenTracing](https://cloudblogs.microsoft.com/opensource/2019/05/23/announcing-opentelemetry-cncf-merged-opencensus-opentracing/) + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +TBD + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +N/A + +# History +[history]: #history + + diff --git a/text/0115-rebase-immutable-image-ref.md b/text/0115-rebase-immutable-image-ref.md new file mode 100644 index 000000000..901911b49 --- /dev/null +++ b/text/0115-rebase-immutable-image-ref.md @@ -0,0 +1,118 @@ +# Meta +[meta]: #meta +- Name: Rebase by Image Digest Reference +- Start Date: 2022-12-08 +- Author(s): [@joeybrown-sf](https://github.com/joeybrown-sf) +- Status: Implemented +- RFC Pull Request: [rfcs#262](https://github.com/buildpacks/rfcs/pull/262) +- CNB Pull Request: https://github.com/buildpacks/lifecycle/pull/985 +- CNB Issue: [buildpacks/lifecycle#983](https://github.com/buildpacks/lifecycle/issues/983) +- Supersedes: N/A + +# Summary +[summary]: #summary + +Allow passing a digest reference as rebase target. + +# Definitions +[definitions]: #definitions + +An **image reference** refers to either a **tag reference** or **digest reference**. + +A **tag reference** refers to an identifier of form `/:` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. + +A **digest reference** refers to a [content addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) identifier of form `/@` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. + + +# Motivation +[motivation]: #motivation + +Enables rebasing by targeting an immutable image digest. There are some scenarios where the **digest reference** is preferred over **tag reference**. + +# What it is +[what-it-is]: #what-it-is + +This is a feature to expand the lifecycle rebase command to allow targeting an image by either `tag` or `digest`. + +Today, `lifecycle` returns the following error when appempting to use a **digest reference**: +``` +ERROR: failed to rebase: failed to write image to the following tags: [localhost:5003/foo/bar@sha256:916a9e100569ee521b86d03b8499b9b93d7d256d6e838868ae720295f2ea2f76: PUT http://localhost:5003/v2/foo/bar/manifests/sha256:916a9e100569ee521b86d03b8499b9b93d7d256d6e838868ae720295f2ea2f76: DIGEST_INVALID: provided digest did not match uploaded content] +``` + +This error could be avoided if digest references were permitted. + +# How it Works +[how-it-works]: #how-it-works + +Today, we can execute rebase by using **tag references** but not **digest references**. + +Here are some examples of valid rebase commands. **Tag** is `latest` if not specified: + +``` +lifecycle rebase my-repo/foo +``` +``` +lifecycle rebase my-repo/foo:latest +``` +``` +lifecycle rebase my-repo/foo:v4 +``` + +It is not currently possible to target an image using a **digest reference**. + +_The proposed feature will provide a mechanism to target an image rebase by tag reference or digest reference._ + +Here is what targeting an image via digest will look like: +``` +lifecycle rebase -previous-image my-repo/foo@sha256:1234 -tag my-repo/foo:rebase my-repo/foo +``` + +- When using a digest reference as the image target, the caller may specify zero or more `` to apply to exported image. If no `tag` is provided, `latest` will be used. +- If `-previous-image` is not provided, it is infered from the first argument. This is similar behavior to `analyzer`, for instance. + +# Migration +[migration]: #migration + +This is backwards compatible. + +# Drawbacks +[drawbacks]: #drawbacks + +# Alternatives +[alternatives]: #alternatives + +# Prior Art +[prior-art]: #prior-art + +`pack` explicitly does not support this. There is a friendly validation message in `pack`: + +` is not a tag reference` + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + + +# History +[history]: #history + + diff --git a/text/0116-stop-deleting-cache-images.md b/text/0116-stop-deleting-cache-images.md new file mode 100644 index 000000000..821ca38df --- /dev/null +++ b/text/0116-stop-deleting-cache-images.md @@ -0,0 +1,74 @@ +# Meta +[meta]: #meta +- Name: Stop deleting cache images +- Start Date: 2022-03-31 +- Author(s): jabrown85 +- Status: Approved +- RFC Pull Request: [rfcs#216](https://github.com/buildpacks/rfcs/pull/216) +- CNB Pull Request: (leave blank) +- CNB Issue: N/A +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +Lifecycle will no longer try to delete previous cache images when publishing cache images. + +# Definitions +[definitions]: #definitions + +* Cache Image: The cache stored between builds - stored in a registry. +* ECR: Amazon's container registry product + +# Motivation +[motivation]: #motivation + +- Why should we do this? + As discussed in [buildpacks/lifecycle#803](https://github.com/buildpacks/lifecycle/issues/803), some registries (ECR) do not support `DELETE`. For platforms that work exclusively with such registries, the warning output by lifecycle and the time taken to fail is unavoidable and lifecycle is wasting time trying to complete an operation that will never succeed. + +- What use cases does it support? + All platforms that use cache images against registries that do not support delete will no longer see warning messages. + +- What is the expected outcome? + Platforms will need to handle cleanup of their cache images on their own, if they desire. + +# What it is +[what-it-is]: #what-it-is + +Lifecycle will no longer attempt to delete cache images during cache image export. + +# How it Works +[how-it-works]: #how-it-works + +Lifecycle will no longer attempt to delete cache images during cache image export. + +# Migration +[migration]: #migration + +Lifecycle will document this behavior change in Release Notes/Changelog along with the associated Platform API that enables the new behavior. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +Platform authors relying on this behavior will need to take additional measures to ensure cache image cleanup or the destination registry will continue to grow. + +# Alternatives +[alternatives]: #alternatives + +- What other designs have been considered? + * Add regex or configuration to drive registry hosts to ignore during DELETE * + * Stop deleting cache images by default in newer platform API versions, but add a platform-level configuration to enable previous behavior. +- Why is this proposal the best? + * Lifecycle is not currently cleaning up any other resources + * Deleting the cache images can hurt reproducibility + * There are more public registries that don't allow DELETE +- What is the impact of not doing this? + * End users continue seeing warnings the platform can do nothing about. + + +# Prior Art +[prior-art]: #prior-art + +Discussion at [buildpacks/lifecycle#803](https://github.com/buildpacks/lifecycle/issues/803). diff --git a/text/0117-buildpacks-community.md b/text/0117-buildpacks-community.md new file mode 100644 index 000000000..d92c089ab --- /dev/null +++ b/text/0117-buildpacks-community.md @@ -0,0 +1,156 @@ +# Meta +[meta]: #meta +- Name: Buildpacks Community +- Start Date: 2023-01-26 +- Author(s): [@samj1912](https://github.com/samj1912) +- Status: Approved +- RFC Pull Request: [rfcs#273](https://github.com/buildpacks/rfcs/pull/273) +- CNB Pull Request: (leave blank) +- CNB Issue: N/A +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +The Buildpacks Community is a vendor-neutral Github organization where trusted community provided Cloud Native Buildpacks tooling, platforms and integrations can live. This would provide users a trusted place to search for Buildpack integrations maintained by the community. + + +# Definitions +[definitions]: #definitions + +- **Buildpacks Community** - The Buildpacks Community is a vendor-neutral Github organization where trusted community provided Cloud Native Buildpacks tooling, platforms and integrations can live. +- **Buildpacks Leadership** - The Buildpacks Leadership is a group of trusted individuals who are responsible for the Buildpacks Community. They have the ability to create repositories in the Buildpacks Community and approve new projects to be added to the Buildpacks Community. This will consist of the TOC and Team-leads of the Buildpacks project. +- **Buildpacks TOC** - The Buildpacks TOC is the technical oversight committee for the Buildpacks project. The TOC is responsible for the technical direction of the Buildpacks project. + + +# Motivation +[motivation]: #motivation + +There are two reasons why this community should exist. + +- The Buildpacks Community will allow for the testing of new technologies or the development of integrations in an environment that is more flexible than that of the core Buildpacks organization. This will provide a staging area for integrations that the Buildpacks team deems important but we are not yet ready to commit to long term maintanance. + +- A trusted repository of community integrations will also allow for a trusted source of integrations that solve common yet still relatively niche problems that are not suitable to be added to core Buildpacks organization. This will highlight integrations of high-quality and provide a vendor-neutral umbrella for them to live. They will also benefit from improved CI/CD resources and a common governance model. + +# What it is +[what-it-is]: #what-it-is + + + +For a project to be admitted to the Buildpacks community organization, it must meet the following criteria: + +- The project must be a tooling, platform or integration that is related to Cloud Native Buildpacks. +- The project must be open source and licensed under Apache 2.0. +- It must follow the Cloud Native Computing Foundation Code of Conduct. +- The project must enable DCO signoff for all commits. +- The project must be open to contributions and have a public issue tracker. +- The project must have a governance document that clearly defines the project maintainers and how they are elected. Each project may choose to define their own governance model as long as it is clearly documented and allows for project maintainers to be elected from the community. +- The list of project maintainers must be publicly available and controlled through a Github team. +- The project must use a CODEOWNERS file to define the maintainers for each repository. The CODEOWNERS file should reference the Github team that controls the list of maintainers. +- All project contributors must be members of the Buildpacks community organization. +- The project must be actively maintained (i.e. issues and pull requests must be addressed regularly, approved pull requests must be merged or updated in a timely manner, etc.). +- There should have visible automated testing for all repositories that are part of the project. +- The project maintainers must conform to a set of best effort SLOs around patching critical CVEs when applicable to the project. +- The project should strive have the following community health files: + - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. + - DEVELOPMENT.md: A guide to how contributors should develop the project. + - ADOPTERS.md: A list of adopters of the project. + - VERSIONING.md: A guide to how versioning is done for the project. + - RELEASE.md: A guide to how releases are done for the project. + - SECURITY.md: A guide to how security vulnerabilities should be reported. + +This criteria is meant to alleviate the following problems: + +- All projects must meet some testing standard to be trusted in order to ensure that the projects support the latest Buildpacks APIs and are actively maintained. +- All projects must have a clearly defined governance model to ensure that the project maintainers are elected from the community and that the project is open to contributions. +- There must be a defined system in place to reap abandonware. +- If a project maintainers are not making a best effort of patching out or updating vulnerable software then the project as a whole is untrustworthy. + + +# How it Works +[how-it-works]: #how-it-works + +## Project Admission + +A project can be admitted to the Buildpacks community organization by creating a Github issue in the Buildpacks community repository. The issue should contain the following information: + +- Name of the project +- Evidence for the above criteria +- A list of maintainers for the project + +The above information will be structured into an appropriate issue template. The Buildpacks Leadership will review the issue and if the project meets the above criteria, the project will be added to the Buildpacks community organization. The Buildpacks Leadership will assign a team to the project and the team lead of the team will steward the project - i.e., will be responsible for ensuring that the project meets the above criteria. + +Once admitted, the team lead of the steward team will create a Github team for the project and add the project maintainers to the team and mark them as the team maintainers allowing them to add other maintainers. The existing team maintainers of the steward team will be added as maintainers to the project team. + +The team lead will also create a CODEOWNERS file for the project and add the project maintainers as the code owners. + +The project maintainers will be responsible for maintaining the list of project maintainers and ensuring that all project contributors are members of the Buildpacks community organization. + +They will be able to add new Github members to the organization by creating a Github issue in the Buildpacks community [invites repository](https://github.com/buildpacks-community/invites). + +## Project Removal + +In case the project fails to meet the above criteria, the Buildpacks Leadership will work with the project maintainers to address the issues and if the project is still not ready, the project will be archived or removed from the Buildpacks community organization. + +## Project Graduation + +In case a project is deemed to be mature enough to be part of the core Buildpacks organization, the project maintainers can request for the project to be graduated to the core Buildpacks organization via the [Component Contribution RFC](https://github.com/buildpacks/rfcs/blob/main/text/0108-governance-component-maintainer-role.md). The Buildpacks TOC will review the request and if the project meets the criteria for graduation, the project will be moved to the core Buildpacks organization. + + +# Migration +[migration]: #migration + +N/A + +# Drawbacks +[drawbacks]: #drawbacks + + +N/A + +# Alternatives +[alternatives]: #alternatives + +N/A + +# Prior Art +[prior-art]: #prior-art + +- [CNCF Sandbox](https://www.cncf.io/sandbox-projects/) +- [Paketo Community](https://github.com/paketo-buildpacks/rfcs/blob/main/text/0008-paketo-community.md) +- [Argoproj Labs](https://github.com/argoproj-labs) +- [Crossplane Contrib](https://github.com/crossplane-contrib) + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- The exact project onboarding issue template. + +# History +[history]: #history + + + diff --git a/text/0118-2023H1-roadmap.md b/text/0118-2023H1-roadmap.md new file mode 100644 index 000000000..ca582b201 --- /dev/null +++ b/text/0118-2023H1-roadmap.md @@ -0,0 +1,166 @@ +# Meta +[meta]: #meta +- Name: 2023H1 Roadmap +- Start Date: 2023-01-24 +- Author(s): hone +- Status: Approved +- RFC Pull Request: [#272](https://github.com/buildpacks/rfcs/pull/272) +- CNB Pull Request: (leave blank) +- CNB Issue: +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +This RFC details the first half of the 2023 Roadmap as well as changing the roadmaps to twice a year aligning with KubeCon EU and NA (CDD - Conference Driven Development). + +# Definitions +[definitions]: #definitions + +N/A + +# Motivation +[motivation]: #motivation + +The project has done annual roadmaps in the past, but they haven't been without their challenges. The items range from concrete to large themes. The larger scoped items like "Integration with the Cloud Native Ecosystem" is nebulous and not always clear what success look at the end of it. The roadmap has felt like a smaller vision document versus a set of prioritized items to be accomplished. + +Without clear guidance on how to shape the roadmap, it's also been a challenge to get one out (not that we're doing great this year). We missed 2022's roadmap altogether and didn't publish one until March for 2021. + +Once we did publish a roadmap, items wouldn't necessarily move forward even though we commited as a project to work on them. Items didn't always have an owner and there was no skin in the game for suggesting an idea. + +In addition, we don't do a good job of reviewing our roadmap whether that's regularly through the year or at the end of the year. + +As an incubation level project, there is an opportunity to broadcast announcements at both KubeCon EU and NA. As a project, we're often caught flatfooted with sharing highlights or announcements during these conferences. + + +# What it is +[what-it-is]: #what-it-is + +## Roadmap Changes + +For 2023, I want to propose trying something different this time around. The project will publish two roadmaps one for each half of the year aligned with work to be completed for KubeCon EU (2023-04-18) and KubeCon NA (2023-11-6). The goals I'm hoping to see: + +- Focus - Decrease the amount we're doing as a project, so we're able to deliver on the things we commit to. +- Accountability - With that focus, we also need to be accountable for what we're putting on this list. +- Marketing - Everyone involved works hard to make this project successful. We should take time to celebrate and talk about the work being done. + +### Smaller Scope + +Going forward, the roadmap will account for 6 months (in this case it's just under 4 months) worth of work. This forces items on the roadmap to be smaller and concrete since they have to be something that can be accomplished in that time frame. While it was nice having the large items, they were hard to execute and it wasn't clear what the finished state looked like. Any larger piece work will need to broken down to make it on the roadmap in a 6 month chunk. It's also not too small, that some larger chunks of work can be planned. Having a second roadmap each year, also allows us to course correct mid-year. + +Not only are individual items smaller, but we as a project should commit to less so they can be accomplished aren't just a bunch of empty promises. This means hard decisions will need to be made to cut highly requested features from the roadmap. + +Since this is an OSS project, things that don't make the roadmap can still be pursued by others and welcome! With finite time of maintainers, there still may be limited support depending on the maintainer. Also, items can also bubble up during the next roadmap cycle and people should advocate for them in the GitHub discussion or slack. + +### Ownership + +In order to ensure things make progress, every item in the roadmap will have an owner from the project leadership team of TOC members or team maintainers. If someone really wants something on there, they will need to volunteer themselves to help keep it on track if no one else will. This will help keep the number of items on the roadmap from ballooning. The owner will also be responsible for keeping everyone up to date. All roadmap items should link to something to GitHub where things can be tracked. + +### KubeCon Alignment + +As stated above, we'll be making roadmaps with work finishing by a KubeCon event. They're coveniently around 6 months apart give or take a few weeks. As an incubation project, we're able to share project announcements at these events. This will keep the key items we're working on top of mind, and what gets finished can easily be shared. Roadmap items can also make good talk material for the maintainer track. + +With how strict travel budgets are in the current economic climate, these events are one of the few times some of us can get together. Nothing can really replace in person conversations and brainstorming. These discussions can feed into the natural recap/review of the conference and as a way to kickstart roadmap review and planning. + +## 2023H1 Roadmap + +### Release Base Image Extension +* Owner: @natalieparellano +* Links: [RFC](https://github.com/buildpacks/rfcs/blob/main/text/0105-dockerfiles.md) + +This started out as [Stack Buildpacks](https://github.com/buildpacks/rfcs/blob/main/text/0069-stack-buildpacks.md) and now Dockerfile Extensions. Significant work has already been done on this feature over the last year. This roadmap item is about seeing this work through with releasing phase 3 in both `lifecycle` and `pack`. + +### Remove Stacks & Mixins +* Owner: @jkutner +* Links: [RFC](https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md) + +This RFC was merged in 2021 and is a dependency on Base Image Extensions. In order to get us to 1.0, we'll need to take on some of these painful backwards breaking changes in the best way possible. This work will include the Buildpack & Platform spec changes with support in `lifecycle` and `pack`. + +### Execution Environments RFC +* Owner: @hone +* Links: [RFC](https://github.com/buildpacks/rfcs/pull/274) + +There has long been a desire for a "Test" support, but it's never been prioritized even though it's made the roadmap before. Not to be over ambitious, the first step is to get a RFC written and accepted. + +### Project Health +* Owner: @samj1912 +* Links: [Buildpacks Community Organization RFC](https://github.com/buildpacks/rfcs/pull/273) + +Like other [CNCF projects](https://github.com/cncf/toc/issues?q=is%3Aissue+sort%3Aupdated-desc+%22health+of%22+-label%3A%22project+onboarding%22+-label%3A%22sandbox%22+), the project has been impacted by the VMware + Broadcom acquisition. The goal of this item is to improve the general health of the project and grow contributors back to our 2020 numbers. This inludes every team having a set of active set of maintainers and contributors, thus removing the TOC needing to step in for platform. + +As for concrete items to be accomplished: + +* Establish a buildpacks-community to be used as a labs/staging area to help hype up experiments that we would be otherwise wary of investing in and, if they succeed, adopt them in the main buildpacks org. +* Participate in mentorship programs to grow contributors like [GSoC](https://summerofcode.withgoogle.com/) and [LFX Mentorship](https://lfx.linuxfoundation.org/tools/mentorship/). + +### Pack Test Cleaning/Optimizations +* Owner: @dfreilich +* Links: [Pack Pull Request](https://github.com/buildpacks/pack/pull/1498) + +Currently, the pack acceptance tests are very complex for newcomers. In order to help with contributions, we can relax some of these tests. + +# How it Works +[how-it-works]: #how-it-works + +See [What it is](#what-it-is) for the bulk of the details. For implementing this plan: + +* Open a PR against the [community repo](https://github.com/buildpacks/community) replacing the `ROADMAP.md`. +* As part of the regular leadership meetings we will hold check ins. +* After each KubeCon there will be a recap session and kicking off the next roadmap planning. +* In 2024, we will review how this compares to the normal annual roadmap we've traditionally done. + +# Migration +[migration]: #migration + +N/A + +# Drawbacks +[drawbacks]: #drawbacks + +- This will be more work with twice the number of roadmap plannings. +- There will be more overhead to ensure accountability. + +# Alternatives +[alternatives]: #alternatives + +## Do Nothing + +We can continue to do the annual roadmap. This hasn't proved very successful, but we can still take the lessons learned and adjust the current process. + +# Prior Art +[prior-art]: #prior-art + +- [Buildpacks 2021 Roadmap](https://github.com/buildpacks/community/pull/72) +- [Rust 2021 Roadmap](https://blog.rust-lang.org/2020/09/03/Planning-2021-Roadmap.html) +- [TypeScript Roadmap](https://github.com/microsoft/TypeScript/wiki/Roadmap) + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- How many items should any one person be able to own? + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +N/A + +# History +[history]: #history + + diff --git a/text/0119-export-to-oci.md b/text/0119-export-to-oci.md new file mode 100644 index 000000000..2dacbc55e --- /dev/null +++ b/text/0119-export-to-oci.md @@ -0,0 +1,732 @@ +# Meta +[meta]: #meta +- Name: Export to OCI format +- Start Date: 2022-02-22 +- Author(s): Juan Bustamante (@jjbustamante) +- Status: Approved +- RFC Pull Request: [rfcs#203](https://github.com/buildpacks/rfcs/pull/203) +- CNB Pull Request: (leave blank) +- CNB Issue: N/A +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +Add the capability to the `Exporter` phase to save the image to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. + +# Definitions +[definitions]: #definitions + +- A [Platform](https://buildpacks.io/docs/concepts/components/platform/) uses a lifecycle, Buildpacks (packaged in a builder), and application source code to produce an OCI image. +- A [Lifecycle](https://buildpacks.io/docs/concepts/components/lifecycle/) orchestrates Buildpacks execution, then assembles the resulting artifacts into a final app image. +- A **Daemon** is a service, popularized by Docker, for downloading container images, and executing and managing containers from those images. +- A **Registry** is a long-running service used for storing and retrieving container images. +- An **image reference** refers to either a tag reference or digest reference. +- A **tag reference** refers to an identifier of form `/:` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. +- A **digest reference** refers to a [content addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) identifier of form `/@` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. +- A **image Manifest** provides a configuration and set of layers for a single container image for a specific architecture and operating system. +- The **layer diffID** is the hash of the uncompressed layer +- The **layer digest** is the hash of the compressed layer. +- An [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) is the directory structure for OCI content-addressable blobs and [location-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage#Content-addressed_vs._location-addressed) references. + +# Motivation +[motivation]: #motivation + +### Why should we do this? + +Lifecycle translates an application source code into an OCI image, in order to do this, it can be configured to interact with a docker daemon (using `daemon` flag) or with an OCI registry. + +The [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) is the directory structure for OCI content-addressable blobs and [location-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage#Content-addressed_vs._location-addressed) references. + +The current process, executed by the lifecycle, does not take into consideration cases where a platform implementor may require to pass through the inputs or want to save the final application image on disk using [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. + +### What use cases does it support? + +Exporting to [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) will enable new user's workflows on top of this functionality. For example: + - It provides a mechanism to reduce the Lifecycle complexity by removing the interaction with the Daemon in the future. + - Solve the problem of losing information when the image is saved into the Daemon, keeping the image on disk along with the metadata generated by the Lifecycle. The OCI Image can be used as input for other tools to offer more capabilities to the end users. + - This feature will help to unblock uses cases like + - OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) + - Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) + - Export to tarball. See [issue](https://github.com/buildpacks/lifecycle/issues/423) + +### What is the expected outcome? + +Lifecycle will be capable of exporting the application image into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. The image saved on disk could have the following considerations: + +- The `blobs` directory MAY be missing `base image` or `run image` blobs. These layers may not be needed on disk as they could be already accessible in a blob store. +- The `blobs` directory SHOULD always have buildpacks generated `blobs`. + + +# What it is +[what-it-is]: #what-it-is + +The proposal is to add a new capability to the lifecycle (enabled by configuration) to resolve any **image reference** (input or output) to a disk location in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. It means, instead of interacting with a daemon or registry lifecycle will interact against the filesystem to read or write any **image reference**. + +The target personas affected by this change are: + +- **Platform implementors**: they will have to take care of the responsibility of creating a store resource on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format and pass it through the lifecycle during the phases execution. + +The process of writing any image on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be expensive in terms of hard drive space or IO operation (compressing or uncompressing layers). In order to provide flexibility for the implementation, the `analyzer` or `exporter` binaries only require the *Image Manifest* and the *Image Config* to execute their operations on the previous image and run image; based on this, we proposed the Lifecycle can be configured to work with a partial representation of the images on disk, meaning that some blobs MAY be missing (which is ok according to the [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format). The missing blobs SHOULD be those that are already available in a daemon or registry. + +Let's see some examples of the proposed behavior + +## Examples + +### Requirements + +Lifecycle will converts image references into local paths following define [rules](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) and the content must be in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. + +Let's suppose a *platform implementor* creates a directory with the following structure: + +```=shell +layout-repo +└── index.docker.io + ├── cnb + │ ├── my-full-stack-run:bionic + │ │ └── bionic + │ │ └── blobs + │ │ ├── sha256 + │ │ │ ├── 1f59...944a // manifest + │ │ │ ├── 6388...af5a // config + │ │ │ ├── 824b...f984e + │ │ │ ├── f5f7...5b38 + │ │ │ └── 870e...f1b09 + │ │ ├── index.json + │ │ └── oci-layout + │ └── my-partial-stack-run:bionic + │ └── bionic + │ ├── blobs + │ │ └── sha256 + │ │ ├── 1f59...944a // manifest + │ │ └── 6388...af5a // config + │ ├── index.json + │ └── oci-layout + └── bar + └── my-previous-app + └── latest + ├── blobs + │ └── sha256 + │ ├── 4bcd5..x // app image manifest + │ ├── 5f789..d // app image config + │ ├── 624b...f984e // run layer + │ └── 4g234..f // buildpack layer + ├── index.json + └── oci-layout +``` + +The images named **cnb/my-full-stack-run** and **cnb/my-partial-stack-run** represents the same image but the partial one has missing `blobs`, those `blobs` are the layers that are already available in the store from it came from. + +For each example case, I will present two ways of enabling the new capability: + +- Using an environment variables +- Using the new `-layout` and `layout-dir` flags + +In any case the expected output is the same. + +#### Analyze phase + +##### Analyzing run-image full saved on disk + +```=shell +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo +> /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/my-full-stack-run:bionic my-app-image +``` + +expected analyzed.toml output + +```=toml +[run-image] + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + +``` + +##### Analyzing run-image partial saved on disk + +```=shell +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo +> /cnb/lifecycle/analyzer -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image +``` + +expected analyzed.toml output + +```=toml +[run-image] + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + +``` + +##### Analyzing previous-image + + ```=shell +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo +> /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic -previous-image bar/my-previous-app my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/my-full-stack-run:bionic-previous-image bar/my-previous-app my-app-image +``` + +expected analyzed.toml output + +```=toml +[run-image] + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + +[previous-image] + reference = "/layout-repo/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" + +``` + +##### Analyzing run-image not saved on disk + +```=shell +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo +> /cnb/lifecycle/analyzer -run-image cnb/bad-run-image my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/bad-run-image my-app-image + +# expected output + +ERROR: the run-image could not be found at path: /layout-repo/index.docker.io/cnb/bad-run-image/latest +``` + +##### Analyzing without run-image argument + +```=shell +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo +> /cnb/lifecycle/analyzer my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo my-app-image + +# expected output + +ERROR: -run-image is required when OCI Layout feature is enabled +``` + +##### Analyzing without layout-dir argument + +```=shell +> export CNB_USE_LAYOUT=true +> /cnb/lifecycle/analyzer -run-image cnb/bad-run-image my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -run-image cnb/bad-run-image my-app-image + +# expected output + +ERROR: defining a layout directory is required when OCI Layout feature is enabled. Use -layout-dir flag or CNB_LAYOUT_DIR environment variable +``` + +Let's also check some examples when the export phase is executed + +#### Export phase + +##### Export to OCI using run-image full saved on disk + +```=shell +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo +> /cnb/lifecycle/exporter my-app-image + +# OR + +> /cnb/lifecycle/exporter -layout -layout-dir /layout-repo my-app-image +``` + +The output will be written into the repository folder described above and it should looks like this: + +```=shell +layout-repo +└── index.docker.io + ├── cnb + │ └── my-full-stack-run:bionic + │ └── bionic + │ └── blobs + │ ├── sha256 + │ │ ├── 1f59...944a // manifest + │ │ ├── 6388...af5a // config + │ │ ├── 824b...f984e + │ │ ├── f5f7...5b38 + │ │ └── 870e...f1b09 + │ ├── index.json + │ └── oci-layout + └── library + └── my-app-image + └── latest + ├── blobs + │ └── sha256 + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ ├── 824b...f984e // run layer + │ ├── f5f7...5b38 // run layer + │ ├── 870e...f1b09 // run layer + │ └── 3g234..f // buildpack layer + ├── index.json + └── oci-layout + +``` + +As we can see, the application image `my-app-image` contains a **full** copy of the layers in its `blobs` folder. + + +##### Export to OCI using run-image partially saved on disk + +```=shell +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo +> /cnb/lifecycle/exporter my-app-image + +# OR + +> /cnb/lifecycle/exporter -layout -layout-dir /layout-repo my-app-image +``` + +Expected output: + +```=shell +layout-repo +└── index.docker.io + ├── cnb + │ └── my-partial-stack-run:bionic + │ └── bionic + │ ├── blobs + │ │ └── sha256 + │ │ ├── 1f59...944a // manifest + │ │ └── 6388...af5a // config + │ ├── index.json + │ └── oci-layout + └── library + └── my-app-image + └── latest + ├── blobs + │ └── sha256 + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ └── 3g234..f // buildpack layer + ├── index.json + └── oci-layout + +``` + +As we can see, the application image `my-app-image` has missing `blobs` because they were not provided as input and the lifecycle just **skip writing** those layers on disk. + +##### Using -layout flag in combination with --daemon or --publish flags + +Any combination of using multiple sources or sinks in the Lifecycle invocation of phases should throw an error to the user. For example: + +```=shell +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo +> /cnb/lifecycle/exporter -daemon -run-image cnb/my-full-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/exporter -layout -layout-dir /layout-repo -daemon -run-image cnb/my-full-stack-run:bionic my-app-image + +ERROR: exporting to multiple targets is unsupported +``` + +# How it Works +[how-it-works]: #how-it-works + +The lifecycle phases affected by this new behavior are: + - [Analyze](https://buildpacks.io/docs/concepts/components/lifecycle/analyze/) + - [Restore](https://buildpacks.io/docs/concepts/components/lifecycle/restore/) + - [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) + - [Create](https://buildpacks.io/docs/concepts/components/lifecycle/create/) + +At a high level view the proposed solution can be summarized with the following system landscape diagram from the C4 model + +![](https://i.imgur.com/y972lTD.png) + +Notice that we are relying on the OCI format Specification to expose the data for `Platforms` + +The following new inputs are proposed to be added to these phases + + | Input | Environment Variable | Default Value | Description + |-------|-----------------------|---------------|-------------- + | `` | `CNB_USE_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | + | `` | `CNB_LAYOUT_DIR` | | Path to a directory where the images are saved in OCI layout format| + +## How to map an image reference into a path in the layout repository + +In the previous examples one key element was how to translate an image reference into a path, let's define those rules. + +Considering an **image reference** refers to either a tag reference or digest reference. It could have the following formats +- A name reference refers to an identifier of form `//:` +- A digest reference refers to a content addressable identifier of form `//@:` + +The image look up will be done following these rules: + - WHEN `the image points to a name reference` + - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `////` + - WHEN `the image points to a digest reference` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `/////` + - WHEN `` is not provided default value will be **index.docker.io** + - IF `` is not also provided, then default value will be **library** + + +## Examples + +In all the examples the new feature is enabled by the use of the new flag `-layout` or by setting +the new environment variable `CNB_USE_LAYOUT` to true. + +Let's review some previous examples + +#### Analyze phase + +##### Analyzing run-image full saved on disk + +Command: + +```=shell +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo +> /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/my-full-stack-run:bionic my-app-image +``` + +Arguments received: + + - `run-image`: `cnb/my-full-stack-run:bionic` + - `image`: `my-app-image` + +The `` is set with the value `/layout-repo` + +Lifecycle applies the rules for looking up the images: + - It takes the **tag reference** `cnb/my-full-stack-run:bionic`, applies the conversion rules and gets `/index.docker.io/cnb/my-full-stack-run/bionic` + - It will append the `` at the beginning, getting the following path: `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic` + - It will look for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. + - In case of the *application image* it will look at path `/layout-repo/index.docker.io/library/my-app-image/latest` + +Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. The `run-image.reference` added into the `analyzed.toml` will contain the path resolved by the lifecycle plus the digest reference to the image with the following format `[path]@[digest]`. In case of this example, it will look like this: + +```=toml +[run-image] + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" +``` + +##### Analyzing run-image partial saved on disk + +Command received: + +```=shell +> export CNB_USE_LAYOUT=true +> /cnb/lifecycle/analyzer -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image +``` + +Arguments received: + + - `run-image`: `cnb/my-full-partial-run:bionic` + - `image`: `my-app-image` + +The `` is set with the default value `/layout-repo` + +Noticed the structure of the `run-image` provided + +```=shell +layout-repo +└── index.docker.io + └── cnb + └── my-partial-stack-run:bionic + └── bionic + ├── blobs + │ └── sha256 + │ ├── 1f59...944a // manifest + │ └── 6388...af5a // config + ├── index.json + └── oci-layout +``` + +Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase with the information from the **Image Manifest** and the **Image Config** + +The output `analyzed.toml` will also include the new `run-image.reference` field the path and the digest of the run image. + +```=toml +[run-image] + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" +``` + +##### Analyzing previous-image + +Command received: + +```=shell +> export CNB_USE_LAYOUT=true +> /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic -previous-image bar/my-previous-app my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic -previous-image bar/my-previous-app my-app-image +``` + +Arguments received: + +- `run-image`: `cnb/my-full-stack-run:bionic` +- `previous-image`: `bar/my-previous-app` +- `image`: `my-app-image` + +The `` is set with the default value `/layout-repo` + +`run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/index.docker.io/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. + +The `analyzed.toml` file es expected to be updated with the `previous-image.reference` containing the path and the digest of the `previous-image` + +```=toml +[run-image] + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + +[previous-image] + reference = "/layout-repo/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" + +``` + +Let's check how the `export` examples works on detailed + +##### Export to OCI using run-image full saved on disk + +Pre-conditions: + +The following directories are accessible by the lifecycle +```=shell +/ +├── layout-repo +│ └── index.docker.io +│ └── cnb +│ └── my-full-stack-run:bionic +│ └── bionic +│ └── blobs +│ ├── sha256 +│ │ ├── 1f59...944a // manifest +│ │ ├── 6388...af5a // config +│ │ ├── 824b...f984e +│ │ ├── f5f7...5b38 +│ │ └── 870e...f1b09 +│ ├── index.json +│ └── oci-layout +└── layers + └── analyzed.tom +``` + +The `/layers/analyzed.toml` file contains the following data: + +```=toml +[run-image] + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + +``` + +Command executed: + +```=shell +> export CNB_USE_LAYOUT=true +> /cnb/lifecycle/exporter my-app-image + +# OR + +> /cnb/lifecycle/exporter -layout my-app-image +``` + +Arguments received: + +- `image`: `my-app-image` + +The `` is set with the default value `/layout-repo` + +Lifecycle: + - It will read the `[run-image]` section in the `analyzed.toml`, it will parse `reference` attribute using the `@` separator and load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. + - Lifecycle could also validate the digest of the image loaded is the same as the one established by the `reference`. + - Lifecycle will execute the export steps and at the end of the process it will write the *application image* at path `/layout-repo/index.docker.io/library/my-app-image/latest` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format + + +The output image will be written at: + +```=shell +layout-repo +└── index.docker.io + └── library + └── my-app-image + └── latest + ├── blobs + │ └── sha256 + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ ├── 824b...f984e // run layer + │ ├── f5f7...5b38 // run layer + │ ├── 870e...f1b09 // run layer + │ └── 3g234..f // buildpack layer + ├── index.json + └── oci-layout + +``` + +## Proof of concept + +In order to validate the feasibility of the proposed feature, we developed a proof of concept with one of the most important side effects this capability can add into the project: **Removing the Daemon Support**. You can also check a recording with the demo in the following [link](https://drive.google.com/file/d/1W1125OHuyUlx88BRroUTLBfrFHhFM5A9/view?usp=sharing) + +As mentioned earlier, if we want to remove the daemon support in the Lifecycle, then all the responsibility to deal with it goes into the platforms implementors, that means, for example: +- Pull the require dependencies (runtime image for example), save them on disk in OCI layout format and pass it through the lifecycle using the `` parameter +- Push the application image (exported in OCI layout format) into the Daemon, because that is what users are expecting. + +During the proof of concept implementation I choose to use [skopeo](https://github.com/containers/skopeo) tool to solve the problem of interacting with the Daemon. The reason to do it was **simplicity** for the PoC developed but we believe this is a good subject to talk about with the community. + +The following workflow was developed: +- Pack download the [skopeo image](https://github.com/containers/skopeo/blob/main/install.md#container-images), similar as it is downloading the other dependencies (Lifecycle, Buildpacks) +- Pack executes [skopeo copy](https://github.com/containers/skopeo/blob/main/docs/skopeo-copy.1.md) command in a container + - Copy image from the Daemon into the filesystem, in OCI layout format, before running Lifecycle + - Copy image from filesystem into the Daemon after the export phase was executed + +The following Dynamic Diagram from the C4 model, can give a little idea of the pieces implemented during the Poc + +![](https://i.imgur.com/SkY3l62.png) + +### Measuring of performance impact + +In order to have an idea on how much is affected the performance of exporting to the Daemon using the OCI layout format, the following metrics were taken. + +Using a local workstation with the following specifications: +- **(MacOS 12.3.1 / 2,4 GHz 8-Core Intel Core i9 / 32 GB 2667 MHz DDR4 / 1 TB APFS SSD HD)** + +We built 5 times the Java, Kotlin and Ruby samples codes from our [repository](https://github.com/buildpacks/samples/tree/main/apps) and took the building's average time using the Daemon and the OCI layout format approaches. + +The table above summarized the results we got. + +![](https://i.imgur.com/zuPZ6Xk.png) + +Times are expressed in **seconds** and the first thing we noticed is for Java and Kotlin the `build time` can be affected by the network and the availability of maven repositories, so I decided to take the `same build time` to compare both approaches. + +Here are my thoughts about these results: +- Java and Kotlin behavior are very similar, exporting only to OCI format increases 5% the time compared to Daemon approach, and from user perspective it represents a 5 seconds increase of time. +- On the other hand for the Ruby application, exporting to OCI format represents a 20% increase of the time but from user perspective it is only 1.5 seconds which is probably difficult to notice from user perspective. +- When the time spent for Pack to prepare the environment for the lifecycle execution (downloading the run-image from a registry to OCI format) and loading the OCI image from disk to the Daemon (which is the expected behavior from Users) is added, then: + - The Java and Kotlin applications time increases was **13%**, representing **+13 seconds** from user perspective + - The Ruby application increases **82%** but from user's side it represents **+7 seconds** + +Let's take a look on what happened when we execute a build for the second time, the table below summarized the results + +![](https://i.imgur.com/zDAOZU6.png) + +On these cases we can see the behavior is consistent compared with the previous case, Java and Kotlin application shows a **5% increase** of time but Ruby application, because it's process time is smaller the sensibility to variation is bigger (23%) but in reality it represents a **+1 second** of difference for the User. Also, when the pre and post processing time is added the variations are bigger for all the applications. As mentioned, [skopeo](https://github.com/containers/skopeo) tool was used here and most of the time spent goes into this category. + +I think, this PoC demonstrate that adding the exporting to OCI layout format is a valuable feature for the project, it opens the door to deprecate the use of Daemon but it will requires that platform implementors to prepare and post-process the output on disk on a smart way to reduce the performance penalties to users. + +# Migration +[migration]: #migration + +## For the scope of this RFC + +- No breaking changes were identified + +# Drawbacks +[drawbacks]: #drawbacks + +- We could increase the disk space if we do not manage the duplication of saving the layers on disk. The proposal suggests to use symbolic links to reference layers on disk and avoid duplication. + +# Alternatives +[alternatives]: #alternatives + +- What other designs have been considered? + - Doing nothing, just keep exporting only to Daemon or Registry + +- What other designs have been considered for removing the Daemon support? + - Instead of exporting to OCI layout format, the other approach considered is exporting to [registry only](https://github.com/buildpacks/rfcs/blob/jjbustamante/feature/deprecate-daemon/text/0000-deprecate-daemon.md#lifecycle-registry-only-approach). In this case, the Lifecycle only interacts with registries. + As part of the PoC, I took some metrics to compare impact of using a ephemeral registry to publish the application image. The strategy done to capture the metrics was: + - I used a script to set up a local [container registry](https://hub.docker.com/_/registry) before executing the `pack build` command + - For the **first build** metrics, the registry was destroyed/re-created before each execution + - `pack build` command was configure to `--publish` in the local registry + - I didn't use the [skopeo]() in these cases to complete the pushing into the Daemon + + Here are the results: + + ![](https://i.imgur.com/vtOjxJP.png) + + - The results are actually very similar to exporting to OCI layout format for Java and Kotlin, but Ruby application is actually worst. + + ![](https://i.imgur.com/FfbqfF6.png) + + - Second build is actually better compared with the export to OCI in disk, Java and Kotlin increases the time just by **2%**, but Ruby again is worst + + Some thoughts about this approaches + + - **Process Management:** Platforms must now manage a parallel process (registry in the daemon). This would entail ensuring that the registry is started and cleaned up appropriately. + - **Networking:** There are additional network complications in order to route images to the ephemeral registry. For example, [network drivers](https://docs.docker.com/network/#network-drivers), [proxy](https://docs.docker.com/desktop/networking/#httphttps-proxy-support) and [DNS configuration](https://docs.docker.com/config/containers/container-networking/#dns-services), [host name resolution](https://docs.docker.com/desktop/networking/#i-want-to-connect-from-a-container-to-a-service-on-the-host), and [TLS certificates](https://betterprogramming.pub/deploy-a-docker-registry-using-tls-and-htpasswd-56dd57a1215a) to name a few. + +- Why is this proposal the best? [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format is a standard from which other tools can create a [OCI Runtime Specification bundle](https://github.com/opencontainers/runtime-spec/blob/v1.0.0/bundle.md) exporting to this format enables Platforms to implement any feature in the top of this format, for example, exporting to [containerd](https://containerd.io) has been [requested](https://github.com/buildpacks/lifecycle/issues/829) by the community and it could be implemented if we can get the application image exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +- What is the impact of not doing this? Probably will never remove the Daemon support in the Lifecycle + +# Prior Art +[prior-art]: #prior-art + +- Discussion around removing the Daemon support [RFC](https://github.com/buildpacks/rfcs/blob/jjbustamante/feature/deprecate-daemon/text/0000-deprecate-daemon.md) + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to be resolved before this gets merged? + - Tools like [umoci](https://umo.ci/) used to create a runtime bundle from an image in OCI layout format, requires the [annotation](https://github.com/opencontainers/image-spec/blob/main/annotations.md#pre-defined-annotation-keys) `org.opencontainers.image.ref.name` to be present. Also tools like [skopeo](https://github.com/containers/skopeo) when an image is `copy` in oci format the annotation is included. + We are not adding the annotation as part of the Buildpacks Specification, but in this case this could make our output incompatible with some other tooling. + - **Answer:** we agreed on adding `org.opencontainers.image.ref.name` annotation + - Exporting to a tarball can be handled on this RFC or a new one must be created? + - **Answer:** this can be handled on a different RFC + +- What parts of the design do you expect to be resolved through implementation of the feature? + - Handle symbolic links to the blobs in the `` repository, this could be more efficient on hard drive space + - **Answer:** this can be handled on the implementation side + + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +The [Platform Interface Specification](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#inputs-5) must be updated to include the following inputs to the [Create](https://buildpacks.io/docs/concepts/components/lifecycle/create/), [Analyze](https://buildpacks.io/docs/concepts/components/lifecycle/analyze/) and [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) phases + +| Input | Environment Variable | Default Value | Description| +|-------|-----------------------|---------------|------------| +| `` | `CNB_USE_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | +| `` | `CNB_LAYOUT_DIR` | | Path to a directory where the images are saved in OCI layout format| + +Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the `reference` format in case of layout is being used. + +```=toml +[image] + reference = "" + +[run-image] + reference = "" + +[previous-image] + reference = "" +``` + +Where + +- `[image|run-image|previos-image].reference` MUST be either: + - A digest reference to an image in an OCI registry + - The ID of an image in a docker daemon + - The path to an image in OCI layout format + diff --git a/text/0120-cvebackports.md b/text/0120-cvebackports.md new file mode 100644 index 000000000..1d20a99ff --- /dev/null +++ b/text/0120-cvebackports.md @@ -0,0 +1,121 @@ +# Meta +[meta]: #meta +- Name: CVE discretional Patching +- Start Date: 2023-03-08 +- Author(s): joe-kimmel-vmw, natalieparellano +- Status: Approved +- RFC Pull Request: [rfcs#281](https://github.com/buildpacks/rfcs/pull/281) +- CNB Pull Request: (leave blank) +- CNB Issue: N/A +- Supersedes: N/A + +# Summary +[summary]: #summary +This RFC describes how maintainers MAY issue patch releases in response to critical and high severity CVEs being detected in past or current releases of the lifecycle binary. + +# Definitions +[definitions]: #definitions +CVE - Literally expands to “Common Vulnerabilities and Exposures” but in general it refers to a security gap which could be exploited by a malicious attacker, which can be fixed by patching a single component. + +CVE Severity: CVEs are announced with a severity score. While this score can vary between vendors, it is made by considering many factors including how easy it is to take advantage of the exploit, what resources are exposed by the exploit, and whether there is already a known exploit in circulation. See https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator + +Critical and High are the two highest severity ratings available under the current severity rating system. + + +# Motivation +[motivation]: #motivation + +- Why should we do this? + +Patching CVEs is industry best practice. By providing patched updates of recent releases we enable CNB-using organizations with low risk tolerance or high upgrade friction to make a minimal change to their runtime infrastructure that still keeps secure against known vulnerabilities. +Following patch version line architecture also boosts user trust vs. a pledge to roll forward security fixes to higher version lines with the promise of no breaking changes, which would be looked upon skeptically by some enterprise users. Those same users who are skeptical of any & all promises that new versions won’t have any regressions are especially likely to be running older versions in the wild, thus providing patches to older versions ensures that they can continue to use CNB lifecycle without risking exposure to the riskiest CVEs. + +- What use cases does it support? + +This is especially important for users who do not use CNB as part of a SaaS product, i.e. for “on-prem” deployments. These on-prem users are pulling down the provided lifecycle images and running the binaries on their own infrastructure, thus increasing their potential risk exposure and liability. + +Additionally, as we do have concrete plans to deprecate platform API versions in the lifecycle this calendar year, there's increased likelihood to learn of +other lifecycle consumers who are not ready to upgrade and who would appreciate patch releases. + +- What is the expected outcome? + +Maintainers MAY issue patch releases in response to critical and high CVEs. Most importantly, users MAY upgrade to consume these patch releases in a timely manner if they are not comfortable consuming the latest version. + +# What it is +[what-it-is]: #what-it-is + +It is risk mitigation for consumer and enterprise customers who want to apply critical patches without any other changes that would come with a minor version upgrade. + +# How it Works +[how-it-works]: #how-it-works + +Patch releases will be published at the discretion of the maintainers in response to Critical and High CVEs. + +The lifecycle will still offer the same strong backwards compatibility guarantees as ever. + +Existing process (patch most recent version N until it becomes N-1) bumps dependencies and the go version each month irrespective of CVEs being present. This proposal does not involve changing that process. + +# Migration +[migration]: #migration + +N/A + +# Drawbacks +[drawbacks]: #drawbacks + +Maintaining past releases does take time. However under this proposal that maintenance is optional and performed at the discretion of the maintainers. + +For a vague guess at the volume of this work: In the year from March 2022 March 2023, the grype scanner found 3 High and 0 Critical CVEs in the 0.13.5 lifecycle binary, so there would have been at most 3 additional patch releases of that line. +Similarly, grype scanner found 1 High and 0 Critical CVEs in the 0.14.3 lifecycle image (from October 2022), and 0 High or Critical CVEs in the 0.15.3 lifecycle image (from Jan 2023). + + + +# Alternatives +[alternatives]: #alternatives + +- What other designs have been considered? + - Only Patch the Most Recent Release: This is a fine idea but it only works in a world where all users are willing to migrate to the latest release. This RFC addresses the wants and needs of users who are not comfortable performing minor version upgrades, even with backwards compatibility guarantees, in order to address CVEs. + - Only Patch development trunk, and wait for the next release: This also only works in a world where minor version upgrades are seen as cheap or low-risk to perform. + - Don’t Patch CVEs: we probably wouldn’t ever do this option. + + +- Why is this proposal the best? +This is the only proposal palatable to orgs that view minor releases as high-risk and/or expensive. +- What is the impact of not doing this? +We risk alienating some enterprise users of CNB. + +# Prior Art +[prior-art]: #prior-art + +N/A + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +N/A beyond general feedback and consensus. + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +N/A + +# History +[history]: #history + + diff --git a/text/0121-kpack-donation-to-cnb.md b/text/0121-kpack-donation-to-cnb.md new file mode 100644 index 000000000..4316a97e6 --- /dev/null +++ b/text/0121-kpack-donation-to-cnb.md @@ -0,0 +1,409 @@ +# Meta + +- Name: kpack donation to CNB +- Start Date: 2022-06-21 +- Author(s): [Juan Bustamante](https://github.com/jjbustamante/) +- Status: Approved +- RFC Pull Request: [rfcs#235](https://github.com/buildpacks/rfcs/pull/235) +- CNB Pull Request: (leave blank) +- CNB Issue: N/A +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary + +This RFC proposes the donation of the open-source project [kpack](https://github.com/pivotal/kpack/) into the [Cloud Native Buildpacks Community Organization](https://github.com/buildpacks-community) as a vendor neutral staging ground under the CNB governance umbrella. Once the project is deemed sufficiently mature, the project will be moved under the [Cloud Native Buildpacks Organization](https://github.com/buildpacks). + +Following the process defined in the [Buildpack Commnity RFC](https://github.com/buildpacks/rfcs/blob/main/text/0117-buildpacks-community.md) the following table presents the criteria used to evaluate the project. + +| Criteria | Evidence | +|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| The project must be a tooling, platform or integration that is related to Cloud Native Buildpacks. | See [Motivation](#motivation) section | +| The project must be open source and licensed under Apache 2.0. | See [License](https://github.com/pivotal/kpack/blob/main/LICENSE) | +| List of all external dependencies with licensing info and they’re permissively licensed with a Apache 2.0 compatible license | See [report](https://drive.google.com/file/d/1SvPdl69Lhn0TTx_TaxcesfajuYY9uiNd/view?usp=share_link) generated using [go-licenses](https://github.com/google/go-licenses) | +| It must follow the Cloud Native Computing Foundation Code of Conduct. | See [Code of conduct](https://github.com/pivotal/kpack/blob/main/CODE_OF_CONDUCT.md) | +| The project must enable DCO signoff for all commits. | See [Sign-off process](https://github.com/pivotal/kpack/blob/main/CONTRIBUTING.md#sign-off-process) | +| The project must be open to contributions and have a public issue tracker. | See public [issue tracker](https://github.com/pivotal/kpack/issues) | +| The project must have a governance document that clearly defines the project maintainers and how they are elected. Each project may choose to define their own governance model as long as it is clearly documented and allows for project maintainers to be elected from the community. | See [Governance](https://github.com/pivotal/kpack/blob/main/GOVERNANCE.md) | +| The list of project maintainers must be publicly available and controlled through a Github team. | See [Maintainers](https://github.com/pivotal/kpack/blob/main/MAINTAINERS.md) | +| The project must use a CODEOWNERS file to define the maintainers for each repository. The CODEOWNERS file should reference the Github team that controls the list of maintainers. | See [CODEOWNERS](https://github.com/pivotal/kpack/blob/main/CODEOWNERS) file | +| All project contributors must be members of the Buildpacks community organization. | See [Team Roles](#team-roles) section and [People](https://github.com/orgs/buildpacks-community/people) in CNB community organization | +| The project must be actively maintained (i.e. issues and pull requests must be addressed regularly, approved pull requests must be merged or updated in a timely manner, etc.). | See [issues](https://github.com/pivotal/kpack/issues) and [pull requests](https://github.com/pivotal/kpack/pulls) | +| There should have visible automated testing for all repositories that are part of the project. | See [codecov](https://app.codecov.io/gh/pivotal/kpack) | +| The project maintainers must conform to a set of best effort SLOs around patching critical CVEs when applicable to the project. | | +| The has a file - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. | See [Contributing](https://github.com/pivotal/kpack/blob/main/CONTRIBUTING.md) | +| The has a file - DEVELOPMENT.md: A guide to how contributors should develop the project. | See [Development](https://github.com/pivotal/kpack/blob/main/DEVELOPMENT.md) | +| The has a file - ADOPTERS.md: A list of adopters of the project. | See [Adopters](https://github.com/pivotal/kpack/blob/main/ADOPTERS.md) | +| The has a file - VERSIONING.md: A guide to how versioning is done for the project. | See [Versioning](https://github.com/pivotal/kpack/blob/main/VERSIONING.md) | +| The has a file - RELEASE.md: A guide to how releases are done for the project. | See [Release](https://github.com/pivotal/kpack/blob/main/RELEASE.md) | +| The has a file - SECURITY.md: A guide to how security vulnerabilities should be reported. | See Security [Pull Request](https://github.com/pivotal/kpack/pull/1149) | + +# Definitions + +- [Kubernetes](https://kubernetes.io/) is an open-source system for automating deployment, scaling, and management of containerized applications. +- [Kpack](https://github.com/pivotal/kpack/) is a VMware-led open-source project that utilizes [Kubernetes](https://kubernetes.io/) primitives to build OCI images as a [platform](https://buildpacks.io/docs/concepts/components/platform/) implementation of [Cloud Native Buildpacks](https://buildpacks.io/). +- A Kubernetes native application is an application designed to run on Kubernetes platforms, managed by Kubernetes APIs and `kubectl` tooling and cohesively deployed on Kubernetes as a single object. + +# Motivation + +### Why should we do this? + +It will benefit the [CNB](https://buildpacks.io/) project by adding a tool to support an out-of-the box [Kubernetes](https://kubernetes.io/) integration, which is part of the [CNB](https://buildpacks.io/) [roadmap](https://github.com/buildpacks/community/blob/main/ROADMAP.md#integration-with-the-cloud-native-ecosystem) goals. + +It will show evidence to the community that the project supports multiple [platform interface specification](https://github.com/buildpacks/spec/blob/main/platform.md) implementers increasing community's confidence on the flexibility of specification maintained by the [CNB](https://buildpacks.io/) project. + +It will help the [CNB](https://buildpacks.io/) community (+550 members on slack channel) to grow by adding all the [kpack](https://github.com/pivotal/kpack/) community into [CNB](https://buildpacks.io/) space. + +[CNB](https://buildpacks.io/) is part of the [Cloud Native Computing Foundation](https://www.cncf.io), an open source, vendor neutral hub of cloud native computing projects, the inclusion of [kpack](https://github.com/pivotal/kpack/) under this umbrella will provide more opportunity to the community: + +- Increase in adopters, users looking to use buildpacks in [Kubernetes](https://kubernetes.io/) will find a tool supported and maintained by the [CNB team](https://github.com/buildpacks/community/blob/main/TEAMS.md). +- Improve efficiency, ensuring that the roadmaps of the two projects are closer aligned will make it easier to coordinate efforts between both communities. + +### What use cases does it support? + +[kpack](https://github.com/pivotal/kpack/) will add support to operators by providing declarative [Kubernetes](https://kubernetes.io/) resources (images, builders, or stacks for example) to monitor for security patches on the underlying builder's buildpacks or stacks and rebuild the OCI image when changes are detected, allowing platforms to roll out new versions of the applications when vulnerabilities are fixed. + +### How does kpack support the goals and use cases of the project? + +The [CNB](https://buildpacks.io/) project turns application source code into OCI-compliant container images; in order to do that, it defines a platform-to-buildpack contract that guarantees interoperability between different implementers. + +The [CNB](https://buildpacks.io/) project embraces modern container standards, and [Kubernetes](https://kubernetes.io/) has become the industry standard for automating deployment, scaling, and management of containerized applications. + +[kpack](https://github.com/pivotal/kpack/) fits perfectly in that direction because it implements the [platform interface specification](https://github.com/buildpacks/spec/blob/main/platform.md) and because is a [Kubernetes](https://kubernetes.io/) native application its community possesses a vast knowledge that can provide valuable feedback to the CNB project. + +### Is there functionality in kpack that is already provided by the project? + +[pack](https://github.com/buildpacks/pack) and [kpack](https://github.com/pivotal/kpack/) offer similar functionality (both tools implement the [platform interface](https://github.com/buildpacks/spec/blob/main/platform.md)[specification](https://github.com/buildpacks/spec/blob/main/platform.md)) but they do it for two non-overlapping contexts: while the first one targets developers and local builds, [kpack](https://github.com/pivotal/kpack/) manages containerization on day-2 and at scale and is a [Kubernetes](https://kubernetes.io/) native implementation. + +### Is kpack integrated with another service or technology that is widely used? + +As mentioned earlier, [kpack](https://github.com/pivotal/kpack/) implements the [platform interface specification](https://github.com/buildpacks/spec/blob/main/platform.md) on [Kubernetes](https://kubernetes.io/), a standard nowadays for automating deployment, scaling, and management of containerized applications. + +# What it is + +[Kubernetes](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/) is a portable, extensible, open-source platform for managing containerized workloads and services. The [Kubernetes](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/) API can be extended in different ways; one of them is using [custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/), a custom resource represents a customization of a particular [Kubernetes](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/) installation. + +[kpack](https://github.com/pivotal/kpack/) extends [Kubernetes](https://kubernetes.io/) using [custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) and utilizes unprivileged [Kubernetes](https://kubernetes.io/) primitives to provide builds of OCI images as a platform implementation of [Cloud Native Buildpacks](https://buildpacks.io/). This means that [kpack](https://github.com/pivotal/kpack/) takes the CNB-defined concepts (image, builder, stacks, etc) and bakes them into the Kubernetes extension model using custom resources and exposing a declarative API for interacting with it. + +The declarative API enforces a separation of responsibilities. Operators declare the configuration for a CNB image or define which buildpacks or stacks must be used, and [kpack](https://github.com/pivotal/kpack/) - using its custom controller - will take care of the heavy lifting, keeping the state of the custom objects in sync with the declared desired state. + +# How it Works + +As mentioned before, [kpack](https://github.com/pivotal/kpack/) uses the [custom resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) extension point to provide the capabilities of building OCI images as a platform implementation of [Cloud Native Buildpacks](https://buildpacks.io/). + +These custom resources have a common definition similar to this: + +```yaml +apiVersion: kpack.io/v1alpha2 +kind: [ClusterStack|ClusterStore|Image|Builder|Build] +metadata: + name: [unique name] +``` + +The _apiVersion_ key specifies which version of the Kubernetes API is used to create the object, in this case **kpack.io/v1alpha2** + +The _kind_ key specifies what kind of objects we want to create for example: **ClusterStack, ClusterStore, Image, Builder or Build** + +The _metadata_ key is used to define the data that can uniquely identify the object. One common key used around all the custom resources is to provide a _name_ to identify the object. + +Some of the [custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) implemented by [kpack](https://github.com/pivotal/kpack/) are describe in the next section, if you want to check the complete reference go to [kpack](https://github.com/pivotal/kpack/) documentation [site](https://github.com/pivotal/kpack/tree/main/docs) + +## ClusterStack + +This resource is an abstraction to group a `build image` and a `run image` required to build the application source code. + +Let's see an example of the [ClusterStack](https://github.com/pivotal/kpack/blob/main/docs/stack.md) definition + +```yaml +apiVersion: kpack.io/v1alpha2 +kind: ClusterStack +metadata: + name: base +spec: + id: "io.buildpacks.stacks.bionic" + buildImage: + image: "my-buildpack-repo/build:cnb" + runImage: + image: "my-buildpack-repo/run:cnb" +``` + +The _spec_ key is used to define the desired state of the ClusterStack and the keys availables under _spec_ match the values expected in a CNB [stack](https://buildpacks.io/docs/concepts/components/stack/) definition: + +- _id_: The 'id' of the stack +- _buildImage.image_: The `build-image` of the [stack](https://buildpacks.io/docs/concepts/components/stack/). +- _runImage.image_: The `run-image` of the [stack](https://buildpacks.io/docs/concepts/components/stack/). + +## Cluster Store + +Creates a repository of buildpacks packaged as OCI artifacts to be used during a build. + +Let's see an example of the [ClusterStore](https://github.com/pivotal/kpack/blob/main/docs/store.md) definition + +``` yaml +apiVersion: kpack.io/v1alpha2 +kind: ClusterStore +metadata: + name: my-cluster-store +spec: + sources: + - image: foo.com/my-buildpack-repo/buildpack-1@sha256:sha123 + - image: foo.com/my-buildpack-repo/buildpack-2@sha256:sha345 + - image: foo.com/my-buildpack-repo/builder:base + ``` + +The _spec_ key is used to define the desired state of the [ClusterStore](https://github.com/pivotal/kpack/blob/main/docs/store.md) + +- _sources_: List of buildpackage images to make available in the ClusterStore. Each image is an object with the key _image_. + +As a side note the [ClusterStore](https://github.com/pivotal/kpack/blob/main/docs/store.md) resource will be deprecated in favor of a new Buildpack resource in the near future according to the following [RFC](https://www.google.com/url?q=https://github.com/pivotal/kpack/pull/931&sa=D&source=docs&ust=1665521917723122&usg=AOvVaw1eNN-XzLf5xiX1nvrHKMRE) + +## Builder or ClusterBuilder + +Creates a [CNB builder](https://buildpacks.io/docs/concepts/components/builder/) image that contains all the components necessary to execute a build. + +An example of the [Builder](https://github.com/pivotal/kpack/blob/main/docs/builders.md) definition is as follows: + +```yaml +apiVersion: kpack.io/v1alpha2 +kind: Builder +metadata: + name: my-builder +spec: + tag: foo.com/sample/builder + stack: + name: base + kind: ClusterStack + store: + name: my-cluster-store + kind: ClusterStore + order: + - group: + - id: my-buildpack-repo/buildpack-1 + - group: + - id: my-buildpack-repo/buildpack-2 + ``` + +It's important to notice that a [ClusterStack](https://github.com/pivotal/kpack/blob/main/docs/stack.md) and [ClusterStore](https://github.com/pivotal/kpack/blob/main/docs/store.md) is required for creating a [Builder](https://github.com/pivotal/kpack/blob/main/docs/builders.md). + +The _spec_ key is used to define the desired state of the [Builder](https://github.com/pivotal/kpack/blob/main/docs/builders.md) + +- _tag_: The tag to save the builder image. +- _stack.name_: The name of the stack resource to use as the builder stack. All buildpacks in the order must be compatible with the [ClusterStack](https://github.com/pivotal/kpack/blob/main/docs/stack.md). +- _stack.kind_: The type as defined in [Kubernetes](https://kubernetes.io/). This will always be [ClusterStack](https://github.com/pivotal/kpack/blob/main/docs/stack.md). +- _store.name_: The name of the [ClusterStore](https://github.com/pivotal/kpack/blob/main/docs/store.md) resource in [Kubernetes](https://kubernetes.io/). +- _store.kind_: The type as defined in [Kubernetes](https://kubernetes.io/). This will always be [ClusterStore](https://github.com/pivotal/kpack/blob/main/docs/store.md). +- _order_: The [builder order](https://buildpacks.io/docs/reference/builder-config/). + +The [ClusterBuilder](https://github.com/pivotal/kpack/blob/main/docs/builders.md#cluster-builders) resource is almost identical to a [Builder](https://github.com/pivotal/kpack/blob/main/docs/builders.md) but it is a cluster scoped resource that can be referenced by an [Image](https://github.com/pivotal/kpack/blob/main/docs/image.md) in any namespace. + +## Build + +Custom resource responsible for scheduling and running a single build. + +An example of a [Build](https://github.com/pivotal/kpack/blob/main/docs/build.md) definition is + +```yaml +apiVersion: kpack.io/v1alpha2 +kind: Build +metadata: + name: sample-build +spec: + tags: + -sample/image + builder: + image: foo.com/sample/builder + projectDescriptorPath: path/to/project.toml + source: + git: + url: https://github.com/my-account/sample-app.git + revision: main +``` + +The _spec_ key is used to define the desired state of the [Build](https://github.com/pivotal/kpack/blob/main/docs/build.md) + +- _tags_: A list of tags to build. At least one tag is required. +- _builder.image_: This is the tag to the [Cloud Native Buildpacks builder image](https://buildpacks.io/docs/concepts/components/builder/) to use in the build. +- _source_: The source location that will be the input to the build. +- _projectDescriptorPath_: Path to the [project descriptor file](https://buildpacks.io/docs/reference/config/project-descriptor/) relative to source root dir or subPath if set. + +## Image + +Provides a configuration to build and maintain an OCI image utilizing [CNB](https://buildpacks.io/). + +An example of an [Image](https://github.com/pivotal/kpack/blob/main/docs/image.md) definition is as follows + +```yaml +apiVersion: kpack.io/v1alpha2 +kind: Image +metadata: + name: my-app-image + namespace: default +spec: + tag: foo.com/my-app-repo/my-app-image + builder: + name: my-builder + kind: Builder + source: + git: + url: https://github.com/my-account/sample-app.git + revision: 82cb521d636b282340378d80a6307a08e3d4a4c4 +``` + +The _spec_ key is used to define the desired state of the [Image](https://github.com/pivotal/kpack/blob/main/docs/image.md) + +- _tag_: The image tag. +- _builder_: Configuration of the [builder](https://github.com/pivotal/kpack/blob/main/docs/builders.md) resource the image builds will use. +- source: The source code that will be monitored/built into images. + +# Contributors + +Contributions to [kpack](https://github.com/pivotal/kpack/) during the period 2022-2019 can be summarized as follow + +```mermaid +pie showData + title Pull Requests Open or Closed + "VMWare or Pivotal" : 438 + "Others" : 37 +``` + +# Migration + +### Repositories + +The suggested strategy for migrating [kpack's](https://github.com/pivotal/kpack/) git repositories to the [CNB](https://buildpacks.io/) is to use the [transfer repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository#transferring-a-repository-owned-by-your-organization) git feature. + +The following table shows the candidates repositories to be transferred + +| Origin Repo | Description | Owner | Destination Repo | Owner | +| --- | --- | --- | --- | --- | +| [https://github.com/pivotal/kpack](https://github.com/pivotal/kpack) | kpack source code | Pivotal | [https://github.com/buildpacks-community/kpack](https://github.com/buildpacks/kpack) | [CNB Technical Oversight Committee](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md#technical-oversight-committee) | +| [https://github.com/vmware-tanzu/kpack-cli](https://github.com/vmware-tanzu/kpack-cli) | kpack CLI | VMware | [https://github.com/buildpacks-community/kpack-cli](https://github.com/buildpacks/kpack-cli) | [CNB Technical Oversight Committee](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md#technical-oversight-committee) | +| [https://github.com/vmware-tanzu/homebrew-kpack-cli](https://github.com/vmware-tanzu/homebrew-kpack-cli) | Homebrew tap for the kpack CLI | VMware | [https://github.com/buildpacks-community/homebrew-kpack-cli](https://github.com/buildpacks/homebrew-kpack-cli) | [CNB Technical Oversight Committee](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md#technical-oversight-committee) | + +For each repository + +- The owner or admin user must follow the steps describe in github [documentation](https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository#transferring-a-repository-owned-by-your-organization) and transfer the repository to the organization [Cloud Native Buildpacks](https://github.com/buildpacks) +- A member of the [TOC team](https://github.com/orgs/buildpacks/teams/toc/members) in [CNB](https://buildpacks.io/) must accept the donation of the repository. The name of the destination repository will be the one described in the table above. + +### CI / CD Pipelines + +[kpack's](https://github.com/pivotal/kpack/) CI/CD pipelines were rebuilt to use [github actions](https://github.com/pivotal/kpack/tree/main/.github). +In order for [kpack's](https://github.com/pivotal/kpack/) to run windows acceptance tests it requires a kubernetes cluster with windows nodes. The hardware requirements are specify in the following section. + +##### Hardware requirements + +The minimal hardware requirements to request to CNCF to recreate the CI/CD pipelines are: + +###### Kubernetes clusters + +**Build cluster** + +- Linux nodes + - 1 amd64 node / 2 CPU / 8GB memory / 50GB ephemeral disk storage +- Windows nodes + - 1 amd64 node / 4 CPU / 16GB memory / 100GB ephemeral disk storage +- At least 100 GB of storage in a public OCI registry + +### Documentation + +[Kpack](https://github.com/pivotal/kpack/) documentation is currently hosted in the base code [repository](https://github.com/pivotal/kpack/tree/main/docs), after migrating to [CNB](https://buildpacks.io/) the documentation will be published into the Cloud Native Buildpack [site](https://buildpacks.io/). + +[CNB](https://buildpacks.io/) already mentioned [kpack](https://github.com/pivotal/kpack/) in their documentation, specifically, in the tools section. The proposal is: + +- Create a new folder name **kpack** inside the [tool](https://github.com/buildpacks/docs/tree/main/content/docs/tools) section in the docs repository +- Copy kpack's [documentation](https://github.com/pivotal/kpack/tree/main/docs) into this new created folder +- Update the references and all the required elements to format the documentation according to [CNB](https://buildpacks.io/) site + +### Governance + +#### Team roles + +Based on the [CNB governance policy](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md) and the fact that [kpack](https://github.com/pivotal/kpack/) is a [platform](https://buildpacks.io/docs/concepts/components/platform/) implementation of [Cloud Native Buildpacks](https://buildpacks.io/), it will be added under the responsibility of the [CNB Platform Team](https://github.com/buildpacks/community/blob/main/TEAMS.md#Platform-Team). + +How do migrate roles and responsibilities into the CNB governance process? + +Currently, the [CNB Platform Team](https://github.com/buildpacks/community/blob/main/TEAMS.md#Platform-Team) already has a **team lead** assigned and, by definition, each team can have only one **team lead**. In order to provide the current [kpack](https://github.com/pivotal/kpack/) team with the same accountability for the migrated repositories the proposal is to follow the guidelines describe on the [Component Maintainer Role RFC](https://github.com/buildpacks/rfcs/pull/234) + +The [kpack's](https://github.com/pivotal/kpack/) maintainers that will be nominated as **component maintainer** in CNB are: + +| Name | Github account | Organization | +|------------------|----------------------------------------------------|--------------| +| Matthew McNew | [@matthewmcnew](https://github.com/matthewmcnew) | VMware | +| Tom Kennedy | [@tomkennedy513](https://github.com/tomkennedy513) | VMware | +| Daniel Chen | [@chenbh](https://github.com/chenbh) | VMware | +| Juan Bustamante | [@jjbustamante](https://github.com/jjbustamante) | VMware | + +Also, those members are willing to become more involved with CNB projects and become **Platform maintainers** in the near future. + +Outside VMware, the following contributors manifested their desired to become [kpack's](https://github.com/pivotal/kpack/) **component maintainer**. + +| Name | Github account | Organization | +|-----------------|--------------------------------------------------|--------------| +| Sambhav Kothari | [@samj1912](https://github.com/samj1912) | Bloomberg | +| Aidan Delaney | [@AidanDelaney](https://github.com/AidanDelaney) | Bloomberg | + +#### RFC process + +Once the migration is completed, [kpack](https://github.com/pivotal/kpack/) will follow the [RFC process](https://github.com/buildpacks/rfcs) and [RFC template](https://github.com/buildpacks/rfcs/blob/main/0000-template.md) stablished in CNB project for any new RFC created in the project. + +##### Existing RFC + +- **Open**: Currently there are less that 10 [open RFCs](https://github.com/pivotal/kpack/pulls?q=is%3Apr+label%3ARFC+is%3Aopen) (some of them opened 2 years ago) in [kpack](https://github.com/pivotal/kpack/) repository. + - The proposal is to suggest the [kpack](https://github.com/pivotal/kpack/) maintainers to: + - Triage those RFCs an update their status before the donation. + - Co-ordinate the announcement of the donation to the RFCs authors and explain them the strategy after the migration (next section) + - After the donation, any open RFCs in [kpack](https://github.com/pivotal/kpack/) repository should be closed + - The RFC author should create a new RFC in the CNB RFC [repository](https://github.com/buildpacks/rfcs) and follow the CNB [RFC process](https://github.com/buildpacks/rfcs) + +- **Closed**: For historical purpose, we will keep those RFC in the repository. + +#### Slack channel + +The proposals are: + - `keep` the [kpack](https://github.com/pivotal/kpack/) slack instance from the [Kubernetes slack instance](https://kubernetes.slack.com/channels/kpack), as [kpack](https://github.com/pivotal/kpack/) is a Kubernetes native application most of their users already use [Kubernetes slack instance](https://kubernetes.slack.com/channels/kpack) for communication. + - `create` a new channel in the [CNCF slack instance](https://slack.cncf.io/), this will bring the two communities (kpack and CNB) together + +[kpack](https://github.com/pivotal/kpack/) maintainers should include the notification of the new channel in the announcement of the donation. + +[Platform maintainers](https://github.com/buildpacks/community/blob/main/TEAMS.md#maintainers-1) will have to request or create the new slack channel with the following name: **buildpacks-kpack** (which will be defined as the preferred channel to be used). + +# Risks + +- So far the main company behind [kpack](https://github.com/pivotal/kpack/) is [VMware](https://www.vmware.com/), a reduction in the investment from [VMware](https://www.vmware.com/) would create a problem and the CNB project would have to either sunset [kpack](https://github.com/pivotal/kpack/) or find investment from the community. +- It's not clear how to handle the budget required to finance the infrastructure to rebuild the CI/CD pipelines on CNCF CNB infrastructure. +- Evaluate any legal requirement from [CNCF](https://www.cncf.io) that must be fulfilled before accepting the project into the [CNB](https://buildpacks.io/) ecosystem. + +# Drawbacks + +Why should we _not_ do this? + +- If the [CNB](https://buildpacks.io/) team expects to implement a different kind of integration with [Kubernetes](https://kubernetes.io/), then accepting the donation of [kpack](https://github.com/pivotal/kpack/) could conflict with that strategy. +- Another component to maintain which requires additional context and expertise in [Kubernetes](https://kubernetes.io/). + +# Alternatives + +- What other designs have been considered? + - [VMware](https://www.vmware.com/) could continue to control the project, but it doesn't help on increase the adoption because it remains as a single-vendor driven project + - [VMware](https://www.vmware.com/) could donate [kpack](https://github.com/pivotal/kpack/) to the [Continuous Delivery Foundation](https://cd.foundation/), but [CNB](https://buildpacks.io/) presents a natural home for [kpack](https://github.com/pivotal/kpack/) (it is an implementation of the platform specification) + - [VMware](https://www.vmware.com/) could create a new [CNCF](https://www.cncf.io/) project and move all [kpack](https://github.com/pivotal/kpack/) resources to it, but in this case it would need to undergo as a sandbox project for example. + +- Why is this proposal the best? + +[kpack](https://github.com/pivotal/kpack/) is a mature Kubernetes-native tool that leverages buildpacks and is used in production environments. The project's maintainers and contributors possess valuable technical and user context, derived from developing [kpack](https://github.com/pivotal/kpack/) and integrating feedback from users utilizing [CNB](https://buildpacks.io/) concepts when presented as part of Kubernetes resources. + +- What is the impact of not doing this? + +The [CNB](https://buildpacks.io/) community would have to develop from scratch any kind of integration with the Cloud Native Ecosystem to satisfy the project goals. + +**Prior Art** + +- Guidelines for accepting component-level contributions [RFC #143](https://github.com/buildpacks/rfcs/pull/143) +- Component Maintainer Role [RFC #234](https://github.com/buildpacks/rfcs/pull/234) +- Proposal to move CNCF slack [RFC #198](https://github.com/buildpacks/rfcs/pull/198) + +# Unresolved Questions + +See the risks section + +# Spec. Changes (OPTIONAL) + +None diff --git a/text/0122-2023H2-roadmap.md b/text/0122-2023H2-roadmap.md new file mode 100644 index 000000000..972ff9eef --- /dev/null +++ b/text/0122-2023H2-roadmap.md @@ -0,0 +1,177 @@ +# Meta +[meta]: #meta +- Name: 2023H2 Roadmap +- Start Date: 2023-06-07 +- Author(s): hone +- Status: Approved +- RFC Pull Request: [rfcs#286](https://github.com/buildpacks/rfcs/pull/286) +- CNB Pull Request: +- CNB Issue: N/A +- Supersedes: [RFC 0118](https://github.com/buildpacks/rfcs/blob/main/text/0118-2023H1-roadmap.md) + +# Summary +[summary]: #summary + +This RFC details the second half of the 2023 Roadmap leading up to KubeCon NA. + +# Definitions +[definitions]: #definitions + +## Readmap Status Items +* Finished - The item has already been completed and there is no need to put it on the current roadmap. +* Continue - More work is needed and it will be continued as part of the current roadmap. +* Defer - This item will be parked for now and doesn't make the cut for the current roadmap. + +# Motivation +[motivation]: #motivation + +With KubeCon NA coming up around the corner, it's time to plan out what we want to achieve as a project. It's also an opportunity to review items from the H1 roadmap. + +# What it is +[what-it-is]: #what-it-is + +## 2023H2 Roadmap + +This roadmap is going to be split into two sections covering the first the H1 Roadmap and status, as well as new items we want to tackle leading up to KubeCon NA. + +### Items from [2023H1] +[items-from-2023h1]: #items-from-2023h1 + +As a project, we've made good progress on our H1 roadmap, but still need some more time to get some of them across the finish line. + +#### Release Base Image Extension +* Owner: @natalieparellano +* Status: Continue +* Links: [RFC](https://github.com/buildpacks/rfcs/blob/main/text/0105-dockerfiles.md) + +This is close and will be released as part of Platform `0.12`, Buildpack `0.10`, lifecycle `0.17.0`, and pack `0.30.0`. + +#### Remove Stacks & Mixins +* Owner: @jkutner +* Status: Continue +* Links: [RFC](https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md) + +This is close and will be released as part of Platform `0.12`, Buildpack `0.10`, lifecycle `0.17.0`, and pack `0.30.0`. + +#### Execution Environments RFC +* Owner: @hone +* Status: Continue +* Links: [RFC](https://github.com/buildpacks/rfcs/pull/274) + +The RFC is written, but feedback needs to be incoporated before re-opening for review. + +#### Project Health +* Owner: @samj1912 +* Status: Finished +* Links: [Buildpacks Community Organization RFC](https://github.com/buildpacks/rfcs/pull/273) + +The RFC has been merged and the buildpacks community GitHub org have been created. + +#### Pack Test Cleaning/Optimizations +* Owner: @dfreilich +* Status: Defer +* Links: [Pack Pull Request](https://github.com/buildpacks/pack/pull/1498) + +This item has been deferred for now, but we will work with anyone who wants to push this forward. + +### New Items + +#### Community Engagement Health Checks +* Owner: @microwavables (Team Lead sponsor @jkutner) +* Links: [VMware Tanzu Community Engagement Health Checks](https://github.com/vmware-tanzu/community-engagement/blob/main/HEALTHCHECKS.md) + +The project is lucky to have a Community Manager! This is one of the projects proposed by @microwavables to set a base line to measure how we're doing as a community. + +#### RFC for Buildpack Author Observability +* Owner: @joshwlewis (Team Lead sponsor @hone) +* Links: TBD + +Currently, Buildpack Authors have little to no tools around visibility with their buildpacks as they run on a platform, including `pack`. In some of the Heroku v2 buildpacks, they implemented logging that could be handed off to the platform by running `bin/report`. This work stream is about standardizing output for both successful AND failed builds that Buildpack Authors can use to instrument their buildpack. + +#### Private Registry Mirrors +* Owner: @jabrown85 +* Links: [RFC](https://github.com/buildpacks/rfcs/pull/285) + +A platform operator can configure registry mirrors that lifecycle could use without needing for the manifest to have to point to it. This will allow a platform to reduce the risk of service operations from external registry sources and reduce public network bandwidth usage. The resulting image when taken off platform will also function without needing access to the registry mirror. + +#### kpack Donation +* Owner: @jjbustamante (Team Lead sponsor @samj1912) +* Links: [RFC](https://github.com/buildpacks/rfcs/pull/235) + +`kpack` is being proposed to be donated as an open source project in the Cloud Native Buildpacks' new Community Organization as a vendor neutral staging ground. This will give the project space to grow the project contributor base from multiple vendors. While work has started on this in H1, this item represents our commitment as a project to see this through and set this project up for success under the CNB governance umbrella. + +#### Cosign Integration / OCI Image Manifest v1.1 +* Owner: @natalieparellano +* Links: [Cosign Integration RFC](https://github.com/buildpacks/rfcs/pull/195), [SBOM layer RFC](https://github.com/buildpacks/rfcs/pull/278), + +While CNBs support SBOMs today, they were designed a few years ago and tooling around them have been evolving. This work stream is about making CNBs integrate better with tools like [cosign's SBOM spec](https://github.com/sigstore/cosign/blob/main/specs/SBOM_SPEC.md) and the upcoming [OCI References](https://github.com/opencontainers/image-spec/issues/827) feature in OCI Image Manifest 1.1. + +#### Pack OCI Manifest Support +* Owner: @jjbustamante (Team Lead sponsor @hone) +* Links: [RFC](https://github.com/buildpacks/rfcs/pull/283) + +Multi-arch support has been a highly request feature with the growing popularity of the ARM architecture. In order to better support this with Buildpacks, the first step will be to able to use manifest lists to provide a single URI for Buildpacks that support multiple architectures. + +#### Export to OCI Layout +* Owner: @jjbustamante (Team Lead sponsor @natalieparellano) +* Links: [RFC](https://github.com/buildpacks/rfcs/blob/main/text/0119-export-to-oci.md) + +The RFC has been merged and the implementation is expected to be released in experimental mode on pack *v0.30.0* + +# How it Works +[how-it-works]: #how-it-works + +See [What it is](#what-it-is) for the details. We'll be following the same process from the [2023H1 Roadmap RFC](https://github.com/buildpacks/rfcs/blob/main/text/0118-2023H1-roadmap.md) if approved. + +# Migration +[migration]: #migration + +N/A + +# Drawbacks +[drawbacks]: #drawbacks + +- Agreeing to more work, while we still need to finish 2023H1 items. + +# Alternatives +[alternatives]: #alternatives + +- Do Nothing and just hunker down on our existing items. + +# Prior Art +[prior-art]: #prior-art + +See [Prior Art from 2023H1 Roadmap RFC](https://github.com/buildpacks/rfcs/blob/main/text/0118-2023H1-roadmap.md#prior-art). + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to be resolved before this gets merged? +- What parts of the design do you expect to be resolved through implementation of the feature? +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +N/A + +# History +[history]: #history + + diff --git a/text/0123-flatten-feature.md b/text/0123-flatten-feature.md new file mode 100644 index 000000000..4f0f31013 --- /dev/null +++ b/text/0123-flatten-feature.md @@ -0,0 +1,222 @@ +# Meta +[meta]: #meta +- Name: Flatten builders +- Start Date: 2023-07-13 +- Author(s): @jjbustamante, @dlion +- Status: Approved +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +We propose to add new capabilities to the Pack tool that allow end-users to reduce the number of Buildpack's layers in a Builder by flattening some Buildpacks according to their requirements. + +This RFC mainly focus on applying this strategy to Builders only. + +# Definitions +[definitions]: #definitions + +- Buildpack: A buildpack is a set of executables that inspects your app source code and creates a plan to build and run your application. +- Builder: A builder is an image that contains all the components necessary to execute a build. A builder image is created by taking a build image and adding a lifecycle, buildpacks, and files that configure aspects of the build including the buildpack detection order and the location(s) of the run image +- Component Buildpack: A component buildpack is a buildpack containing `/bin/detect` and `/bin/build` executables. Component buildpacks implement the [Buildpack Interface](https://github.com/buildpacks/spec/blob/main/buildpack.md). +- Composite Buildpack: A composite buildpack is a buildpack containing an order definition in `buildpack.toml`. Composite buildpacks do not contain `/bin/detect` or `/bin/build` executables. They MUST be [resolvable](https://github.com/buildpacks/spec/blob/main/buildpack.md#order-resolution) into a collection of component buildpacks. +- Buildpackage: A buildpackage is a [distributable](https://github.com/buildpacks/spec/blob/main/distribution.md) artifact that contains a buildpack. + +# Motivation +[motivation]: #motivation + +- Why should we do this? + +There is a limit in the number of layer an image can have, at least on Docker, which is *127*, this feature has being request by the community, issue [#1595](https://github.com/buildpacks/pack/issues/1595), as a workaround to solve error thrown by docker when the limit is reached + +- What use cases does it support? + +Buildpacks provider like Paketo have Composite Buildpacks with several layers, when they pull many of those together into a Builder, hitting the layer limit for a container image happens very often. A feature for the Builder author to group the Buildpacks by any attribute will allow them to squash those groups into one layer and reduce their total number of layers, avoiding the layer limit. + +- What is the expected outcome? + +When Builder Authors execute the command: + +`pack builder create ... ` + +The final Builder (A) SHOULD contain layers blobs with more than *one* buildpack according to the configuration provided by the user. If we compare an artifact (B) created *without* `` then: + +$numberOfBuildpackLayers(A) \leq numberOfBuildpackLayers(B)$ + +A and B MUST be otherwise interchangeable, only differing by their number of layers. + + +# What it is +[what-it-is]: #what-it-is + +The proposal is to include a new experimental flag to the following command on Pack: + +- `pack builder create` + +The new flag will move from experimental status to supported status when maintainers deem it appropriate. + +The new flag to be included is: + +- `--flatten=` will flatten the Buildpacks specified after the `flatten` flag into a single layer. Can be used more than once, with each use resulting in a single layer. + +We also need to define how a Platform implementor needs to consume a flattened Builder. + +- When a Platform consumes a Builder, they will need to inspect each Buildpack layer blob and determine if the blob contains more than one Buildpack, in such as case, they will need to process those Buildpacks correctly. + + +# How it Works +[how-it-works]: #how-it-works + +Let's say we have a Composite Buildpack (CB1) with the following dependency tree: +```mermaid +flowchart TD + A[CB1] + A --> B[G1] + A --> C[G2] + B --> BPA[BP1] + B --> BPFOO[BP2] + B --> BPC[BP4] + C --> BPD[BP1] + C --> BPBAR[BP3] + C --> BPE[BP4] +``` + +Until now, when a Buildpack like this is being shipped into a Builder every individual Buildpack is being saved in one layer, as a result we will have: + +$$ +layer_1 = [CB_1] \\ +layer_2 = [G_1] \\ +layer_3 = [BP_1] \\ +layer_4 = [BP_2] \\ +layer_5 = [BP_4] \\ +layer_6 = [G_2] \\ +layer_7 = [BP_3] \\ +total = \text{7 layers} +$$ + +Noticed that duplicated Buildpacks are cleaned up. + +We can use the new `flatten` flag to reduce the number of Builder layers used by the buildpacks in different ways. + +* `--flatten=` i.e. `--flatten= --flatten=`: + Will group the given Buildpacks into one layer and keep the other ones as single layers Buildpacks, the result will be: + +```mermaid +classDiagram + class Layer1 { + CB1 + } + class Layer2 { + G1 + } + class Layer3 { + BP1 + BP2 + } + class Layer4 { + G2 + } + class Layer5 { + BP3 + BP4 + } +``` + + +$$ +total = \text{5 layers} +$$ + +--- + + +# Migration +[migration]: #migration + + +The current [distribution spec](https://github.com/buildpacks/spec/blob/main/distribution.md#buildpackage) defines: + +``` +Each buildpack layer blob MUST contain a single buildpack at the following file path: + +/cnb/buildpacks/// +``` + +A Builder flattened with this new feature would not be consumable by older platform implementations because they are not expecting to find more than one buildpack on a blob layer. + + + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +It could create artifacts that are not consumable by older platforms. + + +# Alternatives +[alternatives]: #alternatives + +- What other designs have been considered? + +Some other alternatives mentioned are: squashing by the buildpack size or squashing a CNB Builder when the number of layers is reaching the limit, but those ideas, do not provide the freedom to the buildpacks authors to decide which buildpacks to flatten. + + +- Why is this proposal the best? + +Not sure if it is the best, but a way to solve the `layer limit error` is to optimize the uses of the layer in a Builder. + +- What is the impact of not doing this? + +Builder Authors and Platform Operators will keep seeing the layer limit error. + +# Prior Art +[prior-art]: #prior-art + +Discuss prior art, both the good and bad. + +--- + + + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +No spec changes at this time + + + +# History +[history]: #history + +% diff --git a/text/0124-pack-manifest-list-commands.md b/text/0124-pack-manifest-list-commands.md new file mode 100644 index 000000000..639c69f8b --- /dev/null +++ b/text/0124-pack-manifest-list-commands.md @@ -0,0 +1,460 @@ +# Meta +[meta]: #meta +- Name: Manifest List Commands for Pack +- Start Date: 2023-04-19 +- Author(s): [Juan Bustamante](https://github.com/jjbustamante) +- Status: Approved +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +The problem for adding support for multi-arch buildpacks can be divided into two parts: +- Support buildpack authors to **migrate their existing buildpacks** to support multi-arch . +- Support buildpack authors to **create new buildpacks and builders** that handle multi-arch from the beginning. + +This RFC proposes to create a new set of CRUD commands in pack to handle manifest lists, which will be used to support the first part of the problem. + +# Definitions +[definitions]: #definitions + +- Image Manifest: The image manifest provides a configuration and set of layers for a single container image for a specific architecture and operating system. See [spec](https://github.com/opencontainers/image-spec/blob/main/manifest.md) +- Image Index: The image index is a higher-level manifest which points to specific image manifests, ideal for one or more platforms. See [spec](https://github.com/opencontainers/image-spec/blob/main/image-index.md) + +# Motivation +[motivation]: #motivation + +- Why should we do this? + +The uses of ARM architecture in the cloud and edge computing has been growing rapidly. The CNCF community has been also growing in the last years, and there is a need to support multi-arch for all the projects. The buildpacks community is not an exception, issues like: +- [It would be nice to support easily creating a manifest list packages and builders](https://github.com/buildpacks/pack/issues/1460) +- [Provide a way to specify desired platform when creating packages and builders](https://github.com/buildpacks/pack/issues/1459) +- [Multi arch image build support](https://github.com/buildpacks/pack/issues/1570) + +Or the conversations around this topic in our [slack channel](https://cloud-native.slack.com/archives/C032LNSMY0P), even the [talk at Kubecon NA 2022](https://www.youtube.com/watch?v=Sdr5axlOnDI&list=PLj6h78yzYM2O5aNpRM71NQyx3WUe1xpTn&index=76) demonstrate the interest from the community in this feature. + +- What use cases does it support? + +Currently, buildpack authors can build and package their buildpacks for different OS and Architectures, but when they distribute them the URI for a buildpack can’t disambiguate, they need to use different tags to differentiate between them. This makes harder for users to consume those Buildpacks. +The solution is to share a single URI for all the different OS and Architectures, and the way to do that is using a manifest list. + +Adding commands to support the operations to handle the manifest list will allow buildpack authors to migrate their existing buildpacks to support multi-arch, without afecting their current existing process. + +- What is the expected outcome? + +The expected outcome is to have a set of commands in pack to handle manifest lists, for example: +- A command to create a manifest list from a set of images +- A command to push the manifest list to a registry +- A command to update or delete a manifest list + +# What it is +[what-it-is]: #what-it-is + +The proposal is to add a new _experimental_ command `pack manifest` and different subcommands. The `pack manifest` commands will initially be gated behind `pack config experimental`. The `pack manifest` commands will move from experimental status to supported status when maintainers deem it appropriate. +- `pack manifest create` will create a local manifest list for annotating and pushing to a registry +- `pack manifest annotate` will add additional information like os, arch or variant to an existing local manifest list +- `pack manifest add` will add an image to an existing manifest list +- `pack manifest remove` will delete a manifest list from local storage +- `pack manifest rm` will remove an image from a manifest list in the local storage +- `pack manifest push` will push a manifest list to a registry +- `pack manifest inspect` will show the manifest information stored in local storage + +Our target user affected by the feature is: **Buildpack Authors**. Let's see some examples of how this feature will work. + +Currently, if we check [sample-packages](https://hub.docker.com/r/cnbs/sample-package/tags) at dockerhub we will notice that we have a composed buildpack called `hello-universe` and we offer two tags to support different architectures: +- `cnbs/sample-package:hello-universe` for linux and +- `cnbs/sample-package:hello-universe-windows`. + + +Let's suppose our linux version is called `cnbs/sample-package:hello-universe-linux` to keep the same naming convention, but we will keep it as it is for simplicity. If we want to distribute the `hello-universe` buildpack for any **architecture/os/variant** combination we need to use a tool outside the CNB ecosystem to create a manifest list. With the proposed experimental commands on pack we can do: + +```bash +$ pack manifest create cnbs/sample-package:hello-multiarch-universe \ + cnbs/sample-package:hello-universe \ + + cnbs/sample-package:hello-universe-windows +``` + +By default, the command will create a manifest list in the local storage using the docker media types [Version 2 schema 2](https://docs.docker.com/registry/spec/manifest-v2-2/) with a content similar to: + +```json +{ + "schemaVersion": 2, + "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", + "manifests": [ + { + "mediaType": "application/vnd.docker.distribution.manifest.v2+json", + "size": 226083, + "digest": "sha256: 87a832fd6a8d6995d336c740eb6f3da015401a6e564fcbe95ee1bf37557a8225", + "platform": { + "os": "linux", + "architecture": "" + } + }, + { + "mediaType": "application/vnd.docker.distribution.manifest.v2+json", + "size": 226083, + "digest": "sha256:670d62fbee841d256a706801a03be9c84d37fc2cd6ef7538a7af9985c3d2ed8b", + "platform": { + "os": "windows", + "architecture": "" + } + } + ] +} +``` +The idea to save the manifest list locally is to allow the user to update the manifest before pushing it to a registry, + +in this case, we need to define the **architecture** field because it is empty in our examples. + +We can use the `pack manifest annotate` command to add the architecture information: + +```bash +$ pack manifest annotate --arch amd64 cnbs/sample-package:hello-multiarch-universe cnbs/sample-package:hello-universe +$ pack manifest annotate --arch amd64 cnbs/sample-package:hello-multiarch-universe cnbs/sample-package:hello-universe-windows +``` + +After executing these commands, our local manifest list will be updated as follows: + +```json +{ + "schemaVersion": 2, + "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", + "manifests": [ + { + "mediaType": "application/vnd.docker.distribution.manifest.v2+json", + "size": 940, + "digest": "sha256:87a832fd6a8d6995d336c740eb6f3da015401a6e564fcbe95ee1bf37557a8225", + "platform": { + "architecture": "amd64", + "os": "linux" + } + }, + { + "mediaType": "application/vnd.docker.distribution.manifest.v2+json", + "size": 1148, + "digest": "sha256:670d62fbee841d256a706801a03be9c84d37fc2cd6ef7538a7af9985c3d2ed8b", + "platform": { + "architecture": "amd64", + "os": "windows" + } + } + ] +} +``` + +Finally, our manifest list is ready to be pushed to a registry, and we can use the `pack manifest push` command to do it: + +```bash +$ pack manifest push cnbs/sample-package:hello-multiarch-universe +``` +And our manifest list should be published at [dockerhub](https://hub.docker.com/r/cnbs/sample-package/tags) as `cnbs/sample-package:hello-multiarch-universe`, asuming that we have the proper credentials to push the image. + + + +# How it Works +[how-it-works]: #how-it-works + +The proposal is to implement an abstraction of an OCI *Image Index* and expose it to users through `pack manifest` commands. + +## Image Index Abstraction + +A new high level abstraction to represent an OCI Image Index is proposed, similar to the [Image](https://github.com/buildpacks/imgutil/blob/main/image.go) interface exposed in *imgutil* repository, + +we proposed a new *ManifestList* interface to expose the behavior of an OCI Image Index. + +```mermaid +classDiagram + class ManifestList { + <> + +Add(repoName string) error + +Remove(repoName string) error + +Delete(additionalNames []string) error + +AnnotateManifest(manifestName string, opts AnnotateFields) error + +Save() error + } + + class remote_ManifestList { + +NewManifestList(repoName string, keychain authn.Keychain) ManifestList + + } + + class local_ManifestList { + +NewManifestList(repoName string, path string) ManifestList + } + + + class AnnotateFields { + +String Architecture + +String OS + +String Variant + } + + ManifestList <|-- remote_ManifestList + ManifestList <|-- local_ManifestList + +``` + +Two implementations: *remote* and *local* are proposed, *remote* will take care of implementing the interaction with an OCI registry and *local* will deal with the local storage operations. + +### Component Diagram + +Using a [C4 component diagram](https://c4model.com/#ComponentDiagram), we can define the high level interaction on pack. This design follows the same pattern for each command already implemented. + +![](https://hackmd.io/_uploads/B1PpJ-fKh.png) + +- *Image Factory*: is responsible for instantiate the *Image Index* abstraction based on the configuration require, it could be a *remote* or a *local* implementation +- *Image Index*: is the abstraction defined which exposes the operation methods we want to offer to users + - As we can see, depending on the implementation it will interact with the file system or with a remote registry + +### Considerations + +#### When a user wants to create a manifest list using a manifest outside the user's repo. + + +As a pack user I want to create a manifest list `foo/my-manifest:my-tag` using a manifest outside my repository `foo` for example `other/external-manifest:latest`. + +The user invokes a command similar to: `pack manifest create foo/my-manifest:my-tag other/external-manifest:latest` + +In this case, pack will need to *copy* the external image `other/external-manifest:latest` into `foo` repository `foo/external-manifest:latest` and then uses this reference to create the manifest list. In such as case `pack` should: + - *warn* the user about this operation, for example with a message + - *ask the user for confirmation* before executing the operation + +#### When a user wants to create a manifest list referencing a manifest list + +As a pack user I want to create a manifest list using a reference to another manifest list + +The user invokes a command similar to: `pack manifest create foo/my-manifest:my-tag foo/another-manifest:latest` + +In this case, pack should: + - add into the **manifests** array of objects a reference to the manifest index using the media-type `application/vnd.oci.image.index.v1+json` (nested index) + +### Commands details + +#### Create a Manifest List + +Pack will create a manifest a local manifest, it should handle the following scenarios: +- IF user references a manifest list that already exists in a registry: In this case, pack will save a local copy of the remote manifest list, this is useful for updating (adding, updating or deleting) images +- IF user references a manifest list that doesn't exist in a registry: pack will create a local representation of the manifest list that will only save on the remote registry if the user publish it + +```bash +manifest create generates a manifest list for a multi-arch image + +Usage: + pack manifest create [ ... ] [flags] + +Examples: +pack manifest create cnbs/sample-package:hello-multiarch-universe \ + cnbs/sample-package:hello-universe \ + cnbs/sample-package:hello-universe-windows + +Flags: + -f, --format string Format to save image index as ("OCI" or "V2S2") (default "v2s2") + --insecure Allow publishing to insecure registry + --publish Publish to registry + -r, --registry string Registry URL to publish the image index +``` + +#### Annotate (os/arch) a Manifest List + + +Sometimes a manifest list could reference an image that doesn't specify the *architecture*, for example, [check](https://hub.docker.com/r/cnbs/sample-package/tags) our sample buildpack +packages. The `annotate` command allows users to update those values before pushing the manifest list a registry + +```bash +manifest annotate modifies a manifest list (Image index) and update the platform information for an image included in the manifest list. + +Usage: + pack manifest annotate [OPTIONS] [flags] + +Examples: +pack manifest annotate cnbs/sample-package:hello-universe-multiarch \ cnbs/sample-package:hello-universe --arch amd64 + +Flags: + --arch string Set the architecture + --os string Set the operating system + --variant string Set the architecture variant +``` + +#### Add an image to a Manifest List + +When a manifest list exits locally, user can add a new image to the manifest list using this command + +```bash +manifest add modifies a manifest list (Image index) and add a new image to the list of manifests. + +Usage: + pack manifest add [OPTIONS] [flags] + +Examples: +pack manifest add cnbs/sample-package:hello-multiarch-universe \ + cnbs/sample-package:hello-universe-riscv-linux + +Flags: + --all add all of the contents to the local list (applies only if is an index) + --arch string Set the architecture + --os string Set the operating system + --variant string Set the architecture variant +``` + +#### Remove an image from a Manifest List + +In the opposite case, users can remove existing images from a manifest list + +```bash +Delete one or more manifest lists from local storage + +Usage: + pack manifest remove [manifest-list] [manifest-list...] [flags] + +Examples: +pack manifest delete cnbs/sample-package:hello-multiarch-universe + +Flags: + -h, --help Help for 'remove' +``` + +#### Remove a local Manifest List + +Sometimes users can just experiment with the feature locally and they want to discard all the local information created by pack. `rm` command just delete the *local* manifest list + +```bash +manifest remove will remove the specified image manifest if it is already referenced in the index + +Usage: + pack manifest rm [manifest-list] [manifest] [flags] + +Examples: +pack manifest rm cnbs/sample-package:hello-multiarch-universe \ + cnbs/sample-package:hello-universe-windows + +Flags: + -h, --help Help for 'rm' + +``` + +#### Push a Manifest List to a remote registry + +Once a manifest list is ready to be publishe into the registry, the `push` command can be used + +```bash +manifest push pushes a manifest list (Image index) to a registry. + +Usage: + pack manifest push [OPTIONS] [flags] + +Examples: +pack manifest push cnbs/sample-package:hello-multiarch-universe + +Flags: + -f, --format string Manifest list type (oci or v2s2) to use when pushing the list (default is v2s2) + --insecure Allow publishing to insecure registry + -p, --purge Delete the manifest list or image index from local storage if pushing succeeds +``` + +#### Inspect a Manifest List + +Finally, the `inspect` command will help users to view how their local manifest list looks like + +```bash +manifest inspect shows the manifest information stored in local storage + +Usage: + pack manifest inspect [flags] + +Examples: +pack manifest inspect cnbs/sample-builder:multiarch + +Flags: + -h, --help Help for 'inspect' + +``` + +One important concern for users is to inspect the content of a multi-arch builder or buildpack if they are accessible behind a manifest list. The proposal is to implement a `platform` flag for commands: + +- `pack builder inspect` +- `pack buildpack inspect` + +The `--platform` flag specifies the platform in the form os/arch[/variant][:osversion] (e.g. linux/amd64). By default it will reference the host values. + +The output of the commands should remain the same. + + +# Migration +[migration]: #migration + +This section should document breaks to public API and breaks in compatibility due to this RFC's proposed changes. In addition, it should document the proposed steps that one would need to take to work through these changes. Care should be give to include all applicable personas, such as platform developers, buildpack developers, buildpack users and consumers of buildpack images. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +We should decide to do not add this feature and users could use tools like `docker` or `podman` to create and handle their manifest list, however this is a poor user experience forcing users to use different tools to fulfill scenarios that are part of the business domain of pack. + +# Alternatives +[alternatives]: #alternatives + +- What other designs have been considered? + +We also have in mind to improve existing commands like `pack builder create` and `pack buildpack package` to support the creation of a manifest list without the need of a new command. However, this approach could not be suitable for buildpack authors who are maintaining existing buildpacks and they will need to change their current process to generate multi-arch images. + +- Why is this proposal the best? + +Because we will provide the tool to our end users to solve their problems without the need of using other tools. + +- What is the impact of not doing this? + +The impact of not doing this is that users will need to use other tools to create and handle their manifest list, which is a poor user experience. + +# Prior Art +[prior-art]: #prior-art + +These features are inspired in similar commands in other tools like: +- [docker](https://docs.docker.com/engine/reference/commandline/manifest/) +- [podman](https://docs.podman.io/en/v3.2.0/markdown/podman-manifest-create.1.html) + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to be resolved before this gets merged? +- What parts of the design do you expect to be resolved through implementation of the feature? +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes +Does this RFC entail any proposed changes to the core specifications or extensions? If so, please document changes here. +Examples of a spec. change might be new lifecycle flags, new `buildpack.toml` fields, new fields in the buildpackage label, etc. +This section is not intended to be binding, but as discussion of an RFC unfolds, if spec changes are necessary, they should be documented here. + +# History +[history]: #history + + diff --git a/text/0125-lifecycle-parallel-export.md b/text/0125-lifecycle-parallel-export.md new file mode 100644 index 000000000..0d193a8ab --- /dev/null +++ b/text/0125-lifecycle-parallel-export.md @@ -0,0 +1,213 @@ +# Meta +[meta]: #meta +- Name: Export App Image and Cache Image in Parallel +- Start Date: 2023-08-26 +- Author(s): ESWZY +- Status: Approved +- RFC Pull Request: [rfcs#291](https://github.com/buildpacks/rfcs/pull/291) +- CNB Pull Request: [lifecycle#1167](https://github.com/buildpacks/lifecycle/pull/1167) +- CNB Issue: N/A +- Supersedes: N/A + +# Summary +[summary]: #summary + +Export app image and cache image in parallel during export phase of lifecycle. In the original logic, it has to wait for the export of the app image to be completed before exporting the cache image. This will result in a period of idleness, and the network resources and I/O resources will not be fully utilized, resulting in a longer waiting time for the overall export, and also a longer overall build time. By parallelizing export phase, network and I/O resources can be used to the maximum, thereby saving time. + +# Definitions +[definitions]: #definitions + +* lifecycle: software that orchestrates a CNB build. +* Cache Image: The cache stored between builds - stored in a registry. + +# Motivation +[motivation]: #motivation + +In some scenario, the app image and the cache image need to be exported at the same time, but this process is serial in the lifecycle, which means that after the app image is exported, we have to wait for the cache image to be exported. But we don’t need to wait for the export of cache image, only after the app is exported, we can continue to next steps (distribution and deployment). + +So we can try to parallelize this step ([lifecycle#1167](https://github.com/buildpacks/lifecycle/pull/1167)) and compare it with serial exporting. After testing the build on some applications, this modification can shorten the export time. + +- Java (app image is 202.361MB, cache image is 157.525MB, with one same layer: 107.648MB): + - Before: total 18.34s, app 8.96s, cache 9.38s + - After: total 14.70s, app 11.42s, cache 13.93s + - app image layers: 0+1.103MB+15.153MB+107.648MB+49.953MB+0+0+28.502MB + - cache image layers: 9.411MB+40.465MB+107.648MB + +- Go (app image is 114.273MB, cache is 175.833MB, no same layer): + - Before: total 16.57s, app 5.92s, cache 10.65s + - After: total 12.02s, app 7.31s, cache 11.48s + - app image layers: 0+1MB+25.72MB+8.993MB+49.953MB+0+0+28.502MB + - cache image layers: 70.87MB+104.964MB + +# What it is +[what-it-is]: #what-it-is + +The proposal is to add a new capability to the lifecycle (enabled by configuration) to export app image and cache image to registry in parallel. + +The target personas affected by this change are: + + - **buildpack user**: they will experience higher disk I/O or network pressure if this feature is enabled by default. + - **Platform implementors**: they will choose parallel or serial export, to suit how the platform works. Serial export helps to get app image faster, while parallel export can complete the build process faster. + +This proposal, in addition to acceleration, has the greatest impact on the export time of app image. This will lead to resource competition, that is, when the app image and cache image are exported at the same time, the app image export time will become longer. For some users, they only care about the export time of the app image, but not the overall optimization effect, and enabling this capability will affect their performance. Therefore, this ability is optional. + +The flag name, refer to other boolean flag like `daemon`, `layout`, I think it can be named `parallel`. And then the usage is just like this: +``` +/cnb/lifecycle/exporter \ + [-analyzed ] \ + [-app ] \ + [-cache-dir ] \ + [-cache-image ] \ + [-daemon] \ # sets + [-extended ] \ + [-gid ] \ + [-group ] \ + [-launch-cache ] \ + [-launcher ] \ + [-launcher-sbom ] \ + [-layers ] \ + [-layout] \ # sets + [-layout-dir] \ # sets + [-log-level ] \ + [-parallel] \ # sets + [-process-type ] \ + [-project-metadata ] \ + [-report ] \ + [-run ] \ + [-uid ] \ + [...] +``` + +# How it Works +[how-it-works]: #how-it-works + +It will be done using goroutines. Goroutine is a lightweight multi-threading mechanism, which can avoid a lot of extra overhead caused by the multi-threaded parallel operation. This function encapsulates the export process of the app image and the cache image into two goroutines to execute in parallel. + +The working principle is shown in the following code (go-like pseudocode). Execute in parallel through two goroutines and wait for all export processes to finish. If parallel export is not enabled, the process is exactly the same as the original serial export. + +```go +func export() { + exporter := &lifecycle.Exporter{} + // ... + + var wg sync.WaitGroup + + wg.Add(1) + go func() { + defer wg.Done() + exporter.Export() + }() + + if !enableParallelExport { + wg.Wait() + } + + wg.Add(1) + go func() { + defer wg.Done() + exporter.Cache() + }() + + wg.Wait() + + // ... +} +``` + +## Examples + +For command line use, control this process through the `parallel` flag. + +### Export both app image and cache image + +By specifying environment variable `CNB_PARALLEL_EXPORT`, or pass a `-parallel` flag, images will be pushed to `cr1.example.com` and `cr2.example.com` simultaneously. + +```shell +> export CNB_PARALLEL_EXPORT=true +> /cnb/lifecycle/exporter -app cr1.example.com/foo:app -cache-image cr2.example.com/foo:cache + +# OR + +> /cnb/lifecycle/exporter -app cr1.example.com/foo:app -cache-image cr2.example.com/foo:cache -parallel +``` + +### Export app image only or export cache image only + +If export one image, the effect of this function is not very obvious. + +```shell +> export CNB_PARALLEL_EXPORT=true +> /cnb/lifecycle/exporter -app cr1.example.com/foo:app +[debug] Parsing inputs... +[warn] parallel export has been enabled, but it has not taken effect because cache image (-cache-image) has not been specified. + +# OR + +> /cnb/lifecycle/exporter -app cr1.example.com/foo:app -parallel +[debug] Parsing inputs... +[warn] parallel export has been enabled, but it has not taken effect because cache image (-cache-image) has not been specified. + +# EQUAL TO + +> /cnb/lifecycle/exporter -app cr1.example.com/foo:app +``` + +# Migration +[migration]: #migration + +We maybe need to add a new API option for buildpack users, to choose whether this feature should be enabled. + +# Drawbacks +[drawbacks]: #drawbacks + + - This will lead to resource competition, the app image export time will become longer. + +# Alternatives +[alternatives]: #alternatives + +N/A. + +# Prior Art +[prior-art]: #prior-art + +N/A. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- How to allow users to choose the export method? environment variable? Or a parameter of the creator? +- Does it also need to be specified in the pack tool? +- Should this feature be enabled by default? + +# Spec. Changes +[spec-changes]: #spec-changes + +This new feature will affect the API of [Create](https://buildpacks.io/docs/concepts/components/lifecycle/create/) and [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) phases, by adding the following fields. + +Back to API changes, we will add a new flag to control this. + +| Input | Environment Variable | DefaultValue | Description | +|--------------|-----------------------|--------------|----------------------------------------------| +| `` | `CNB_PARALLEL_EXPORT` | `false` | Export app image and cache image in parallel | + + +# History +[history]: #history + + \ No newline at end of file diff --git a/text/0126-creator-skip-sbom.md b/text/0126-creator-skip-sbom.md new file mode 100644 index 000000000..eebd2b1ad --- /dev/null +++ b/text/0126-creator-skip-sbom.md @@ -0,0 +1,131 @@ +# Meta +[meta]: #meta +- Name: Enable CNB_SKIP_SBOM IN /cnb/lifecycle/creator +- Start Date: (fill in today's date: 2023-10-17) +- Author(s): kritkasahni-google +- Status: Approved +- RFC Pull Request: +- CNB Pull Request: +- CNB Issue: +- Supersedes: N/A + +# Summary +[summary]: #summary + +Enable CNB_SKIP_SBOM IN /cnb/lifecycle/creator to skip restoring SBOM layer from previous app image. We support CNB_SKIP_LAYERS in analyzer which does the same thing and we should support the same in creator also. + +# Definitions +[definitions]: #definitions +* lifecycle: software that orchestrates a CNB build. +* creator: executes all the lifecycle phases one by one in order. +* analyzer: lifecycle phase that restores SBOM layer from previous app image. +* restorer: lifecycle phase that restores layers from cache. +* SBOM: a software bill of materials (SBOM) is a list of all the components that make up the app image. + +# Motivation +[motivation]: #motivation + +To skip restoring SBOM layer from previous image when platform executes lifecycle by calling /cnb/lifecycle/creator. Restoring SBOM layer from previous app image can cause degraded build latency but if buildpack logic does not rely on SBOM from previous app image then should be able to skip restoring it. + +# What it is +[what-it-is]: #what-it-is + +CNB_SKIP_LAYERS is used by /cnb/lifecycle/analyzer to skip restoring SBOM layer from previous app image. +Need a similar mechanism for /cnb/lifecyle/creator specifically to skip restoring only the SBOM layer. + +The target personas affected by this change are: + + - **buildpack user**: if buildpacks don't rely on reusing SBOM layer then buildpack user should ideally see improved build latency by skipping SBOM restoration but reusing other layers from previous app image. + - **Platform implementors**: they will choose to skip restoring SBOM by providing CNB_SKIP_SBOM to trigger /cnb/lifecycle/creator. + + +# How it Works +[how-it-works]: #how-it-works + +Similar to how CNB_SKIP_LAYERS is handled in analyzer whether SBOM needs to be [restored](https://github.com/buildpacks/lifecycle/blob/292aa492a72f4e180bb92d109a73ebf7c8a0451d/phase/analyzer.go#L38) or [not](https://github.com/buildpacks/lifecycle/blob/292aa492a72f4e180bb92d109a73ebf7c8a0451d/phase/analyzer.go#L30) today, CNB_SKIP_SBOM will be be handled in same way in analyzer. +At the platform level, it would be input same way as CNB_SKIP_LAYERS [here](https://github.com/buildpacks/lifecycle/blob/292aa492a72f4e180bb92d109a73ebf7c8a0451d/platform/defaults.go#L184) and [handled](https://github.com/buildpacks/lifecycle/blob/main/platform/lifecycle_inputs.go#L82) like:- + + +``` + var skipSBOM bool + if boolEnv(EnvSkipSBOM){ + skipSBOM = true + } +``` + +In the analyzer, + +``` +analyzer := &Analyzer{ + Logger: logger, + SBOMRestorer: &layer.NopSBOMRestorer{}, + PlatformAPI: f.platformAPI, + } + + ... + if f.platformAPI.AtLeast("0.8") && !inputs.SkipLayers && !inputs.SkipSBOM { + analyzer.SBOMRestorer = &layer.DefaultSBOMRestorer{ + LayersDir: inputs.LayersDir, + Logger: logger, + } + } +``` + +# Migration +[migration]: #migration + +CNB_SKIP_SBOM/ will be an optional input to /cnb/lifecycle/creator, and will be false by default. We maybe need to add a new API option for buildpack users, to choose whether this should be enabled. + +# Drawbacks +[drawbacks]: #drawbacks + +N/A + +# Alternatives +[alternatives]: #alternatives + +Platforms that execute lifecycle today via /cnb/lifecycle/creator are unable to skip restoring SBOM layer from previous app image unless they skip reusing previous app image entirely. + +# Prior Art +[prior-art]: #prior-art + +We already support enabling CNB_SKIP_LAYERS in /cnb/lifecycle/analyzer and /cnb/lifecycle/restorer, and CNB_SKIP_RESTORE in /cnb/lifecycle/creator. +* CNB_SKIP_LAYERS in /cnb/lifecycle/analyzer to skip restoring SBOM from previous app image. +* CNB_SKIP_LAYERS in /cnb/lifecycle/restorer to skip reusing previous app image layers entirely. +* CNB_SKIP_RESTORE in /cnb/lifecycle/creator to skips restoring SBOM plus all other layers entirely from previous app image. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +N/A + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes +This new feature will affect the API of [Create](https://buildpacks.io/docs/concepts/components/lifecycle/create/) phase by adding the following fields. + +Back to API changes, we will add a new flag to control this. + +| Input | Environment Variable | DefaultValue | Description | +|----------------|-----------------------|--------------|----------------------------------------------| +| `` | `CNB_SKIP_SBOM` | `false` | Skip SBOM restoration | + +# History +[history]: #history + + diff --git a/text/0127-extension-layer.md b/text/0127-extension-layer.md new file mode 100644 index 000000000..55f033e66 --- /dev/null +++ b/text/0127-extension-layer.md @@ -0,0 +1,131 @@ +# Meta +[meta]: #meta +- Name: Add extension layer to exchange data +- Start Date: 2023-10-09 +- Author(s): [c0d1ngm0nk3y](https://github.com/c0d1ngm0nk3y), [pbusko](https://github.com/pbusko) +- Status: Approved +- RFC Pull Request: +- CNB Pull Request: +- CNB Issue: +- Related: [RFC#0105 Support Dockerfiles](https://github.com/buildpacks/rfcs/blob/main/text/0105-dockerfiles.md) +- Supersedes: N/A + +# Summary +[summary]: #summary + +This RFC introduces support for Extension configurable context to allow data transfer between the build environment and the Kaniko execution. + +# Motivation +[motivation]: #motivation + +This change allows extensions to create their own context for the extend phase during the generation phase. Additionally, it ensures that extension output does not inadvertently interfere with other extension or buildpack layers during the build, and it does not unintentionally become part of the final application image. + +This would allow distroless run images to be extended. + +# What it is +[what-it-is]: #what-it-is + +This follows up on RFC-0105 and proposes that during the execution of the extension's `./bin/generate`, an extension is allowed to write arbitrary data to the `context` folder within its exclusive output directory. This data then becomes accessible during the execution of the `extend` phase via Kaniko build context. The content of these extension-specific context is ignored at build and launch time, it serves only the extension phase. + +# How it Works +[how-it-works]: #how-it-works + +- Before execution of the `./bin/generate`, the lifecycle will create a distinct writable layer `$CNB_GENERATED_DIR/` for each extension which passed detection. +- The `$CNB_GENERATED_DIR/` is provided to the `./bin/generate` as `` (`$CNB_OUTPUT_DIR`) directory. +- In addition to the files specified in [RFC#0105](https://github.com/buildpacks/rfcs/blob/main/text/0105-dockerfiles.md), the extension may create the following folders with an arbitrary content: + + either: + + - `/context` + + or the image-specific folders: + + - `/context.run` + - `/context.build` + + If the `/context` is provided together with any of the image-specific folders the detection phase must fail. +- If the folder `/context` is present it will be set as Kaniko build context during the `extend` phase of the build and run images. +- If the folder `/context.run` is present it will be set as Kaniko build context during the `extend` phase of the run image only. +- If the folder `/context.build` is present it will be set as Kaniko build context during the `extend` phase of the build image only. +- If none of these folders is not present, Kaniko build context defaults to the `` folder. + +The `$CNB_GENERATED_DIR/` folders will not be included in the final image by the lifecycle. + +### Example: Extend distroless run image with Debian packages. + +This example extension would allow to install `tar` package on the run image without package manager (distroless image). The extension contains `./bin/generate` and `./bin/custom-installer` file, which installs `.deb` files. + +##### `./bin/generate` + +```bash +#!/bin/sh + +mkdir -p ${CNB_OUTPUT_DIR}/context.run + +cp ${CNB_EXTENSION_DIR}/bin/custom-installer ${CNB_OUTPUT_DIR}/context.run/ +curl -o ${CNB_OUTPUT_DIR}/context.run/tar.deb http://security.ubuntu.com/ubuntu/pool/main/t/tar/tar_1.34+dfsg-1ubuntu0.1.22.04.1_amd64.deb + +cat >> "${CNB_OUTPUT_DIR}/run.Dockerfile" <` folder instead of a temporary directory. +- allow optional folders `$CNB_GENERATED_DIR//context`, `$CNB_GENERATED_DIR//context.run` and `$CNB_GENERATED_DIR//context.build` with an arbitrary content to be provided by extension. +- if the context folders are present, kaniko context should be set to the corresponding folder instead of the `` (following the rules defined in [#how-it-works](#how-it-works)). + + \ No newline at end of file