diff --git a/enhancements/rhcos/split-rhcos-into-layers.md b/enhancements/rhcos/split-rhcos-into-layers.md new file mode 100644 index 0000000000..a5cfead911 --- /dev/null +++ b/enhancements/rhcos/split-rhcos-into-layers.md @@ -0,0 +1,428 @@ +--- +title: split-rhcos-into-layers +authors: + - "@jlebon" + - "@cverna" + - "@travier" +reviewers: + - "@patrickdillon, for installer impact" + - "@rphillips, for node impact" + - "@joepvd, for ART impact" + - "@sinnykumari, for MCO impact" + - "@LorbusChris, for OKD impact" + - "@zaneb, for agent installer impact" + - "@sdodson, for overall architecture" + - "@cgwalters, for overall architecture" +approvers: + - "@mrunalp" +api-approvers: + - None +creation-date: 2024-06-07 +last-updated: 2024-06-07 +tracking-link: + - https://issues.redhat.com/browse/OCPSTRAT-1190 +see-also: + - "/enhancements/ocp-coreos-layering/ocp-coreos-layering.md" +--- + +# Split RHCOS into layers + +## Summary + +This enhancement describes improvements to the way RHEL CoreOS (RHCOS) is built +so that it will better align with image mode for RHEL, all while also providing +benefits on the OpenShift side. Currently, RHCOS is built as a single layer that +includes both RHEL and OCP content. This enhancement proposes splitting it into +three layers. Going from bottom to top: +1. the (RHEL-versioned) bootc layer (i.e. the base rhel-bootc image shared with image mode for RHEL) +2. the (RHEL-versioned) CoreOS layer (i.e. coreos-installer, ignition, afterburn, scripts, etc...) +3. the (OCP-versioned) node layer (i.e. kubelet, cri-o, etc...) + +The terms "bootc layer", "CoreOS layer", and "node layer" will be used +throughout this enhancement to refer to these. + +The details of this enhancement focus on doing the first split: creating the node +layer as distinct from the CoreOS layer (which will not yet be rebased on top of +a bootc layer). The two changes involved which most affect OCP are: +1. bootimages will no longer contain OCP components (e.g. kubelet, cri-o, etc...) +2. the `rhel-coreos` payload image will be built in Prow/Konflux (as any other) + +This enhancement is based on initial sketches by @cgwalters in +https://github.com/openshift/os/issues/799. + +## Motivation + +Image mode for RHEL provides an opportunity to rethink how RHCOS is built +and allow us to share a lot more with other image mode variants. A key benefit +on the RHEL side is that we would have only one stream of RHCOS per RHEL release, +rather than one per OpenShift release. This greatly reduces the workload on the +CoreOS team. Another benefit is easier integration in the CI processes of +rhel-bootc and centos-bootc, as well as better shared documentation. + +On the OCP side, the final node image would now be built as a layered +container image, just like most other OpenShift components are built. This should +allow simplifying CI and ART tooling related to RHCOS, much of which is bespoke. + +For example, ART would no longer have to sync RHCOS images to CI; CI could +build its own like it does other images. Layering also means faster iteration on +OCP packages (e.g. openshift-clients can keep churning fast without incurring a +full RHCOS rebuild each time). And it also opens the door to more easily ship +binaries in the OS without necessarily having to package them as RPMs first. + +Finally, keeping the CoreOS layer separate from the node layer provides major +benefits to OKD. First, because it makes it easier to share the derivation process +for building the node image (today, OKD maintains its own separate definitions). +Second, because it allows the sharing of CentOS Stream-based bootimages between +OCP and OKD (today, OKD uses FCOS bootimages). These changes would take OKD one +step closer to become a true upstream of OCP. + +### User Stories + +* As an OpenShift engineer, I want the openshift/os repo to work similarly to other OpenShift repos (with a `Containerfile`), so that CI can easily build and test a PR in a cluster and humans too can more easily build and test PRs or e.g. use Cluster Bot. + +* As a CoreOS engineer, I want to only have a single stream of RHCOS per RHEL stream, so that I can greatly reduce the amount of streams to maintain (build, test, monitor for failures, flakes, etc...), reduce the number of bootimage bumps, and enforce more consistency across OCP releases that span the same RHEL version. + +* As a CoreOS engineer, I want to build RHCOS on top of rhel-bootc, so that I can benefit from support by the wider RHEL organization on those images and collaborate on CI testing and fixing bugs. + +* As an OpenShift customer, I want to be able to follow (most of) the image mode for RHEL documentation and use (most of) the same tooling surrounding it when customizing RHCOS, so that I have a consistent experience across image mode for RHEL and RHCOS and a clear understanding of how they relate to each other. + +* As an ART engineer, I want to be able to build the OpenShift node image in a similar way to how the other OpenShift components are built, so that I can simplify my tooling and pipelines. + +* As a RHEL kernel engineer, I want to be able to debug an OCP kernel bug the same way I do for image mode for RHEL, so that I don't have to learn different ways to do things. + +* As an OKD contributor, I want to be able to use CentOS Stream-based boot images, so that I don't have to keep pivoting from Fedora CoreOS boot images into CentOS Stream everytime. + +* As an OKD contributor, I want to be able to share node image building processes with OCP, so that I can reduce the maintenance load. + +* (Longer-term) As a product manager, I want image mode for RHEL certification and support guarantees to automatically apply to OpenShift clusters. + +### Goals + +- Build RHCOS/SCOS on top of rhel-bootc/centos-bootc +- Share efforts, especially CI and docs, with the larger rhel/centos-bootc project +- Maintain a single RHCOS stream per RHEL stream +- Build the OpenShift node image as a layered image on top of RHCOS/SCOS +- Build the OpenShift node image using Prow (in CI) and Konflux (in prod) + +### Non-Goals + +- Change the cluster installation flow. It should remain the same whether IPI/UPI/AI/etc... +- Introduce cluster administrator-visible changes. This change should be transparent to + admistrators. CoreOS layering instructions should keep working as is, but documentation + should ideally be reworked to leverage rhel-bootc docs more. +- Remove Ignition. While image mode for RHEL lessens the need for Ignition, it remains a + crucial part of the provisioning flow and will stick around for the foreseeable future. + +## Proposal + +The overall proposal consists of two parts. In the first part, we will focus on moving +the OCP-specific packages out of RHCOS and into its own layer (the node layer). In the second +part, we'll change RHCOS itself to become a layered image (the CoreOS layer) built on top +of the rhel-bootc work (the bootc layer). + +As mentioned in the summary, this enhancement is focused on detailing the first part only. +Firstly, because it's the one we'll start with, and secondly, because it's the one that +will most impact OpenShift. The second part will require changes that are more contained +to the CoreOS and ART teams. That said, we felt it important to introduce them together +in this enhancement to provide a larger picture. + +Diving into the first part then, at a high-level this is split into three phases. + +#### Phase 1: Using RHEL/CentOS Stream-only bootimages + +In this phase, the goal is to have the OpenShift installer use bootimages containing only +RHEL/CentOS Stream content. + +To do this, we will start building two new streams in the RHCOS pipeline containing only pure +RHEL/CentOS Stream content (let's call these the "pure RHEL stream" and "pure CentOS stream"). +Those streams will also be building the usual bootimages and uploading them to cloud. + +Once we have these bootimages, we can better start adapting components that will need it +to account for the lack of OpenShift components in the bootimages. Likely suspects here are +any components involved in the bringup of the cluster (installer, Assisted Installer, MCO, etc...). + +Once all known issues have been resolved, the installer bootimage metadata will be updated +to point to these new images. At this point, the OpenShift node image (i.e. the +`rhel-coreos` image in the release payload) has not changed. + +Similarly, the installer bootimage metadata for OKD will be updated to use the new bootimages +from the CentOS stream. + +#### Phase 2: Switching `rhel-coreos` in CI + +In this phase, the goal is to have the OpenShift node image be built in, and used by, CI. + +To do this, we will configure the RHCOS pipeline to push the bootable container image output +from the pure RHEL stream to a development Quay repo. We will then create a new Prow CI job +which will build the OpenShift node image by layering the OpenShift components on top: RPM +packages, config files, etc... + +After some testing, the Prow job will be configured to push to the `rhel-coreos` tag in the CI +imagestream. The ART job that syncs the RHCOS container images to CI will be disabled. + +(Ideally, we'd use a new tag for this here and in the release payload. E.g. `node` or `os` +would be more appropriate, but... there's a lot of work involved in doing this.) + +#### Phase 3: Switching `rhel-coreos` in production + +In this phase, the goal is to have the OpenShift node image be built in, and used by, production. + +To do this, we will setup a Konflux pipeline to build the layered node image. After some +additional testing, this layered image will replace the current `rhel-coreos` image in the +production release payload. + +### Workflow Description + +This enhancement should have no immediate impact on cluster administrators. However, better +alignment with image mode for RHEL should result in better documentation and tooling down the line. + +### Topology Considerations + +#### Hypershift / Hosted Control Planes + +TODO + + + +#### Standalone Clusters + +TODO + + + +#### Single-node Deployments or MicroShift + +TODO + + + +### Implementation Details/Notes/Constraints + +As mentioned in the proposal, the work is split into three phases. Here, we go into more +explicit implementation details for each phase. + +##### Phase 1: Using RHEL/CentOS Stream-only bootimages + +- In openshift/os, create a new variant of RHCOS which contains solely RHEL packages, + i.e. today's RHCOS but without any OCP components. (Note: this work has already been done: + https://github.com/openshift/os/pull/1445.) +- In the RHCOS pipeline, start building this new stream variant. This will push the bootable + container to quay.io/openshift-release-dev/ocp-v4.0-art-dev. Bootimages will also be built + as usual. +- Use these bootimages to perform some preliminary tests and determine the areas that + need attention. Some areas are expected to require adapting (e.g. bootstrapping), though it's + likely that other issues as well will surface. +- Work with the teams of the affected components to adapt them. +- At some point of the iteration cycle when we've made some good initial progress, + open an openshift/installer PR which updates the bootimages. This will allow us to run more + extensive CI tests on the PR. +- Once all issues have been resolved, merge the PR. + +##### Phase 2: Switching `rhel-coreos` in CI + +- Set up a new job in openshift/release which builds the layered OpenShift node image on + top of the bootable container image from the new stream variant. +- Open MCO and DTK PRs pointing to this layered image in CI (these are the only two components + that explicitly reference `rhel-coreos`) as a way to verify CI is happy. + - NOTE: There shouldn't be any major CI failures here. One key thing to understand is that + we're changing *how* the node image is built, but not *what* its final contents are from + a squashed image perspective. So this is mostly a sanity-check that they are indeed + equivalent. +- Once any issues have been resolved, make the new job push to the `:rhel-coreos` tag of the + CI imagestream. This simultaneously requires disabling ART's job which syncs RHCOS to that + same tag. + +##### Phase 3: Switching `rhel-coreos` in production + +- Work with DPTP(?) to update the OCP release page to better handle the new layered approach. + - It currently links to diffs in the RHCOS release browser. But now the release browser + will only contain the RHEL diff, not the OCP diff in the node layer. We should close this + gap before moving prod. +- Work with ART to set up a Konflux pipeline to build the layered OpenShift node image. +- Do a final set of tests to verify that CI is happy with this built image. + - There should be no surprises here. +- Once any final issues have been resolved, switch over `rhel-coreos` in prod to come from + the node image built by the Konflux pipeline. + +### Follow-ups + +With these changes in place, one follow-up worth investigating is to move the base RHCOS +definition out of openshift/os. This would better clarify the dividing line between the CoreOS +and node layers, by having their definitions live in separate repos. + +The CoreOS layer definition at this point would probably better live somewhere under gitlab.com/redhat +alongside other rhel/centos-bootc projects. Then, openshift/os would be free to contain even more +OpenShift-specific things. For example, we might consider moving content currently in the MCO's +templates into openshift/os. or we might consider rewriting things currently written in bash +into a statically-typed language that can literally live in openshift/os and get compiled and +added as part of the derived (now multi-staged) layered build. + +At a more practical level, this helps reduce git churn for both the RHCOS pipeline and the +OCP layered build. + +Another important follow-up would to be to hook up better CI testing to openshift/os. There is +currently no CI test on that repo which actually launches a cluster. The reason is that RHCOS +is just built differently. But now that the node image is simply yet another layered image build, +it fits perfectly in Prow's opinionated model of building the image and shoving it in the test +release payload with which to launch test cluster. + +### Risks and Mitigations + +TODO + + + +### Drawbacks + +TODO + + + +## Open Questions [optional] + +TODO + + + +## Test Plan + +This has been covered in the proposal and implementation sections. + +This isn't an optional feature that's getting added; once we cut over to the +new model, it will be implicitly continuously tested by all OpenShift CI tests. + +## Graduation Criteria + +This is not a new user-facing (or even internal-facing) OpenShift API. However, +for completeness, let's consider "graduation" in this case as switching OCP over +to using the layered image in production. + +For this to happen, the criteria are: +- all affected OpenShift components and processes have been adapted to deal with + the new bootimages (which no longer have OpenShift packages) +- the OpenShift installer now uses the new bootimages +- the OpenShift node image can be built in Prow/Konflux +- the new artifacts are used as part of the CI streams and all tests pass + +## Upgrade / Downgrade Strategy + +TODO + + + +## Version Skew Strategy + +TODO + + + +## Support Procedures + +In general, support procedures will be exactly the same as they currently are for +OS-level issues. However, awareness that the node image is now a layered image will +make it easier for the support team to debug issues by looking in the right places. + +## Alternatives + +TODO + +