Kata main payload #298

stevenhorsman · 2023-11-29T17:42:01Z

Switch the CCv0 payload to the kata-deploy main version
- Note: I've not replaced the enclave-cc payloads as they are created in a different way IIUC
Update operator and reqs-payload images to use latest tags

stevenhorsman · 2023-11-29T17:56:06Z

/test

wainersm · 2023-11-29T20:31:45Z

hi @stevenhorsman , the changes you made look correct but we still need to fix at least one problem: not all runtimeClasses passed to the kata-deploy (samples/ccruntime/default/kustomization.yaml) are supported on main. At least kata-clh-tdx isn't as you can see on the error below when trying to install the operator:

~/operator/install/pre-install-payload
ccruntime.confidentialcontainers.org/ccruntime-sample created
ERROR: cc-operator-daemon-install pod is not running
DEBUG: Pod cc-operator-daemon-install-8gcdf
+ kubectl describe pods/cc-operator-daemon-install-8gcdf -n confidential-containers-system
Name:         cc-operator-daemon-install-8gcdf
Namespace:    confidential-containers-system
Priority:     0
Node:         myvm/192.168.122.243
Start Time:   Wed, 29 Nov 2023 20:06:16 +0000
Labels:       controller-revision-hash=54c88d84cf
              name=cc-operator-daemon-install
              pod-template-generation=1
Annotations:  <none>
Status:       Running
IP:           10.244.0.6
IPs:
  IP:           10.244.0.6
Controlled By:  DaemonSet/cc-operator-daemon-install
Containers:
  cc-runtime-install-pod:
    Container ID:  containerd://79da992890804f4adc73e35aaefa849cb4d615ac633636bb7d0fd35153a34f83
    Image:         quay.io/kata-containers/kata-deploy-ci:kata-containers-latest
    Image ID:      quay.io/kata-containers/kata-deploy-ci@sha256:538779d4e59d9a50b4fbee6252e8043ba30d909641cce751f70ba501bd7b613e
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/kata-artifacts/scripts/kata-deploy.sh
      install
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 29 Nov 2023 20:16:00 +0000
      Finished:     Wed, 29 Nov 2023 20:16:01 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 29 Nov 2023 20:13:13 +0000
      Finished:     Wed, 29 Nov 2023 20:13:14 +0000
    Ready:          False
    Restart Count:  6
    Environment:
      NODE_NAME:                     (v1:spec.nodeName)
      DEBUG:                        false
      DEFAULT_SHIM:                 qemu
      CREATE_DEFAULT_RUNTIMECLASS:  true
      CREATE_RUNTIMECLASSES:        true
      SHIMS:                        clh clh-tdx qemu qemu-tdx qemu-sev qemu-snp
      SNAPSHOTTER:                  nydus
      INSTALL_OFFICIAL_CONTAINERD:  false
    Mounts:
      /etc/containerd/ from containerd-conf (rw)
      /etc/crio/ from crio-conf (rw)
      /opt/kata/ from kata-artifacts (rw)
      /usr/local/bin/ from local-bin (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-995wl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  crio-conf:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/crio/
    HostPathType:  
  containerd-conf:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/containerd/
    HostPathType:  
  kata-artifacts:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/kata/
    HostPathType:  DirectoryOrCreate
  local-bin:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/bin/
    HostPathType:  
  kube-api-access-995wl:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              node.kubernetes.io/worker=
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  9m56s                  default-scheduler  Successfully assigned confidential-containers-system/cc-operator-daemon-install-8gcdf to myvm
  Normal   Pulled     5m53s                  kubelet            Successfully pulled image "quay.io/kata-containers/kata-deploy-ci:kata-containers-latest" in 4m1.87092816s
  Normal   Pulled     5m49s                  kubelet            Successfully pulled image "quay.io/kata-containers/kata-deploy-ci:kata-containers-latest" in 986.334737ms
  Normal   Pulled     5m34s                  kubelet            Successfully pulled image "quay.io/kata-containers/kata-deploy-ci:kata-containers-latest" in 1.03330351s
  Normal   Pulled     5m6s                   kubelet            Successfully pulled image "quay.io/kata-containers/kata-deploy-ci:kata-containers-latest" in 1.082074027s
  Warning  BackOff    4m38s (x6 over 5m47s)  kubelet            Back-off restarting failed container
  Normal   Pulling    4m23s (x5 over 9m55s)  kubelet            Pulling image "quay.io/kata-containers/kata-deploy-ci:kata-containers-latest"
  Normal   Created    4m22s (x5 over 5m53s)  kubelet            Created container cc-runtime-install-pod
  Normal   Started    4m22s (x5 over 5m53s)  kubelet            Started container cc-runtime-install-pod
  Normal   Pulled     4m22s                  kubelet            Successfully pulled image "quay.io/kata-containers/kata-deploy-ci:kata-containers-latest" in 1.116094312s
+ kubectl logs pods/cc-operator-daemon-install-8gcdf -n confidential-containers-system
Environment variables passed to this script
* NODE_NAME: myvm
* DEBUG: false
* SHIMS: clh clh-tdx qemu qemu-tdx qemu-sev qemu-snp
* DEFAULT_SHIM: qemu
* CREATE_RUNTIMECLASSES: true
* CREATE_DEFAULT_RUNTIMECLASS: true
* ALLOWED_HYPERVISOR_ANNOTATIONS: 
copying kata artifacts onto host
Creating the runtime classes
Creating the kata-clh runtime class
runtimeclass.node.k8s.io/kata-clh unchanged
Creating the kata-clh-tdx runtime class
error: the path "/opt/kata-artifacts/runtimeclasses/kata-clh-tdx.yaml" does not exist
+ set +x

stevenhorsman · 2023-11-30T11:34:14Z

hi @stevenhorsman , the changes you made look correct but we still need to fix at least one problem: not all runtimeClasses passed to the kata-deploy (samples/ccruntime/default/kustomization.yaml) are supported on main. At least kata-clh-tdx isn't as you can see on the error below when trying to install the operator:

I'm not sure if it is formally documented anywhere, but @fidencio and I have discussed TDX with CLH before not being in main and the decision was that it's not coming over in the short-term, so I've added commits to drop references to it.

stevenhorsman · 2023-11-30T11:34:18Z

/test

wainersm · 2023-11-30T13:29:24Z

bundle/manifests/cc-operator.clusterserviceversion.yaml

@@ -215,11 +215,7 @@ metadata:
                  "name": "kata-clh",
                  "snapshotter": "nydus"
                },
-                {


just one comment for the records (maybe I should document it). We have been updating manually the bundle/manifests files but in reality they should be generated by make bundle.

On the case of this PR, first edit the configuration file:

diff --git a/config/samples/ccruntime/default/kustomization.yaml b/config/samples/ccruntime/default/kustomization.yaml index 03c49f9..b22bc16 100644 --- a/config/samples/ccruntime/default/kustomization.yaml +++ b/config/samples/ccruntime/default/kustomization.yaml @@ -20,8 +20,6 @@ patches: value: - name: "kata-clh" snapshotter: "nydus" - - name: "kata-clh-tdx" - snapshotter: "nydus" - name: "kata-qemu" snapshotter: "nydus" - name: "kata-qemu-tdx"

Then run make bundle IMG=quay.io/confidential-containers/operator:v0.8.0 resulting in:

diff --git a/bundle/manifests/cc-operator.clusterserviceversion.yaml b/bundle/manifests/cc-operator.clusterserviceversion.yaml index bb27a36..3be280d 100644 --- a/bundle/manifests/cc-operator.clusterserviceversion.yaml +++ b/bundle/manifests/cc-operator.clusterserviceversion.yaml @@ -215,10 +215,6 @@ metadata: "name": "kata-clh", "snapshotter": "nydus" }, - { - "name": "kata-clh-tdx", - "snapshotter": "nydus" - }, { "name": "kata-qemu", "snapshotter": "nydus" @@ -250,8 +246,8 @@ metadata: ] capabilities: Basic Install categories: Security - containerImage: quay.io/confidential-containers/operator:v0.8.0 - createdAt: "2023-11-15T15:02:03Z" + containerImage: quay.io/confidential-containers/operator:latest + createdAt: "2023-11-30T13:25:18Z" operators.operatorframework.io/builder: operator-sdk-v1.30.0 operators.operatorframework.io/project_layout: go.kubebuilder.io/v3 name: cc-operator.v0.8.0 diff --git a/config/manager/kustomization.yaml b/config/manager/kustomization.yaml index 1c64f81..b247c45 100644 --- a/config/manager/kustomization.yaml +++ b/config/manager/kustomization.yaml @@ -13,4 +13,4 @@ kind: Kustomization images: - name: controller newName: quay.io/confidential-containers/operator - newTag: latest + newTag: v0.8.0 diff --git a/config/samples/ccruntime/default/kustomization.yaml b/config/samples/ccruntime/default/kustomization.yaml index 03c49f9..b22bc16 100644 --- a/config/samples/ccruntime/default/kustomization.yaml +++ b/config/samples/ccruntime/default/kustomization.yaml @@ -20,8 +20,6 @@ patches: value: - name: "kata-clh" snapshotter: "nydus" - - name: "kata-clh-tdx" - snapshotter: "nydus" - name: "kata-qemu" snapshotter: "nydus" - name: "kata-qemu-tdx"

ps: don't know why it changed containerImage to latest, it is supposed to keep v0.8.0, probably a misconfiguration somewhere

I've re-gened the bundle using your command, and fixup'd. It looks like it's mostly the timestamp and formatting that have updated. We'll see how the tests go once #299 is merged

stevenhorsman · 2023-12-04T10:30:44Z

/test

stevenhorsman · 2023-12-04T14:41:47Z

@BbolroC - hey Choi, the test for this are failing on s390x (in both Jenkins and locally) as the s390x kustomization.yaml file references the kata-qemu-se runtimeclass:

operator/config/samples/ccruntime/s390x/kustomization.yaml

Lines 21 to 22 in 4d65952

    
                 - name: "kata-qemu-se" 
        
                   snapshotter: "nydus"

which is not in the kata-deploy payload yet, so the operator-daemon-install fails:

# kubectl logs pod/cc-operator-daemon-install-dz2d9 -n confidential-containers-system
Environment variables passed to this script
* NODE_NAME: sh-s390x-2204-operator-test
* DEBUG: false
* SHIMS: qemu qemu-se
* DEFAULT_SHIM: qemu
* CREATE_RUNTIMECLASSES: true
* CREATE_DEFAULT_RUNTIMECLASS: true
* ALLOWED_HYPERVISOR_ANNOTATIONS:
copying kata artifacts onto host
Creating the runtime classes
Creating the kata-qemu runtime class
runtimeclass.node.k8s.io/kata-qemu unchanged
Creating the kata-qemu-se runtime class
error: the path "/opt/kata-artifacts/runtimeclasses/kata-qemu-se.yaml" does not exist

I'm guessing that will be done under kata-containers/kata-containers#6755, so wondered if that work is going to be merged soon, or if we can drop the kata-qemu-se runtime class now and then re-introduce it once it's gone into kata-deploy as we've done with CLH on TDX? Thanks!

BbolroC · 2023-12-04T17:07:09Z

I'm guessing that will be done under kata-containers/kata-containers#6755, so wondered if that work is going to be merged soon, or if we can drop the kata-qemu-se runtime class now and then re-introduce it once it's gone into kata-deploy as we've done with CLH on TDX? Thanks!

Yeah, let's drop it off now. I will raise a PR for the re-introduction. Thanks for the notice. 😉

stevenhorsman · 2023-12-04T18:14:07Z

/test

- Switch the `CCv0` payload to the kata-deploy main version - Note: I've not replaced the enclave-cc payloads as they are created in a different way IIUC Fixes: confidential-containers#297 Signed-off-by: stevenhorsman <[email protected]>

- Now 0.8 has release, switch back to testing latest reqs-payload image Signed-off-by: stevenhorsman <[email protected]>

- Now 0.8 is released update operator to pick up latest image Signed-off-by: stevenhorsman <[email protected]>

The clh-tdx runtime class isn't supported at the moment on `main`, so drop references to it as part of switching to use kata-deploy Signed-off-by: stevenhorsman <[email protected]>

The kata-qemu-se runtime class isn't supported at the moment on `main`, so drop references to it as part of switching to use kata-deploy Signed-off-by: stevenhorsman <[email protected]>

stevenhorsman · 2023-12-05T18:00:21Z

/test

wainersm · 2023-12-06T12:30:40Z

"hey, let's switch to the main branch, it should be as easy as point to the other payload".... and here we are with 9 commits

@stevenhorsman is there any missing piece? Can I approve and merge when I get one more ack?

wainersm

@stevenhorsman thanks again!

stevenhorsman · 2023-12-06T13:11:31Z

"hey, let's switch to the main branch, it should be as easy as point to the other payload".... and here we are with 9 commits

I'm just trying to break it down for ease of review (I've learnt from you and Fabiano !)

@stevenhorsman is there any missing piece? Can I approve and merge when I get one more ack?

Nothing missing that I know about

wainersm · 2023-12-06T14:24:08Z

Hi @bpradipt ! do you have time to review this one? :D

fidencio

lgtm, thanks @stevenhorsman!

stevenhorsman force-pushed the kata-main-payload branch from 377f290 to 6aaa279 Compare November 29, 2023 17:45

wainersm reviewed Nov 30, 2023

View reviewed changes

stevenhorsman force-pushed the kata-main-payload branch from 1adaa47 to ca3577e Compare November 30, 2023 13:56

wainersm mentioned this pull request Dec 1, 2023

tests/e2e: Remove kata-containers test #299

Merged

stevenhorsman force-pushed the kata-main-payload branch from ca3577e to 825af65 Compare December 4, 2023 10:30

stevenhorsman added do-not-merge and removed do-not-merge labels Dec 4, 2023

stevenhorsman marked this pull request as draft December 4, 2023 12:24

stevenhorsman added 9 commits December 5, 2023 17:58

kata-payload: update kata-payload to main

213262a

- Switch the `CCv0` payload to the kata-deploy main version - Note: I've not replaced the enclave-cc payloads as they are created in a different way IIUC Fixes: confidential-containers#297 Signed-off-by: stevenhorsman <[email protected]>

req-payload: Update to latest

af0db4f

- Now 0.8 has release, switch back to testing latest reqs-payload image Signed-off-by: stevenhorsman <[email protected]>

operator image: Switch to latest

81a89f9

- Now 0.8 is released update operator to pick up latest image Signed-off-by: stevenhorsman <[email protected]>

doc: Remove kata-clh-tdx

dd35b09

The clh-tdx runtime class isn't supported at the moment on `main`, so drop references to it as part of switching to use kata-deploy Signed-off-by: stevenhorsman <[email protected]>

bundle: Remove kata-clh-tdx runtime class

f914c11

The clh-tdx runtime class isn't supported at the moment on `main`, so drop references to it as part of switching to use kata-deploy Signed-off-by: stevenhorsman <[email protected]>

config: Remove kata-clh-tdx runtime class

4459b3c

The clh-tdx runtime class isn't supported at the moment on `main`, so drop references to it as part of switching to use kata-deploy Signed-off-by: stevenhorsman <[email protected]>

tests/e2e: Remove kata-clh-tdx runtime class

12e9eb6

The clh-tdx runtime class isn't supported at the moment on `main`, so drop references to it as part of switching to use kata-deploy Signed-off-by: stevenhorsman <[email protected]>

config: Remove kata-qemu-se

bfba38f

The kata-qemu-se runtime class isn't supported at the moment on `main`, so drop references to it as part of switching to use kata-deploy Signed-off-by: stevenhorsman <[email protected]>

tests/e2e: Remove kata-qemu-se runtime class

63fa8b0

The kata-qemu-se runtime class isn't supported at the moment on `main`, so drop references to it as part of switching to use kata-deploy Signed-off-by: stevenhorsman <[email protected]>

stevenhorsman force-pushed the kata-main-payload branch from 234b111 to 63fa8b0 Compare December 5, 2023 17:59

stevenhorsman marked this pull request as ready for review December 5, 2023 18:39

wainersm approved these changes Dec 6, 2023

View reviewed changes

fidencio approved these changes Dec 7, 2023

View reviewed changes

wainersm merged commit 5abf0c5 into confidential-containers:main Dec 7, 2023
7 checks passed

stevenhorsman deleted the kata-main-payload branch December 7, 2023 14:30

stevenhorsman linked an issue Dec 11, 2023 that may be closed by this pull request

Switch kata payload to main branch #297

Closed

stevenhorsman mentioned this pull request Dec 11, 2023

Attempt to update kata-runtime to point to the main version confidential-containers/cloud-api-adaptor#1596

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kata main payload #298

Kata main payload #298

stevenhorsman commented Nov 29, 2023

stevenhorsman commented Nov 29, 2023

wainersm commented Nov 29, 2023

stevenhorsman commented Nov 30, 2023

stevenhorsman commented Nov 30, 2023

wainersm Nov 30, 2023

stevenhorsman Nov 30, 2023

stevenhorsman commented Dec 4, 2023

stevenhorsman commented Dec 4, 2023 •

edited

Loading

BbolroC commented Dec 4, 2023

stevenhorsman commented Dec 4, 2023

stevenhorsman commented Dec 5, 2023

wainersm commented Dec 6, 2023

wainersm left a comment

stevenhorsman commented Dec 6, 2023

wainersm commented Dec 6, 2023

fidencio left a comment

Kata main payload #298

Kata main payload #298

Conversation

stevenhorsman commented Nov 29, 2023

stevenhorsman commented Nov 29, 2023

wainersm commented Nov 29, 2023

stevenhorsman commented Nov 30, 2023

stevenhorsman commented Nov 30, 2023

wainersm Nov 30, 2023

Choose a reason for hiding this comment

stevenhorsman Nov 30, 2023

Choose a reason for hiding this comment

stevenhorsman commented Dec 4, 2023

stevenhorsman commented Dec 4, 2023 • edited Loading

BbolroC commented Dec 4, 2023

stevenhorsman commented Dec 4, 2023

stevenhorsman commented Dec 5, 2023

wainersm commented Dec 6, 2023

wainersm left a comment

Choose a reason for hiding this comment

stevenhorsman commented Dec 6, 2023

wainersm commented Dec 6, 2023

fidencio left a comment

Choose a reason for hiding this comment

stevenhorsman commented Dec 4, 2023 •

edited

Loading