Skip to content

Latest commit

 

History

History
405 lines (330 loc) · 16.2 KB

how-to-run-kata-containers-with-SE-VMs.md

File metadata and controls

405 lines (330 loc) · 16.2 KB
Raw

Kata Containers with IBM Secure Execution VMs

This document assumes a trusted environment with a functioning kata container, as per the developer guide. The term "trusted" implies that the system is authorized, authenticated and attested to use your artifacts and secrets safely.

Tested Environment Specifications

  1. Machine: IBM z16 LPAR
  2. OS: Ubuntu 22.04.1 LTS
  3. CPU: 16 vCPU
  4. Memory: 16G

Manual configuration

Prerequisite

  1. Host capable of Secure Execution

To take advantage of the IBM Secure Execution capability, the host machine on which you intend to run workloads must be an IBM z15 (or a newer model) or an IBM LinuxONE III (or a newer model). In addition to the hardware requirement, you need to verify the CPU facility and kernel configuration, as outlined below:

$ # To check the protected virtualization support from kernel
$ cat /sys/firmware/uv/prot_virt_host
1
$ # To check if an ultravisor reserves memory for the current boot
$ sudo dmesg | grep -i ultravisor
[    0.063630] prot_virt.f9efb6: Reserving 98MB as ultravisor base storage
$ # To check a facility bit for Secure Execution
$ cat /proc/cpuinfo | grep 158
facilities      :  ... numbers ... 158 ... numbers ...

If any of the results are not identifiable, please reach out to the responsible cloud provider to enable the Secure Execution capability. Alternatively, if you possess administrative privileges and the facility bit is set, you can enable the Secure Execution capability by adding prot_virt=1 to the kernel parameters and performing a system reboot like:

$ sudo sed -i 's/^\(parameters.*\)/\1 prot_virt=1/g' /etc/zipl.conf
$ sudo zipl -V
$ sudo systemctl reboot

Please note that the method of enabling the Secure Execution capability may vary among Linux distributions.

  1. Artifacts from Kata Containers

A secure image is constructed using the following artifacts

  • A raw kernel
  • An initial RAM disk

The most straightforward approach to obtain these artifacts is by reusing kata-containers:

$ export PATH="$PATH:/opt/kata/bin"
$ ls -1 $(dirname $(kata-runtime env --json | jq -r '.Kernel.Path'))
config-6.1.62-121
kata-containers.img
kata-containers-confidential.img
kata-containers-initrd.img
kata-containers-initrd-confidential.img
kata-ubuntu-20.04.initrd
kata-ubuntu-20.04-confidential.initrd
kata-ubuntu-latest.image
kata-ubuntu-latest-confidential.image
vmlinux-6.1.62-121
vmlinux-6.1.62-121-confidential
vmlinux.container
vmlinux-confidential.container
vmlinuz-6.1.62-121
vmlinuz-6.1.62-121-confidential
vmlinuz.container
vmlinuz-confidential.container

The output indicates the deployment of the kernel (vmlinux-6.1.62-121-confidential, though the version may vary at the time of testing), rootfs-image (kata-ubuntu-latest-confidential.image), and rootfs-initrd (kata-ubuntu-20.04-confidential.initrd). In this scenario, the available kernel and initrd can be utilized for a secure image. However, if any of these components are absent, they must be built from the project source as follows:

$ # Assume that the project is cloned at $GOPATH/src/github.com/kata-containers
$ cd $GOPATH/src/github.com/kata-containers/kata-containers
$ make rootfs-initrd-confidential-tarball
$ tar -tf build/kata-static-kernel-confidential.tar.xz | grep vmlinuz
./opt/kata/share/kata-containers/vmlinuz-confidential.container
./opt/kata/share/kata-containers/vmlinuz-6.7-136-confidential
$ kernel_version=6.7-136
$ tar -tf build/kata-static-rootfs-initrd-confidential.tar.xz | grep initrd
./opt/kata/share/kata-containers/kata-containers-initrd-confidential.img
./opt/kata/share/kata-containers/kata-ubuntu-20.04-confidential.initrd
$ mkdir artifacts
$ tar -xvf build/kata-static-kernel-confidential.tar.xz -C artifacts ./opt/kata/share/kata-containers/vmlinuz-${kernel_version}-confidential
$ tar -xvf build/kata-static-rootfs-initrd-confidential.tar.xz -C artifacts ./opt/kata/share/kata-containers/kata-ubuntu-20.04-confidential.initrd
$ ls artifacts/opt/kata/share/kata-containers/
kata-ubuntu-20.04-confidential.initrd  vmlinuz-${kernel_version}-confidential
  1. Secure Image Generation Tool

genprotimg is a utility designed to generate an IBM Secure Execution image. It can be installed either from the package manager of a distribution or from the source code. The tool is included in the s390-tools package. Please ensure that you have a version of the tool equal to or greater than 2.17.0. If not, you will need to specify an additional argument, --x-pcf '0xe0', when running the command. Here is an example of a native build from the source:

$ sudo apt-get install gcc libglib2.0-dev libssl-dev libcurl4-openssl-dev
$ tool_version=v2.34.0
$ git clone -b $tool_version https://github.com/ibm-s390-linux/s390-tools.git
$ pushd s390-tools/genprotimg && make && sudo make install && popd
$ rm -rf s390-tools
  1. Host Key Document

A host key document is a public key employed for encrypting a secure image, which is subsequently decrypted using a corresponding private key during the VM bootstrap process. You can obtain the host key document either through IBM's designated Resource Link(you need to log in to access it) or by requesting it from the cloud provider responsible for the IBM Z and LinuxONE instances where your workloads are intended to run.

To ensure security, it is essential to verify the authenticity and integrity of the host key document belonging to an authentic IBM machine. To achieve this, please additionally obtain the following files from the Resource Link:

  • IBM Z signing key certificate
  • IBM Z host key certificate revocation list
  • DigiCert intermediate CA certificate

These files will be used for verification during secure image construction in the next section.

Build a Secure Image

Assuming you have placed a host key document at $HOME/host-key-document:

  • Host key document as HKD-0000-0000000.crt

and two certificates and one revocation list at $HOME/certificates:

  • IBM Z signing-key certificate as ibm-z-host-key-signing-gen2.crt
  • DigiCert intermediate CA certificate as DigiCertCA.crt
  • IBM Z host key certificate revocation list as ibm-z-host-key-gen2.crl

you can construct a secure image using the following procedure:

$ # Change a directory to the project root
$ cd $GOPATH/src/github.com/kata-containers/kata-containers
$ host_key_document=$HOME/host-key-document/HKD-0000-0000000.crt
$ kernel_image=artifacts/opt/kata/share/kata-containers/vmlinuz-${kernel_version}-confidential
$ initrd_image=artifacts/opt/kata/share/kata-containers/kata-ubuntu-20.04-confidential.initrd
$ echo "panic=1 scsi_mod.scan=none swiotlb=262144 agent.log=debug" > parmfile
$ genprotimg --host-key-document=${host_key_document} \
--output=kata-containers-se.img --image=${kernel_image} --ramdisk=${initrd_image} \
--parmfile=parmfile --no-verify
WARNING: host-key document verification is disabled. Your workload is not secured.
$ file kata-containers-se.img
kata-containers-se.img: data
$ sudo cp kata-containers-se.img /opt/kata/share/kata-containers/

It is important to note that the --no-verify parameter, which allows skipping the key verification process, is intended to be used solely in a development or testing environment. In production, the image construction should incorporate the verification in the following manner:

$ signcert=$HOME/certificates/ibm-z-host-key-signing-gen2.crt
$ cacert=$HOME/certificates/DigiCertCA.crt
$ crl=$HOME/certificates/ibm-z-host-key-gen2.crl
$ genprotimg --host-key-document=${host_key_document} \
--output=kata-containers-se.img --image=${kernel_image} --ramdisk=${initrd_image} \
--cert=${cacert} --cert=${signcert} --crl=${crl} --parmfile=parmfile

The steps with no verification, including the dependencies for the kernel and initrd, can be easily accomplished by issuing the following make target:

$ cd $GOPATH/src/github.com/kata-containers/kata-containers
$ mkdir hkd_dir && cp $host_key_document hkd_dir
$ HKD_PATH=hkd_dir SE_KERNEL_PARAMS="agent.log=debug" make boot-image-se-tarball
$ ls build/kata-static-boot-image-se.tar.xz
build/kata-static-boot-image-se.tar.xz

SE_KERNEL_PARAMS could be used to add any extra kernel parameters. If no additional kernel configuration is required, this can be omitted.

In production, you could build an image by running the same command, but with the following environment variables for key verification:

$ export SIGNING_KEY_CERT_PATH=$HOME/certificates/ibm-z-host-key-signing-gen2.crt
$ export INTERMEDIATE_CA_CERT_PATH=$HOME/certificates/DigiCertCA.crt
$ export HOST_KEY_CRL_PATH=$HOME/certificates/ibm-z-host-key-gen2.crl

To build an image on the x86_64 platform, set the following environment variables together with the variables above before make boot-image-se-tarball:

CROSS_BUILD=true TARGET_ARCH=s390x ARCH=s390x

Adjust the configuration

There still remains an opportunity to fine-tune the configuration file:

$ export PATH=$PATH:/opt/kata/bin
$ runtime_config_path=$(kata-runtime kata-env --json | jq -r '.Runtime.Config.Path')
$ sudo cp ${runtime_config_path} ${runtime_config_path}.old
$ # Make the following adjustment to the original config file
$ diff ${runtime_config_path}.old ${runtime_config_path}
16,17c16,17
< kernel = "/opt/kata/share/kata-containers/vmlinux.container"
< image = "/opt/kata/share/kata-containers/kata-containers.img"
---
> kernel = "/opt/kata/share/kata-containers/kata-containers-se.img"
> # image = "/opt/kata/share/kata-containers/kata-containers.img"
41c41
< # confidential_guest = true
---
> confidential_guest = true
544c544
< dial_timeout = 45
---
> dial_timeout = 90

Verification

To verify the successful decryption and loading of the secure image within a test VM, please refer to the following commands:

$ cd $GOPATH/src/github.com/kata-containers/kata-containers
$ hypervisor_command=$(kata-runtime kata-env --json | jq -r '.Hypervisor.Path')
$ secure_kernel=kata-containers-se.img
$ sudo $hypervisor_command -machine confidential-guest-support=pv0 \
-object s390-pv-guest,id=pv0 -accel kvm -smp 2 --m 4096 -serial mon:stdio \
--nographic --nodefaults --kernel "${secure_kernel}"
[    0.110277] Linux version 5.19.2 (root@637f067c5f7d) (gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #1 SMP Wed May 31 09:06:49 UTC 2023                                                                     [    0.110279] setup: Linux is running under KVM in 64-bit mode

... log skipped ...

[    1.467228] Run /init as init process
{"msg":"baremount source=\"proc\", dest=\"/proc\", fs_type=\"proc\", options=\"\", flags=MS_NOSUID | MS_NODEV | MS_NOEXEC","level":"INFO","ts":"2023-06-07T10:17:23.537542429Z","pid":"1","subsystem":"baremount","name":"kata-agent","source":"agent
","version":"0.1.0"}

... log skipped ...

$ # Press ctrl + a + x to exit

Unless the host key document is legitimate, you will encounter the following error message:

qemu-system-s390x: KVM PV command 2 (KVM_PV_SET_SEC_PARMS) failed: header rc 108 rrc 5 IOCTL rc: -22
Protected boot has failed: 0xa02

If the hypervisor log does not indicate any errors, it provides assurance that the image has been successfully loaded, and a Virtual Machine (VM) initiated by the kata runtime will function properly.

Let us proceed with the final verification by running a test container in a Kubernetes cluster. Please make user you have a running cluster like:

$ kubectl get node
NAME           STATUS   ROLES                  AGE     VERSION
test-cluster   Ready    control-plane,master   7m28s   v1.23.1

Please execute the following command to run a container:

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx-kata
spec:
  runtimeClassName: kata-qemu
  containers:
  - name: nginx
    image: nginx
EOF
pod/nginx-kata created
$ kubectl get po
NAME         READY   STATUS    RESTARTS   AGE
nginx-kata   1/1     Running   0          29s
$ kubectl get po -oyaml | grep "runtimeClassName:"
    runtimeClassName: kata-qemu
$ # Please make sure if confidential-guest-support is set and a secure image is used
$ $ ps -ef | grep qemu | grep -v grep
root       76972   76959  0 13:40 ?        00:00:02 /opt/kata/bin/qemu-system-s390x
... qemu arguments ...
-machine s390-ccw-virtio,accel=kvm,confidential-guest-support=pv0
... qemu arguments ...
-kernel /opt/kata/share/kata-containers/kata-containers-se.img
... qemu arguments ...

Finally, an operational kata container with IBM Secure Execution is now running.

Using Kata-Deploy with Confidential Containers Operator

It is reasonable to expect that the manual steps mentioned above can be easily executed. Typically, you can use kata-deploy to install Kata Containers on a Kubernetes cluster. However, when leveraging IBM Secure Execution, you need to employ the confidential container's operator. During this process, a kata-deploy container image serves as a payload image in a custom resource ccruntime for confidential containers, enabling the operator to install Kata binary artifacts such as kernel, shim-v2, and more.

This section will explain how to build a payload image (i.e., kata-deploy) for confidential containers. For the remaining instructions, please refer to the documentation for confidential containers.

$ cd $GOPATH/src/github.com/kata-containers/kata-containers
$ host_key_document=$HOME/host-key-document/HKD-0000-0000000.crt
$ mkdir hkd_dir && cp $host_key_document hkd_dir
$ # kernel-confidential and rootfs-initrd-confidential are built automactially by the command below
$ HKD_PATH=hkd_dir SE_KERNEL_PARAMS="agent.log=debug" make boot-image-se-tarball
$ make qemu-tarball
$ make virtiofsd-tarball
$ make shim-v2-tarball
$ mkdir kata-artifacts
$ build_dir=$(readlink -f build)
$ cp -r $build_dir/*.tar.xz kata-artifacts
$ ls -1 kata-artifacts
kata-static-agent.tar.xz
kata-static-boot-image-se.tar.xz
kata-static-coco-guest-components.tar.xz
kata-static-kernel-confidential-modules.tar.xz
kata-static-kernel-confidential.tar.xz
kata-static-pause-image.tar.xz
kata-static-qemu.tar.xz
kata-static-rootfs-initrd-confidential.tar.xz
kata-static-shim-v2.tar.xz
kata-static-virtiofsd.tar.xz
$ ./tools/packaging/kata-deploy/local-build/kata-deploy-merge-builds.sh kata-artifacts

In production, the environment variables SIGNING_KEY_CERT_PATH, INTERMEDIATE_CA_CERT_PATH and SIGNING_KEY_CERT_PATH should be exported like the manual configuration. If a rootfs-image is required for other available runtime classes (e.g. kata and kata-qemu) without the Secure Execution functionality, please run the following command before running kata-deploy-merge-builds.sh:

$ make rootfs-image-tarball

At this point, you should have an archive file named kata-static.tar.xz at the project root, which will be used to build a payload image. If you are using a local container registry at localhost:5000, proceed with the following:

$ docker run -d -p 5000:5000 --name local-registry registry:2.8.1

Build and push a payload image with the name localhost:5000/build-kata-deploy and the tag latest using the following:

$ ./tools/packaging/kata-deploy/local-build/kata-deploy-build-and-upload-payload.sh kata-static.tar.xz localhost:5000/build-kata-deploy latest
... logs ...
Pushing the image localhost:5000/build-kata-deploy:latest to the registry
The push refers to repository [localhost:5000/build-kata-deploy]
76c6644d9790: Layer already exists
2413aff53bb1: Layer already exists
91462f44bb06: Layer already exists
2ad49fac591a: Layer already exists
5c75aa64ef7a: Layer already exists
test: digest: sha256:25825c7a4352f75403ee59a683eb122d5518e8ed6a244aacd869e41e2cafd385 size: 1369

Considerations for CI

If you intend to integrate the aforementioned procedure with a CI system, configure the following setup for an environment variable. The setup helps speed up CI jobs by caching container images used during the build:

$ export BUILDER_REGISTRY=$YOUR_PRIVATE_REGISTRY_FOR_CI
$ export PUSH_TO_REGISTRY=yes