Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

podvm-mkosi: need allocate more space for "/run" to store large image data #1921

Open
genjuro214 opened this issue Jul 17, 2024 · 3 comments · May be fixed by #2212
Open

podvm-mkosi: need allocate more space for "/run" to store large image data #1921

genjuro214 opened this issue Jul 17, 2024 · 3 comments · May be fixed by #2212
Labels
podvm Related to podvm images

Comments

@genjuro214
Copy link
Contributor

genjuro214 commented Jul 17, 2024 •
edited
Loading

I attempted to run e2e tests on a Fedora s390x Pod VM image built using mkosi, and found the case TestCreatePeerPodWithLargeImage fails on the Fedora image but succeeds on an Ubuntu image built using Packer.

The root cause is there is no enough space to store the image data.

The large test image exceeds 2GB in size and is pulled into the directory "/run/kata-containers/image/layers". However, the mount point "/run" is only allocated 1.6GB of space via tmpfs:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb2        98G  7.2G   86G   8% /
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           4.0G     0  4.0G   0% /dev/shm
tmpfs           1.6G  162M  1.5G  11% /run
tmpfs           4.0G  4.0K  4.0G   1% /tmp
/dev/vdb1       223M   98M  108M  48% /boot
tmpfs           802M  4.0K  802M   1% /run/user/0

So it run out of space when pulling an image larger than 1.6GB.

There is no such problem for Ubuntu image built using Packer, because the mount service "run-kata\x2dcontainers.mount" will mount "/run/kata-containers" to hard disk "/kata-containers".

However, this mount service was removed for mkosi. I can find the related PR and commit:
#1606
c6ac908

Since "/" should be read only, should we allocate more space to "/run" for runtime data?

Based on my investigation, Fedora typically allocates 20% of physical RAM to "/run" via tmpfs by default. We can increase this allocation to 50% or more in /etc/fstab:

tmpfs           /run            tmpfs   rw,nosuid,nodev,size=50%   0 0

Moreover, we can allocate an even larger size than physical RAM by enabling swap. However, we must prepare the swap file on the hard disk in advance.

I hope we can discuss the following options:

  1. increase size percentage to 50% or 75%
  2. enable swap and allocate a larger size.
  3. keep the default size for "/run" but reduce the size of the large test image.
@genjuro214 genjuro214 added the podvm Related to podvm images label Jul 17, 2024
@bpradipt
Copy link
Member

@genjuro214 mkosi generated images are more closer to what we eventually want, ie

  1. verity based ro rootfs
  2. All writeable stuff in memory (since memory is encrypted)
  3. Mount encrypted storage if needed.

Now for large images we would want the VM memory to be sufficient to hold the image content and correspondingly the fs size should be adjusted as needed.

I think increasing the size of /run to 75% of mem is good option and run the tests to see if there are any isses.
Enabling swap will result in the risk of leaking sensitive data without additional protection in place and better avoided
Reducing the size of the large test image doesn't solve the real problem :-)

@stevenhorsman
Copy link
Member

All writeable stuff in memory (since memory is encrypted)

I'm not necessarily against this. But I remember that we had a discussion about the security implication before adding the /run/kata-containers -> /kata-containers and Azure and SE were using encrypted disks, so effectively gave us encrypted storage for free, which is why we were comfortable with this change. Has this position changed?

@bpradipt
Copy link
Member

All writeable stuff in memory (since memory is encrypted)

I'm not necessarily against this. But I remember that we had a discussion about the security implication before adding the /run/kata-containers -> /kata-containers and Azure and SE were using encrypted disks, so effectively gave us encrypted storage for free, which is why we were comfortable with this change. Has this position changed?

The position remains the same. If the disk is encrypted then we put the sensitive data there.
Since encrypted root disk with end user key needs quite a few manual steps (afaik), it's not safe to mount /run/kata-containers into a disk storage by default for mkosi based builds imho.

@bpradipt bpradipt mentioned this issue Jul 28, 2024
Open
@mkulke mkulke linked a pull request Dec 16, 2024 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
podvm Related to podvm images
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

RFC: add encrypted scratch space for sandbox mkulke/cloud-api-adaptor
3 participants
@bpradipt @genjuro214 @stevenhorsman