Refine tarfs overall design and implementation details #499

jiangliu · 2023-06-24T03:16:13Z

This PR is based on #497 , and improve tarfs by:

move loopdev setup from Prepare()/Merge() into Mount(), preparing for support of nodev mode in addition to blockdev.
store tar file in the global cache directory, mount tarfs at "mnt" instead of "tarfs", to keep consistence with fusedev/fscache drivers.
use tempfile/rename mechanism to easy error recover
use lock to avoid some possible race conditions
use mode 0750 instead of 0755 for new directories
associate a Manager with filesystems managed by blockdev

codecov · 2023-06-24T17:30:20Z

Codecov Report

Merging #499 (f3c1541) into main (33bca86) will decrease coverage by 0.13%.
The diff coverage is 0.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #499      +/-   ##
==========================================
- Coverage   37.68%   37.55%   -0.13%     
==========================================
  Files          60       60              
  Lines        7115     7139      +24     
==========================================
  Hits         2681     2681              
- Misses       4122     4146      +24     
  Partials      312      312

Files Changed	Coverage Δ
config/config.go	`31.38% <ø> (ø)`
config/global.go	`24.27% <0.00%> (-7.38%)`	⬇️
pkg/manager/manager.go	`14.88% <0.00%> (ø)`

jiangliu · 2023-06-26T03:11:13Z

With this PR applied:

root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# nerdctl run --rm -ti --snapshotter nydus busybox
docker.io/library/busybox:latest:                                                 resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:6e494387c901caf429c1bf77bd92fb82b33a68c0e19f6d1aa6a3ac8d27a7049d:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:1b0a26bd07a3d17473d8d8468bea84015e27f87124b283b91d781bce13f61370: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b539af69bc01c6c1c1eae5474a94b0abaab36b93c165c0cf30b7a0ab294135a3:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 5.6 s                                                                    total:  2.8 Ki (504.0 B/s)                                       
[2023-06-26 10:58:49.234445 +08:00] INFO RAFS features: COMPRESSION_NONE | HASH_SHA256 | EXPLICIT_UID_GID | TARTFS_MODE
/ # 

root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# mount
/dev/loop13 on /var/lib/containerd-nydus/snapshots/2/mnt type erofs (ro,relatime,user_xattr,acl,cache_strategy=readaround)
root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# losetup 
NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                                                            DIO LOG-SEC
/dev/loop13
             0      0         0  0 /var/lib/containerd-nydus/snapshots/2/fs/image/image.boot              0     512
/dev/loop12
             0      0         0  0 /var/lib/containerd-nydus/cache/71d064a1ac7d46bdcac82ea768aba4ebbe2a05ccbd3a4a82174c18cf51b67ab7 0     512

root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# ls -l /var/lib/containerd-nydus/cache/
total 10812
-rw-r--r-- 1 root root 5092352 Jun 26 11:05 71d064a1ac7d46bdcac82ea768aba4ebbe2a05ccbd3a4a82174c18cf51b67ab7
-rw-r--r-- 1 root root 5976064 Jun 26 11:05 71d064a1ac7d46bdcac82ea768aba4ebbe2a05ccbd3a4a82174c18cf51b67ab7.image.disk

root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# ls -l /var/lib/containerd-nydus/snapshots/1/fs/image/
total 48
-rw-r--r-- 1 root root 28672 Jun 26 11:05 image.boot
-rw-r--r-- 1 root root 28672 Jun 26 11:05 layer.boot

root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# losetup /dev/loop100 /var/lib/containerd-nydus/cache/71d064a1ac7d46bdcac82ea768aba4ebbe2a05ccbd3a4a82174c18cf51b67ab7.image.disk
root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# veritysetup open --no-superblock --format=1 -s "" --hash=sha256 --data-block-size=512 --hash-block-size=4096 --data-blocks 10970 --hash-offset 5619712 /dev/loop100 erofs_verity /dev/loop100 618d2431d6799ea489c66b2cd94b578b570af0e43f3866bcc53320c8aceb1ec3
root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# mount -t erofs -o ro /dev/mapper/erofs_verity /mnt/
root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# ls /mnt/
bin  dev  etc  home  lib  lib64  root  tmp  usr  var

jiangliu · 2023-06-26T03:51:14Z

Configuration for above test:
vim /etc/nydus/config.toml

enable_tarfs = true
tarfs_hint = false
tarfs_max_concurrent_proc = 10
tarfs_export_mode = "image_block_with_verity"

pkg/tarfs/tarfs.go

snapshot/process.go

config/config.go

kevinXYin · 2023-08-01T11:07:50Z

Tested with some top images on dockerhub, compare rootfs generated by tarfs and overlayfs snapshotter for runc containers, there is no difference.

Tested-By Xin Yin

misc/snapshotter/config.toml

snapshot/snapshot.go

changweige · 2023-08-02T09:11:29Z

As this PR is quite large, I tried to review it a few times but they were not easy for me. So please forgive us for the slowwww procedure of reviewing it. :-)

imeoer · 2023-08-02T09:35:30Z

With this PR applied:
...

Maybe add this to a formal doc for the tarfs feature.

misc/snapshotter/config.toml

config/global.go

changweige · 2023-08-02T10:56:29Z

pkg/filesystem/config.go

+		if pm != nil {
+			switch pm.FsDriver {
+			case config.FsDriverBlockdev:
+				fs.blockdevManager = pm


The manager manages all the nydusd daemons which serve nydus image, but looks like tarfs does not need any nydusd daemon. Why there is a blockdev manager which never abord a nydusd daemon.

Though tarfs doesn't use daemon, it does use Manager to support:

cache space

recovery

metrics

I think we already have a dedicated cache manager which can work alone without a daemon manager.
As tarfs is an erofs mount, nydus-snapshotter does not have to recover any nydus daemon.
Tarfs is running as an Erofs mount without a nydusd daemon, I think there is no metric exported.

changweige

Thanks, maybe we need more work to do

Add 'enable_tarfs' option in config.toml, to enable run oci image with erofs tarfs. Snapshotter will download all image layers, uncompress to tar files and generate tarfs boostraps. Finally mount erofs based on loop device as the image read only rootfs. Signed-off-by: Xin Yin <[email protected]>

Add 'tarfs_max_concurrent_proc' option to set number of concurrent goroutines for tarfs blob processing. Signed-off-by: Xin Yin <[email protected]>

Support VIEW for tarfs snapshot. Signed-off-by: Xin Yin <[email protected]>

Syntax only changes: - fix some typos - rename manifest to manifestDigest Signed-off-by: Jiang Liu <[email protected]>

Limit maximum size of image manifest and config, to avoid DoS attack. Signed-off-by: Jiang Liu <[email protected]>

Only caches needed information according to manager configuration. Signed-off-by: Jiang Liu <[email protected]>

pkg/label/label.go

1) move loopdev setup from Prepare()/Merge() into Mount(), preparing for support of nodev mode in addition to blockdev. 2) store tar file in the global cache directory, mount tarfs at "mnt" instead of "tarfs", to keep consistence with fusedev/fscache drivers. 3) use tempfile/rename mechanism to easy error recover 4) use lock to avoid some possible race conditions 5) use mode 0750 instead of 0755 for new directories 6) associate a Manager with filesystems managed by blockdev Signed-off-by: Jiang Liu <[email protected]>

Remove function prepareRemoteSnapshot() to simplify the code and improve readability. Signed-off-by: Jiang Liu <[email protected]>

The overlay volatile option is only supported for OCIv1 images, so enhance nydus remote images to support it too. Signed-off-by: Jiang Liu <[email protected]>

Change if else to switch according to golint suggestions. Signed-off-by: Jiang Liu <[email protected]>

Enhance tarfs implementation to support following operations: - export an tarfs image as a block device - export an tarfs image as a block device with verity - export an tarfs layer as a block device - export an tarfs layer as a block device with verity Signed-off-by: Jiang Liu <[email protected]>

Simplify tarfs implementation by: 1) simplify the way to detech images without data blob 2) do not regenerate tarfs data when it already exists Signed-off-by: Jiang Liu <[email protected]>

Add documentation for tarfs. Signed-off-by: Jiang Liu <[email protected]>

Move ExtraOption to dedicated file and extend it for more usage cases. Signed-off-by: Jiang Liu <[email protected]>

changweige

LGTM~ Thanks

jiangliu force-pushed the tarfs branch 2 times, most recently from ff33840 to 2795e50 Compare June 24, 2023 17:21

jiangliu force-pushed the tarfs branch 3 times, most recently from a7dad0d to 7edc269 Compare June 26, 2023 03:42

jiangliu force-pushed the tarfs branch 4 times, most recently from 8067384 to 2eec840 Compare June 26, 2023 10:12

sctb512 reviewed Jun 27, 2023

View reviewed changes

jiangliu closed this Jun 27, 2023

jiangliu force-pushed the tarfs branch from 2eec840 to 88728c1 Compare June 27, 2023 15:48

jiangliu reopened this Jun 27, 2023

changweige mentioned this pull request Jun 29, 2023

Tarfs #497

Closed

jiangliu force-pushed the tarfs branch 5 times, most recently from e1ef00f to a89976f Compare June 30, 2023 06:25

jiangliu force-pushed the tarfs branch 2 times, most recently from f77d93b to 4e0db99 Compare July 19, 2023 03:57

changweige reviewed Aug 2, 2023

View reviewed changes

misc/snapshotter/config.toml Outdated Show resolved Hide resolved

misc/snapshotter/config.toml Outdated Show resolved Hide resolved

snapshot/snapshot.go Outdated Show resolved Hide resolved

changweige reviewed Aug 2, 2023

View reviewed changes

misc/snapshotter/config.toml Show resolved Hide resolved

changweige reviewed Aug 2, 2023

View reviewed changes

config/global.go Outdated Show resolved Hide resolved

changweige reviewed Aug 2, 2023

View reviewed changes

changweige requested changes Aug 2, 2023

View reviewed changes

ChengyuZhu6 mentioned this pull request Aug 10, 2023

Roadmap to develop image sharing on the host with integrity for CoCo confidential-containers/confidential-containers#142

Open

5 tasks

This was referenced Aug 18, 2023

runtime: Extend the extraOption structure kata-containers/kata-containers#7686

Closed

Extend the extraOption structure kata-containers/kata-containers#7687

Closed

This was referenced Aug 30, 2023

CC | image pulling in the guest without forked containerd kata-containers/kata-containers#7676

Merged

Roadmap to develop image pulling in the guest without forked containerd confidential-containers/confidential-containers#146

Closed

jiangliu force-pushed the tarfs branch 3 times, most recently from 4c9d88e to 34e8534 Compare August 31, 2023 01:44

Xin Yin and others added 6 commits August 31, 2023 09:51

pkg: add concurrent control for tarfs blob process

fba1c89

Add 'tarfs_max_concurrent_proc' option to set number of concurrent goroutines for tarfs blob processing. Signed-off-by: Xin Yin <[email protected]>

pkg: support VIEW for tarfs

ace7d56

Support VIEW for tarfs snapshot. Signed-off-by: Xin Yin <[email protected]>

tarfs: syntax only changes

91fb535

Syntax only changes: - fix some typos - rename manifest to manifestDigest Signed-off-by: Jiang Liu <[email protected]>

tarfs: limit maximum size of image manifest and config

47372c1

Limit maximum size of image manifest and config, to avoid DoS attack. Signed-off-by: Jiang Liu <[email protected]>

tarfs: only caches needed information

1eb7bf2

Only caches needed information according to manager configuration. Signed-off-by: Jiang Liu <[email protected]>

jiangliu force-pushed the tarfs branch from 34e8534 to 2f2f196 Compare August 31, 2023 01:52

changweige reviewed Aug 31, 2023

View reviewed changes

pkg/label/label.go Outdated Show resolved Hide resolved

changweige reviewed Aug 31, 2023

View reviewed changes

pkg/label/label.go Outdated Show resolved Hide resolved

jiangliu added 8 commits August 31, 2023 10:37

snapshot: remove function prepareRemoteSnapshot()

ea6f182

Remove function prepareRemoteSnapshot() to simplify the code and improve readability. Signed-off-by: Jiang Liu <[email protected]>

snapshot: support overlay volatile option for nydus

d90b73d

The overlay volatile option is only supported for OCIv1 images, so enhance nydus remote images to support it too. Signed-off-by: Jiang Liu <[email protected]>

snapshot: change if else to switch

2543daa

Change if else to switch according to golint suggestions. Signed-off-by: Jiang Liu <[email protected]>

tarfs: simplify tarfs implementation

d9c0b38

Simplify tarfs implementation by: 1) simplify the way to detech images without data blob 2) do not regenerate tarfs data when it already exists Signed-off-by: Jiang Liu <[email protected]>

tarfs: add documentation for tarfs

14d2417

Add documentation for tarfs. Signed-off-by: Jiang Liu <[email protected]>

snapshot: extend the ExtraOption struct

f3c1541

Move ExtraOption to dedicated file and extend it for more usage cases. Signed-off-by: Jiang Liu <[email protected]>

jiangliu force-pushed the tarfs branch from 2f2f196 to f3c1541 Compare August 31, 2023 02:37

changweige approved these changes Aug 31, 2023

View reviewed changes

changweige merged commit 9a2575b into containerd:main Aug 31, 2023
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refine tarfs overall design and implementation details #499

Refine tarfs overall design and implementation details #499

jiangliu commented Jun 24, 2023

codecov bot commented Jun 24, 2023 •

edited

Loading

jiangliu commented Jun 26, 2023

jiangliu commented Jun 26, 2023

kevinXYin commented Aug 1, 2023

changweige commented Aug 2, 2023

imeoer commented Aug 2, 2023

changweige Aug 2, 2023

jiangliu Aug 7, 2023

changweige Aug 8, 2023

changweige left a comment

changweige left a comment

Refine tarfs overall design and implementation details #499

Refine tarfs overall design and implementation details #499

Conversation

jiangliu commented Jun 24, 2023

codecov bot commented Jun 24, 2023 • edited Loading

Codecov Report

jiangliu commented Jun 26, 2023

jiangliu commented Jun 26, 2023

kevinXYin commented Aug 1, 2023

changweige commented Aug 2, 2023

imeoer commented Aug 2, 2023

changweige Aug 2, 2023

Choose a reason for hiding this comment

jiangliu Aug 7, 2023

Choose a reason for hiding this comment

changweige Aug 8, 2023

Choose a reason for hiding this comment

changweige left a comment

Choose a reason for hiding this comment

changweige left a comment

Choose a reason for hiding this comment

codecov bot commented Jun 24, 2023 •

edited

Loading