Expose virtual size of qcow2 backing images #208

tserong · 2024-02-29T11:27:42Z

Which issue(s) this PR fixes:

What this PR does / why we need it:

This commit adds VirtualSize to the BackingImage, BackingImageStatus, BackingImageSpec and FileInfo structs. This will allow longhorn-manager to in turn pick up the virtual image size and expose it in the status section of the BackingImage and BackingImageManager CRDs.

The virtual size of an image is determined by running qemu-img info. For qcow2 images, the virtual size will usually be larger than the actual physical file size. For raw images, qemu-img info still reports virtual size, but it's the same as the physical file size in this case.

It's important to note that we can only report the virtual size of an image once the syncing file is ready. If the syncing file is not yet ready (or if for some reason the call to qemu-img info fails), virtual size will be set to zero.

Special notes for your reviewer:

In order to actually be useful, this requires matching changes in longhorn-manager. I'm still working on that piece :-)

My implementation of util.GetImageVirtualSize() uses qemu-img info --output=json then unmarshalls the JSON to extract the virtual size. There's a similar existing function - util.DetectFileFormat() - which instead parses the human readable output using bufio.NewScanner, but IMO the JSON form makes for simpler code. I wasn't sure if I should mess with util.DetectFileFormat() while working on this PR, so I left it alone.

Note that there's an extra commit in here to fix make when run locally (see the commit message for details). I can split that out into a separate PR if you'd prefer.

Additional documentation or context

Here's some sample output from backing-image-manager backing-image list with this change applied:

A ~955MiB (1,001,652,224 bytes) qcow2 image with a virtualSize of 20GiB (21,474,836,480 bytes):

"default-image-gjm5l": {
	"name": "default-image-gjm5l",
	"uuid": "c5ba7ef9",
	"size": 1001652224,
	"virtualSize": 21474836480,
	"expectedChecksum": "1bbfc258229011fc4be34d35e823dab72431f577de5027d88f9236bbb8b786f25ca976422dfb55a02194215b0b717fbba033516c2b8c32b46c6587a580aaa748",
	"status": {
		"state": "ready",
		"currentChecksum": "1bbfc258229011fc4be34d35e823dab72431f577de5027d88f9236bbb8b786f25ca976422dfb55a02194215b0b717fbba033516c2b8c32b46c6587a580aaa748",
		"sendingReference": 0,
		"errorMsg": "",
		"senderManagerAddress": "",
		"progress": 100
	}
}

A ~13.5GiB (14,548,992,000 bytes) ISO (raw) image, while syncing. Note that virtualSize is zero here because we don't know it yet:

"default-image-87b7d": {
	"name": "default-image-87b7d",
	"uuid": "aa95cadc",
	"size": 14548992000,
	"virtualSize": 0,
	"expectedChecksum": "88a21ac07b673bd60fd8733e6dd5d93444651fd4e806c9922904b431d75b90aa1bac7da4553ffff3a8badc64b0be44e41f5155961fcc565aa2b0f5045862a367",
	"status": {
		"state": "in-progress",
		"currentChecksum": "",
		"sendingReference": 0,
		"errorMsg": "",
		"senderManagerAddress": "",
		"progress": 13
	}
},

The same image once sync is complete. Note that size and virtualSize are identical now:

"default-image-87b7d": {
	"name": "default-image-87b7d",
	"uuid": "aa95cadc",
	"size": 14548992000,
	"virtualSize": 14548992000,
	"expectedChecksum": "88a21ac07b673bd60fd8733e6dd5d93444651fd4e806c9922904b431d75b90aa1bac7da4553ffff3a8badc64b0be44e41f5155961fcc565aa2b0f5045862a367",
	"status": {
		"state": "ready",
		"currentChecksum": "88a21ac07b673bd60fd8733e6dd5d93444651fd4e806c9922904b431d75b90aa1bac7da4553ffff3a8badc64b0be44e41f5155961fcc565aa2b0f5045862a367",
		"sendingReference": 0,
		"errorMsg": "",
		"senderManagerAddress": "",
		"progress": 100
	}
},

tserong · 2024-03-01T02:09:38Z

Force push to make the 512 byte padding in createQcow2File() work properly regardless of the size of the image file.

shuo-wu

LGTM

pkg/sync/sync_file.go

tserong · 2024-03-07T07:55:03Z

(rebased on latest master)

pkg/util/util.go

This manually applies the relevant changes from longhorn/backing-image-manager#208 to vendor/github.com/longhorn/backing-image-manager because using `go get ; go mod tidy ; go mod vendor` pulls in a giant mess of other stuff that's changed in the meantime :-/ Signed-off-by: Tim Serong <[email protected]>

tserong · 2024-03-18T03:20:49Z

(rebased on latest master)

This manually applies the relevant changes from longhorn/backing-image-manager#208 and longhorn/longhorn-manager#2680 to vendor/github.com/longhorn/backing-image-manager and vendor/github.com/longhorn/longhorn-manager because using `go get ; go mod tidy ; go mod vendor` pulls in a giant mess of other stuff that's changed in the meantime :-/ Signed-off-by: Tim Serong <[email protected]>

This requires: - longhorn/backing-image-manager#208 - longhorn/longhorn-manager#2680 With this change, we can do this: ``` NAME DISPLAY-NAME SIZE VIRTUALSIZE AGE image-26r4s sles15-sp5-minimal-vm.x86_64-kvm-and-xen-qu1.qcow2 267845632 25769803776 6m36s image-bgnhb sle-micro.x86_64-5.5.0-default-qcow-gm.qcow2 1001652224 21474836480 20m image-nmmvj sle-15-sp5-full-x86_64-gm-media1.iso 14548992000 14548992000 19m ``` Related issue: harvester#4905 Signed-off-by: Tim Serong <[email protected]>

This requires: - longhorn/backing-image-manager#208 - longhorn/longhorn-manager#2680 With this change, we can do this: # kubectl get vmimages NAME DISPLAY-NAME SIZE VIRTUALSIZE AGE image-26r4s sles15-sp5-minimal-vm.x86_64-kvm-and-xen-qu1.qcow2 267845632 25769803776 6m36s image-bgnhb sle-micro.x86_64-5.5.0-default-qcow-gm.qcow2 1001652224 21474836480 20m image-nmmvj sle-15-sp5-full-x86_64-gm-media1.iso 14548992000 14548992000 19m Related issue: harvester#4905 Signed-off-by: Tim Serong <[email protected]>

shuo-wu

LGTM

pkg/util/util.go

This requires: - longhorn/backing-image-manager#208 - longhorn/longhorn-manager#2680 With this change, we can do this: # kubectl get vmimages NAME DISPLAY-NAME SIZE VIRTUALSIZE AGE image-26r4s sles15-sp5-minimal-vm.x86_64-kvm-and-xen-qu1.qcow2 267845632 25769803776 6m36s image-bgnhb sle-micro.x86_64-5.5.0-default-qcow-gm.qcow2 1001652224 21474836480 20m image-nmmvj sle-15-sp5-full-x86_64-gm-media1.iso 14548992000 14548992000 19m Related issue: harvester#4905 Signed-off-by: Tim Serong <[email protected]>

innobead · 2024-03-19T11:43:01Z

@ChanYiLin Please review this.

When doing local builds (i.e. without $DRONE_REPO and $DRONE_PULL_REQUEST, or $DRONE_COMMIT_REF set), the build fails with "fatal: Needed a single revision" in `scripts/validate`: ``` > make REPO=tserong [...] Running validation Running: go vet refs/heads/wip-add-virtualSize fatal: Needed a single revision FATA[0032] exit status 128 make: *** [Makefile:11: ci] Error 1 ``` If I run the commands in `scripts/validate` by hand, I see this: ``` > git symbolic-ref -q HEAD && REV="origin/HEAD" || REV="HEAD^" refs/heads/wip-add-virtualSize > echo $REV origin/HEAD > headSHA=$(git rev-parse --short=12 ${REV}) fatal: Needed a single revision ``` I don't know if this is just something weird about my environment, but if I change "origin/HEAD" to "HEAD", it works fine: ``` > headSHA=$(git rev-parse --short=12 HEAD) > echo $headSHA eb27fbf6e678 ``` Signed-off-by: Tim Serong <[email protected]>

This commit adds VirtualSize to the BackingImage, BackingImageStatus, BackingImageSpec and FileInfo structs. This will allow longhorn-manager to in turn pick up the virtual image size and expose it in the status section of the BackingImage and BackingImageManager CRDs. The virtual size of an image is determined by running `qemu-img info`. For qcow2 images, the virtual size will usually be larger than the actual physical file size. For raw images, `qemu-img info` still reports virtual size, but it's the same as the physical file size in this case. It's important to note that we can only report the virtual size of an image once the syncing file is ready. If the syncing file is not yet ready (or if for some reason the call to `qemu-img info` fails), virtual size will be set to zero. Related issue: longhorn/longhorn#7923 Signed-off-by: Tim Serong <[email protected]>

Signed-off-by: Tim Serong <[email protected]>

ChanYiLin

LGTM, thanks for the contribution

shuo-wu · 2024-03-26T18:31:00Z

@mergify backport v1.5.x v1.6.x

mergify · 2024-03-26T18:31:19Z

backport v1.5.x v1.6.x

✅ Backports have been created

#220 Expose virtual size of qcow2 backing images (backport #208) has been created for branch v1.5.x but encountered conflicts
#221 Expose virtual size of qcow2 backing images (backport #208) has been created for branch v1.6.x

shuo-wu · 2024-03-26T18:31:50Z

Would you help update longhorn-manager and longhorn-instance-manager to fit this change?

tserong · 2024-03-27T00:27:38Z

Sure, will do.

tserong · 2024-03-27T07:40:35Z

The longhorn-manager PR is longhorn/longhorn-manager#2680.

I'm not sure yet what needs to change in longhorn-instance-manager as I can't find any references to backing-image-manager in the longhorn-instance-manager source. Can you please advise?

shuo-wu · 2024-03-27T19:08:51Z

I can't find any references to backing-image-manager in the longhorn-instance-manager source. Can you please advise?

I just re-checked the code. You are right there is no reference in longhorn-instance-manager now. Please ignore this.

This is necessary to bring in support for backing image virtual size: - longhorn/backing-image-manager#208 - longhorn/longhorn-manager#2680 Signed-off-by: Tim Serong <[email protected]>

This is necessary to bring in support for backing image virtual size: - longhorn/backing-image-manager#208 - longhorn/longhorn-manager#2680 Note the the longhorn chart version in dependency_charts/longhorn is v1.6.0-dev because that's what it's set to at the time of writing in the upstream longhorn git repo, but the tags for the various longhorn images are all "master-head". IMO we really shouldn't take this change until we at least get a proper v1.7.0 RC or dev tag release. Signed-off-by: Tim Serong <[email protected]>

With this change, we can do this: # kubectl get vmimages NAME DISPLAY-NAME SIZE VIRTUALSIZE AGE image-26r4s sles15-sp5-minimal-vm.x86_64-kvm-and-xen-qu1.qcow2 267845632 25769803776 6m36s image-bgnhb sle-micro.x86_64-5.5.0-default-qcow-gm.qcow2 1001652224 21474836480 20m image-nmmvj sle-15-sp5-full-x86_64-gm-media1.iso 14548992000 14548992000 19m This in turn means we can update the UI to take image virtual size into account when creating VMs. Note that this requires the following Longhorn PRs, which have been backported to longhorn 1.5.5 and 1.6.2: - longhorn/backing-image-manager#208 - longhorn/longhorn-manager#2680 - longhorn/longhorn#8267 Related issue: harvester#4905 Signed-off-by: Tim Serong <[email protected]>

With this change, we can do this: # kubectl get vmimages NAME DISPLAY-NAME SIZE VIRTUALSIZE AGE image-26r4s sles15-sp5-minimal-vm.x86_64-kvm-and-xen-qu1.qcow2 267845632 25769803776 6m36s image-bgnhb sle-micro.x86_64-5.5.0-default-qcow-gm.qcow2 1001652224 21474836480 20m image-nmmvj sle-15-sp5-full-x86_64-gm-media1.iso 14548992000 14548992000 19m This in turn means we can update the UI to take image virtual size into account when creating VMs. Note that this requires the following Longhorn PRs, which have been backported to longhorn 1.5.5 and 1.6.2: - longhorn/backing-image-manager#208 - longhorn/longhorn-manager#2680 - longhorn/longhorn#8267 Related issue: #4905 Signed-off-by: Tim Serong <[email protected]>

With this change, we can do this: # kubectl get vmimages NAME DISPLAY-NAME SIZE VIRTUALSIZE AGE image-26r4s sles15-sp5-minimal-vm.x86_64-kvm-and-xen-qu1.qcow2 267845632 25769803776 6m36s image-bgnhb sle-micro.x86_64-5.5.0-default-qcow-gm.qcow2 1001652224 21474836480 20m image-nmmvj sle-15-sp5-full-x86_64-gm-media1.iso 14548992000 14548992000 19m This in turn means we can update the UI to take image virtual size into account when creating VMs. Note that this requires the following Longhorn PRs, which have been backported to longhorn 1.5.5 and 1.6.2: - longhorn/backing-image-manager#208 - longhorn/longhorn-manager#2680 - longhorn/longhorn#8267 Related issue: harvester#4905 Signed-off-by: Tim Serong <[email protected]> (cherry picked from commit 79f6cdc)

tserong requested a review from a team as a code owner February 29, 2024 11:27

tserong force-pushed the wip-add-virtualSize branch from e249fc0 to 0e0854e Compare March 1, 2024 02:08

innobead requested review from ChanYiLin and shuo-wu March 6, 2024 02:10

shuo-wu previously approved these changes Mar 6, 2024

View reviewed changes

pkg/sync/sync_file.go Outdated Show resolved Hide resolved

tserong force-pushed the wip-add-virtualSize branch from 0e0854e to a8b08c1 Compare March 7, 2024 07:53

tserong dismissed shuo-wu’s stale review via eca5852 March 8, 2024 01:55

tserong force-pushed the wip-add-virtualSize branch from a8b08c1 to eca5852 Compare March 8, 2024 01:55

shuo-wu reviewed Mar 8, 2024

View reviewed changes

pkg/util/util.go Outdated Show resolved Hide resolved

tserong mentioned this pull request Mar 8, 2024

Expose virtual size of backing images longhorn/longhorn-manager#2680

Merged

tserong force-pushed the wip-add-virtualSize branch from a2a1c0e to 4f3b4c7 Compare March 18, 2024 03:20

tserong mentioned this pull request Mar 18, 2024

Expose virtual size of VM images harvester/harvester#5387

Merged

shuo-wu previously approved these changes Mar 19, 2024

View reviewed changes

pkg/util/util.go Show resolved Hide resolved

tserong dismissed shuo-wu’s stale review via c4c63a1 March 19, 2024 01:20

tserong force-pushed the wip-add-virtualSize branch from 4f3b4c7 to c4c63a1 Compare March 19, 2024 01:20

tserong added 3 commits March 20, 2024 10:16

Consolidate GetImageVirtualSize and DetectFileFormat functions

f08df28

Signed-off-by: Tim Serong <[email protected]>

tserong force-pushed the wip-add-virtualSize branch from c4c63a1 to f08df28 Compare March 19, 2024 23:16

shuo-wu approved these changes Mar 20, 2024

View reviewed changes

ChanYiLin approved these changes Mar 26, 2024

View reviewed changes

mergify bot merged commit c5288d7 into longhorn:master Mar 26, 2024
6 checks passed

tserong deleted the wip-add-virtualSize branch March 27, 2024 05:11

shuo-wu mentioned this pull request Apr 9, 2024

[IMPROVEMENT] Expose virtual size of qcow2 backing images longhorn/longhorn#7923

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose virtual size of qcow2 backing images #208

Expose virtual size of qcow2 backing images #208

tserong commented Feb 29, 2024

tserong commented Mar 1, 2024

shuo-wu left a comment

tserong commented Mar 7, 2024

tserong commented Mar 18, 2024

shuo-wu left a comment

innobead commented Mar 19, 2024

ChanYiLin left a comment

shuo-wu commented Mar 26, 2024

mergify bot commented Mar 26, 2024 •

edited

Loading

shuo-wu commented Mar 26, 2024

tserong commented Mar 27, 2024

tserong commented Mar 27, 2024 •

edited

Loading

shuo-wu commented Mar 27, 2024

Expose virtual size of qcow2 backing images #208

Expose virtual size of qcow2 backing images #208

Conversation

tserong commented Feb 29, 2024

Which issue(s) this PR fixes:

What this PR does / why we need it:

Special notes for your reviewer:

Additional documentation or context

tserong commented Mar 1, 2024

shuo-wu left a comment

Choose a reason for hiding this comment

tserong commented Mar 7, 2024

tserong commented Mar 18, 2024

shuo-wu left a comment

Choose a reason for hiding this comment

innobead commented Mar 19, 2024

ChanYiLin left a comment

Choose a reason for hiding this comment

shuo-wu commented Mar 26, 2024

mergify bot commented Mar 26, 2024 • edited Loading

✅ Backports have been created

shuo-wu commented Mar 26, 2024

tserong commented Mar 27, 2024

tserong commented Mar 27, 2024 • edited Loading

shuo-wu commented Mar 27, 2024

mergify bot commented Mar 26, 2024 •

edited

Loading

tserong commented Mar 27, 2024 •

edited

Loading