Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Validate CloudConfig using schema #305

Conversation

akutz
Copy link
Collaborator

@akutz akutz commented Dec 14, 2023

What does this PR do, and why is it needed?

This patch ensures the CloudConfig YAML produced by the inline CloudInit spec is validated against the official CloudConfig schema from Cloud-Init.

This patch also relocates the cloudinit utility package from ./pkg/vmprovider/providers/vsphere2/cloudinit to ./pkg/util/cloudinit. Since we do no anticipate future API revisions as a huge departure from what has been established in v1alpha2, it is safe to use those types in utility code as they can be find/replaced with later schema versions, ex. v1alpha3, when that time comes. Other than the reference to a github.com/vmware-tanzu/vm-operator/api/v1alpha2/cloudinit.CloudConfig, the code in ./pkg/util/cloudinit is completely unrelated to the vSphere provider, hence its relocation under the utility package.

Which issue(s) is/are addressed by this PR? (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

Fixes NONE

Are there any special notes for your reviewer:

I considered using the validation logic to also verify the raw cloud config, but I elected to not do this for the following reasons:

  1. The validation is strict. If there are unknown fields, it will fail.
  2. We should allow the users to use the raw cloud config to pass in data we may consider to be invalid, but may work for a given guest.
  3. The schema is a moving target, and so we only update it with new VM Op releases.

In other words, the raw cloud config is not just a means to send in data that our version of the schema may consider to be invalid, but it's a way to ensure users have a way to bypass validation entirely if that is what they want to do.

Please add a release note if necessary:

Validate the inline CloudConfig using the CloudConfig schema.

@github-actions github-actions bot added testing-needed-e2e-fast size/XXL Denotes a PR that changes 1000+ lines. labels Dec 14, 2023
@akutz akutz force-pushed the feature/validate-marshaled-cloud-init-userdata-against-schema branch 2 times, most recently from 4042057 to 9b571fc Compare December 14, 2023 17:30
@akutz akutz requested a review from bryanv December 14, 2023 17:33
@akutz akutz force-pushed the feature/validate-marshaled-cloud-init-userdata-against-schema branch 2 times, most recently from 0a934e7 to 033bd91 Compare December 14, 2023 21:23
@akutz akutz force-pushed the feature/validate-marshaled-cloud-init-userdata-against-schema branch from 033bd91 to 639f834 Compare December 15, 2023 16:14
This patch ensures the CloudConfig YAML produced by the inline
CloudInit spec is validated against the official CloudConfig
schema from Cloud-Init.

This patch also relocates the `cloudinit` utility package from
`./pkg/vmprovider/providers/vsphere2/cloudinit` to
`./pkg/util/cloudinit`. Since we do no anticipate future API
revisions as a _huge_ departure from what has been established
in v1alpha2, it is safe to use those types in utility code as
they can be find/replaced with later schema versions,
ex. `v1alpha3`, when that time comes. Other than the reference
to a `github.com/vmware-tanzu/vm-operator/api/v1alpha2/cloudinit.CloudConfig`,
the code in `./pkg/util/cloudinit` is completely unrelated to
the vSphere provider, hence its relocation under the utility
package.
@akutz akutz force-pushed the feature/validate-marshaled-cloud-init-userdata-against-schema branch from 639f834 to 5709ca5 Compare December 15, 2023 16:29
@akutz
Copy link
Collaborator Author

akutz commented Dec 15, 2023

Merging as there were successful integration test runs already, and the last two have failed due to an unrelated image flake and the second a race condition.

@akutz akutz merged commit a97829f into vmware-tanzu:main Dec 15, 2023
14 of 15 checks passed
Copy link

Code Coverage

Package Line Rate Health
github.com/vmware-tanzu/vm-operator/api/utilconversion 91%
github.com/vmware-tanzu/vm-operator/api/v1alpha1 27%
github.com/vmware-tanzu/vm-operator/api/v1alpha2 16%
github.com/vmware-tanzu/vm-operator/api/v1alpha2/cloudinit 0%
github.com/vmware-tanzu/vm-operator/api/v1alpha2/common 0%
github.com/vmware-tanzu/vm-operator/api/v1alpha2/sysprep 0%
github.com/vmware-tanzu/vm-operator/controllers/contentlibrary/v1alpha1/clustercontentlibraryitem 83%
github.com/vmware-tanzu/vm-operator/controllers/contentlibrary/v1alpha1/contentlibraryitem 82%
github.com/vmware-tanzu/vm-operator/controllers/contentlibrary/v1alpha1/contentsource 88%
github.com/vmware-tanzu/vm-operator/controllers/contentlibrary/v1alpha1/utils 95%
github.com/vmware-tanzu/vm-operator/controllers/contentlibrary/v1alpha2/clustercontentlibraryitem 86%
github.com/vmware-tanzu/vm-operator/controllers/contentlibrary/v1alpha2/contentlibraryitem 85%
github.com/vmware-tanzu/vm-operator/controllers/contentlibrary/v1alpha2/utils 91%
github.com/vmware-tanzu/vm-operator/controllers/infracluster 75%
github.com/vmware-tanzu/vm-operator/controllers/infraprovider 75%
github.com/vmware-tanzu/vm-operator/controllers/providerconfigmap 74%
github.com/vmware-tanzu/vm-operator/controllers/util/encoding 73%
github.com/vmware-tanzu/vm-operator/controllers/util/remote 41%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine/v1alpha1 45%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine/v1alpha2 70%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineclass/v1alpha1 31%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineclass/v1alpha2 22%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinepublishrequest/v1alpha1 83%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinepublishrequest/v1alpha2 81%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineservice/v1alpha1 81%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineservice/v1alpha1/providers 96%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineservice/v1alpha1/providers/simplelb 66%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineservice/v1alpha1/utils 83%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineservice/v1alpha2 84%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineservice/v1alpha2/providers 96%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineservice/v1alpha2/providers/simplelb 66%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineservice/v1alpha2/utils 84%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinesetresourcepolicy/v1alpha1 80%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinesetresourcepolicy/v1alpha2 80%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinewebconsolerequest/v1alpha1 74%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinewebconsolerequest/v1alpha2 72%
github.com/vmware-tanzu/vm-operator/controllers/volume/v1alpha1 86%
github.com/vmware-tanzu/vm-operator/controllers/volume/v1alpha2 86%
github.com/vmware-tanzu/vm-operator/pkg/builder 73%
github.com/vmware-tanzu/vm-operator/pkg/conditions 90%
github.com/vmware-tanzu/vm-operator/pkg/conditions2 92%
github.com/vmware-tanzu/vm-operator/pkg/context 0%
github.com/vmware-tanzu/vm-operator/pkg/context/fake 100%
github.com/vmware-tanzu/vm-operator/pkg/lib 85%
github.com/vmware-tanzu/vm-operator/pkg/manager 78%
github.com/vmware-tanzu/vm-operator/pkg/metrics 90%
github.com/vmware-tanzu/vm-operator/pkg/metrics2 85%
github.com/vmware-tanzu/vm-operator/pkg/patch 79%
github.com/vmware-tanzu/vm-operator/pkg/patch2 78%
github.com/vmware-tanzu/vm-operator/pkg/prober 92%
github.com/vmware-tanzu/vm-operator/pkg/prober/context 100%
github.com/vmware-tanzu/vm-operator/pkg/prober/fake 85%
github.com/vmware-tanzu/vm-operator/pkg/prober/fake/probe 83%
github.com/vmware-tanzu/vm-operator/pkg/prober/fake/worker 88%
github.com/vmware-tanzu/vm-operator/pkg/prober/probe 83%
github.com/vmware-tanzu/vm-operator/pkg/prober/worker 86%
github.com/vmware-tanzu/vm-operator/pkg/prober2 91%
github.com/vmware-tanzu/vm-operator/pkg/prober2/context 100%
github.com/vmware-tanzu/vm-operator/pkg/prober2/fake 85%
github.com/vmware-tanzu/vm-operator/pkg/prober2/fake/probe 83%
github.com/vmware-tanzu/vm-operator/pkg/prober2/fake/worker 89%
github.com/vmware-tanzu/vm-operator/pkg/prober2/probe 92%
github.com/vmware-tanzu/vm-operator/pkg/prober2/worker 80%
github.com/vmware-tanzu/vm-operator/pkg/record 89%
github.com/vmware-tanzu/vm-operator/pkg/topology 85%
github.com/vmware-tanzu/vm-operator/pkg/util 85%
github.com/vmware-tanzu/vm-operator/pkg/util/cloudinit 90%
github.com/vmware-tanzu/vm-operator/pkg/util/cloudinit/schema 25%
github.com/vmware-tanzu/vm-operator/pkg/util/cloudinit/validate 92%
github.com/vmware-tanzu/vm-operator/pkg/util/kube 89%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/vm 74%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/fake 74%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere 71%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/client 49%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/clustermodules 85%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/config 85%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/contentlibrary 71%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/credentials 100%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/instancestorage 92%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/internal 0%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/network 88%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/placement 83%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/resources 47%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session 84%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/storage 77%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/test 98%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/vcenter 80%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/virtualmachine 83%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2 69%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/client 49%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/clustermodules 85%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/config 70%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/contentlibrary 72%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/credentials 100%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/instancestorage 92%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/internal 0%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/network 85%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/placement 70%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/resources 26%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/session 83%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/storage 77%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/test 98%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/vcenter 80%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/virtualmachine 84%
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere2/vmlifecycle 75%
github.com/vmware-tanzu/vm-operator/pkg/webconsolevalidation 53%
github.com/vmware-tanzu/vm-operator/webhooks/common 100%
github.com/vmware-tanzu/vm-operator/webhooks/persistentvolumeclaim/validation 95%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachine/v1alpha1/mutation 81%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachine/v1alpha1/validation 94%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachine/v1alpha2/mutation 81%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachine/v1alpha2/validation 95%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineclass/v1alpha1/mutation 59%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineclass/v1alpha1/validation 89%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineclass/v1alpha2/mutation 62%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineclass/v1alpha2/validation 89%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinepublishrequest/v1alpha1/validation 92%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinepublishrequest/v1alpha2/validation 92%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineservice/v1alpha1/mutation 62%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineservice/v1alpha1/validation 91%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineservice/v1alpha2/mutation 67%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineservice/v1alpha2/validation 91%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinesetresourcepolicy/v1alpha1/mutation 62%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinesetresourcepolicy/v1alpha1/validation 89%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinesetresourcepolicy/v1alpha2/validation 89%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinewebconsolerequest/v1alpha1/validation 92%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinewebconsolerequest/v1alpha2/validation 92%
Summary 67% (12950 / 19288)

Minimum allowed line rate is 60%

@akutz akutz deleted the feature/validate-marshaled-cloud-init-userdata-against-schema branch December 15, 2023 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-not-required size/XXL Denotes a PR that changes 1000+ lines. testing-needed-e2e-fast
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants