Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: test CRD versions #1317

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

m-ildefons
Copy link
Contributor

Add test which checks if the CRD versions defined in the cluster match the versions in the subcharts in the git repo.
This test ensures that the subcharts in the release version don't unexpectedly change their API versions within a minor Harvester release.

To implement this test, additional infrastructure for the Kubernetes API has been added.

fixes: #1314

@m-ildefons m-ildefons requested a review from albinsun June 10, 2024 10:37
@m-ildefons m-ildefons self-assigned this Jun 10, 2024
@m-ildefons m-ildefons force-pushed the wip/test-crd-versions branch 3 times, most recently from 943710d to 26cc904 Compare June 10, 2024 10:59
Add test which checks if the CRD versions defined in the cluster match
the versions in the subcharts in the git repo.
This test ensures that the subcharts in the release version don't
unexpectedly change their API versions within a minor Harvester release.

To implement this test, additional infrastructure for the Kubernetes API
has been added.

fixes: harvester#1314

Signed-off-by: Moritz Röhrich <[email protected]>
Copy link
Member

@lanfon72 lanfon72 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, the library apiclient is focusing for Harvester and related endpoint, we don't and will not have plan to implement kube APIs, as it already have many other libs does.

and as Albin's snippets, it shows we can simply get those CRD info from bash, so I would prefer use that version rather than create a new API set.

It would be more readable and fit our manual test steps.

and I don't think we need to verify those CRDs every time, usually we will only verify them after the cluster upgraded into new version, so we will not place then in apis.

assert code == 200
assert data.get("value") is not None

yield semver.VersionInfo.parse(data.get("value").lstrip("v"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already have apiclient.cluster_version to expose the cluster's version.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The apiclient.cluster_version returns a pkg_resources.Version object, but my tests are using the semver library to parse versioning information.
The semver library has several advantages. One being that it isn't in the process of being deprecated. Another advantage is that it has good documentation and deals with a lot of different operations on version information better. Third, the pkg_resources library is supposed to deal with python-packages and not generic version information. This is a difference in so far as python packages have specific requirements on the format of the version string that we don't adhere to with Harvester

@albinsun
Copy link
Contributor

albinsun commented Jun 17, 2024

Test target and classification are important, we should not mix different purpose test code.
However we also want to automate tests rather than manual, (of course we can keep both of them for specific timing of test.)

Or we can have this automation in another folder or something thus not break test organization? How do you guys think?

@m-ildefons
Copy link
Contributor Author

IMO, the library apiclient is focusing for Harvester and related endpoint, we don't and will not have plan to implement kube APIs, as it already have many other libs does.

Well, it does not. There's also a Rancher API client part in it.
I also don't just need a generic K8s client, I need a K8s client library where I can just give it a Harvester endpoint, username and password and it deals with the authentication. This isn't the case for normal K8s API client libraries (as you can tell, because I've had to implement that part).

and as Albin's snippets, it shows we can simply get those CRD info from bash, so I would prefer use that version rather than create a new API set.

It would be more readable and fit our manual test steps.

Please no. We have a test suite, let's not add a second test suite.
Bash is also a terrible choice for maintaining any significant amount of code, since it quickly becomes very hard to read.

and I don't think we need to verify those CRDs every time, usually we will only verify them after the cluster upgraded into new version, so we will not place then in apis.

I see your point, but I don't see where else in the test suite an appropriate place would be. Maybe we should create a separate directory for upgrade tests under harvester_e2e_tests/upgrade?

@lanfon72
Copy link
Member

IMO, the library apiclient is focusing for Harvester and related endpoint, we don't and will not have plan to implement kube APIs, as it already have many other libs does.

Well, it does not. There's also a Rancher API client part in it. I also don't just need a generic K8s client, I need a K8s client library where I can just give it a Harvester endpoint, username and password and it deals with the authentication. This isn't the case for normal K8s API client libraries (as you can tell, because I've had to implement that part).

and as Albin's snippets, it shows we can simply get those CRD info from bash, so I would prefer use that version rather than create a new API set.
It would be more readable and fit our manual test steps.

Please no. We have a test suite, let's not add a second test suite. Bash is also a terrible choice for maintaining any significant amount of code, since it quickly becomes very hard to read.

and I don't think we need to verify those CRDs every time, usually we will only verify them after the cluster upgraded into new version, so we will not place then in apis.

I see your point, but I don't see where else in the test suite an appropriate place would be. Maybe we should create a separate directory for upgrade tests under harvester_e2e_tests/upgrade?

  1. We will not implement all Rancher APIs, we would only implement those parts related to Rancher Integration, so the RancherAPI is not be included by default. and we have VCluster addon to enable embedded Rancher, so it is make sense to have the API client.
  2. This repo is for test automation, not a product to modify frequently. For test automation, readability, simply and manually reproducible are important than others. That's why I would prefer to have a bash version and not include another library, less dependencies would let us more easily to address the root cause when test fails.
  3. Ideally, we will check CRD versions after cluster upgraded, so the automation test cases should be placed into upgrade.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[TEST] [Release Testing] Test installed CRD API versions of system charts
3 participants