docs/DEVELOPMENT: document the CI #224

wainersm · 2023-07-07T19:20:57Z

Created a new "Continuous Integration (CI)" section in the developer's guide to explain the e2e CI jobs types, how/who can execute them as well as how to run some of them locally.

Fixes #113
Signed-off-by: Wainer dos Santos Moschetta [email protected]

katexochen · 2023-07-08T06:26:06Z

docs/DEVELOPMENT.md

+```
+
+Notice that the `-r` parameter passed to `run-local.sh` above specifies the runtimeClass to be tested. You can switch to, for example, `kata-clh` to test Cloud Hypervisor. See the script's help (`run-local.sh -h`) for further details.
+


Suggested change

### Running e2e test locally with Vagrant

katexochen · 2023-07-08T06:27:22Z

docs/DEVELOPMENT.md

+
+We recommend that you run the e2e Non-TEE tests on your local machine before opening a PR to check your changes will not break the CI so to avoid wasting resources. You can also use the approach describe below to debug and fix failures, or test changes on the scripts themselves.
+
+The entry point script is [tests/e2e/run-local.sh/run-local.sh](../tests/e2e/run-local.sh). It is going to install softwares and change the system's configuration, so we recommend that you run the e2e tests on VMs.


Suggested change

The entry point script is [tests/e2e/run-local.sh/run-local.sh](../tests/e2e/run-local.sh). It is going to install softwares and change the system's configuration, so we recommend that you run the e2e tests on VMs.

The entry point script is [tests/e2e/run-local.sh](../tests/e2e/run-local.sh). It is going to install software and change the system's configuration, so we recommend that you run the e2e tests on VMs.

Maybe we could add what the requirements are for the underlying system/VM, like min. CPU/Mem and nested virtualization support for VMs.

katexochen · 2023-07-08T07:45:11Z

docs/DEVELOPMENT.md

+sudo apt-get install -y ansible python-is-python3
+cd tests/e2e
+export PATH="$PATH:/usr/local/bin"
+./run-local.sh -r "kata-qemu"


Suggested change

./run-local.sh -r "kata-qemu"

./run-local.sh -u -r "kata-qemu"

With the recent addition it should IMO stay without the -u

wainersm · 2023-07-10T13:40:46Z

hey @katexochen , thanks for your review!

Updated this PR to address his comments:

Added mininum requirements for the VM
Explained the '-u' flag
Added the 'Running e2e test locally with Vagrant' section

katexochen

Great, thanks! :)

fitzthum

Looks good. I think there are two things missing.

First, when you run the tests locally, how do you point them to your changes? If you follow the instructions here you will just run the tests against the upstream code which isn't super useful (although it's probably a good idea to do that before trying your changes). So a section about how to update the Kustomize files to point to your own bundle (either a diffrent one from Kata upstream or one you built yourself) would be nice. Also maybe a note about how to point the tests-runner.sh script to a different version of the tests.

Second, one tip that I might add is that when you run run-local.sh it will setup your node and run things. Once you've done this you can run tests-runner.sh (with the -r flag) as many times as you want to rerun the tests (as long as you didn't use the -u flag when you set things up). Ofc this won't pickup any changes you make to the bundle, but it can pickup changes to the tests and you can make changes locally. In fact my main way of debugging serious problems is to run tests-runner.sh again and again while adjust components manually on the node.

docs/DEVELOPMENT.md

fidencio · 2023-08-29T16:11:48Z

@wainersm, aren't there secrets stored on jenkins that are needed for running kata-containers related tests?
If so, shall we expose those here so those tests do not fail when running locally?

ldoktor · 2023-11-28T08:01:16Z

docs/DEVELOPMENT.md

+
+## Continuous Integration (CI)
+
+In order to be merged your opened pull request (PR) should pass the static analysis checks and end-to-end (e2e) tests.


The e2e testing around kubernetes is often referred to kubernetes e2e conformance tests. How about using the term coco operator functional testing or such? Anyway not as part of this PR... (it would require changing the references in tests/e2e but I think it could improve the first impression for newcomers)

ldoktor · 2023-11-28T12:42:02Z

docs/DEVELOPMENT.md

+
+```shell
+export RUNTIMECLASS="kata-qemu"
+vagrant up tests-e2e-ubuntu2004


This works quite well, syncs everything, performs all the steps and reports correct exit status (so, unlike when executed manually, there is a big fat warning it failed), but it takes forever to execute. Still I think it'd be better to move this above the manual steps to give users single entrypoint they could run. Anyway there should be a way to speedup the deployment, perhaps with the good old deploy/snapshot/run/run/run/delete fashion.

43m on my machine, perhaps worth mentioning here as well

Mentioned that on the guide.

wainersm · 2023-12-11T18:40:56Z

@ldoktor @fitzthum update the CI guide to address most of the comments you made. The name of the jobs currently executed on pull requests will be updated alongside #260 as well as more cases (e.g. rebuild the pre-install image) can be added on following PRs.

fitzthum

LGTM

Seems like a few things here might be referring to our old jenkins things, but I guess this will be updated separately.

ldoktor · 2023-12-12T09:25:31Z

docs/DEVELOPMENT.md

+sudo chown "$USER" ~/.kube/config
+```
+
+Then use the `tests_runner.sh` script to re-run the tests like shown below (`-r` has the same meaning of in `run-local.sh`, i.e., use to set the runtimeclass):


has the same meaning as in

or even shorter "similarly to run-local the -r sets the runtimeclass"

ldoktor · 2023-12-12T10:02:07Z

docs/DEVELOPMENT.md

+sudo -E make docker-build
+sudo -E make docker-push
+./tests/e2e/tests_runner.sh -r kata-qemu
+```


Wouldn't it be better to promote the tests/e2e/operator.sh -h for available targets, rather than individual commands? On the other hands developers probably prefer the individual commands but both can be useful.

Created a new "Continuous Integration (CI)" section in the developer's guide to explain the e2e CI jobs types, how/who can execute them as well as how to run some of them locally. Fixes confidential-containers#113 Signed-off-by: Wainer dos Santos Moschetta <[email protected]>

wainersm · 2023-12-12T14:08:27Z

Updates:

removed the old jenkins jobs from the table of run jobs
promoting operator.sh as @ldoktor suggested

ldoktor

lgtm

portersrc · 2023-12-12T19:34:03Z

LGTM, thanks for adding this

stevenhorsman

LGTM. Thanks!

wainersm added documentation Improvements or additions to documentation technical debt Issue releated with a technical debt in the project testing labels Jul 7, 2023

wainersm force-pushed the document_ci branch from cb5fa1f to 57a24fd Compare July 7, 2023 19:22

katexochen reviewed Jul 8, 2023

View reviewed changes

wainersm force-pushed the document_ci branch from 57a24fd to 1b55caa Compare July 10, 2023 13:38

katexochen approved these changes Jul 10, 2023

View reviewed changes

wainersm force-pushed the document_ci branch from 1b55caa to 00c39f0 Compare August 29, 2023 13:22

fitzthum reviewed Aug 29, 2023

View reviewed changes

docs/DEVELOPMENT.md Outdated Show resolved Hide resolved

docs/DEVELOPMENT.md Outdated Show resolved Hide resolved

ldoktor reviewed Nov 28, 2023

View reviewed changes

wainersm force-pushed the document_ci branch from 00c39f0 to 6414813 Compare December 11, 2023 18:37

fitzthum approved these changes Dec 11, 2023

View reviewed changes

ldoktor reviewed Dec 12, 2023

View reviewed changes

wainersm force-pushed the document_ci branch from 6414813 to cde64ab Compare December 12, 2023 14:02

ldoktor approved these changes Dec 12, 2023

View reviewed changes

portersrc approved these changes Dec 12, 2023

View reviewed changes

stevenhorsman approved these changes Dec 13, 2023

View reviewed changes

wainersm merged commit f1a4ec5 into confidential-containers:main Dec 13, 2023
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs/DEVELOPMENT: document the CI #224

docs/DEVELOPMENT: document the CI #224

wainersm commented Jul 7, 2023

katexochen Jul 8, 2023

katexochen Jul 8, 2023

katexochen Jul 8, 2023

katexochen Jul 8, 2023

ldoktor Dec 12, 2023

wainersm commented Jul 10, 2023

katexochen left a comment

fitzthum left a comment

fidencio commented Aug 29, 2023

ldoktor Nov 28, 2023 •

edited

Loading

ldoktor Nov 28, 2023

ldoktor Nov 28, 2023

wainersm Dec 11, 2023

wainersm commented Dec 11, 2023

fitzthum left a comment

ldoktor Dec 12, 2023

ldoktor Dec 12, 2023 •

edited

Loading

ldoktor Dec 12, 2023

wainersm Dec 12, 2023

wainersm commented Dec 12, 2023

ldoktor left a comment

portersrc commented Dec 12, 2023

stevenhorsman left a comment

		```

		Notice that the `-r` parameter passed to `run-local.sh` above specifies the runtimeClass to be tested. You can switch to, for example, `kata-clh` to test Cloud Hypervisor. See the script's help (`run-local.sh -h`) for further details.


		We recommend that you run the e2e Non-TEE tests on your local machine before opening a PR to check your changes will not break the CI so to avoid wasting resources. You can also use the approach describe below to debug and fix failures, or test changes on the scripts themselves.

		The entry point script is [tests/e2e/run-local.sh/run-local.sh](../tests/e2e/run-local.sh). It is going to install softwares and change the system's configuration, so we recommend that you run the e2e tests on VMs.

	./run-local.sh -r "kata-qemu"
	./run-local.sh -u -r "kata-qemu"


		## Continuous Integration (CI)

		In order to be merged your opened pull request (PR) should pass the static analysis checks and end-to-end (e2e) tests.

docs/DEVELOPMENT: document the CI #224

docs/DEVELOPMENT: document the CI #224

Conversation

wainersm commented Jul 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wainersm commented Jul 10, 2023

katexochen left a comment

Choose a reason for hiding this comment

fitzthum left a comment

Choose a reason for hiding this comment

fidencio commented Aug 29, 2023

ldoktor Nov 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wainersm commented Dec 11, 2023

fitzthum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ldoktor Dec 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wainersm commented Dec 12, 2023

ldoktor left a comment

Choose a reason for hiding this comment

portersrc commented Dec 12, 2023

stevenhorsman left a comment

Choose a reason for hiding this comment

ldoktor Nov 28, 2023 •

edited

Loading

ldoktor Dec 12, 2023 •

edited

Loading