Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pytest-based integration test framework #25

Conversation

bernhardkaindl
Copy link
Collaborator

@bernhardkaindl bernhardkaindl commented Nov 22, 2023

Add a pytest-based integration test framework, addressing the weaknesses of the initial prototype tests based on shell scripts:

  • The initial shell-based tests were quick to create, but only meant as an early prototype:
    • They need root capabilities through GitHub CI or act (using a local docker or podman as backend container runtime) to run. Such container runtime is not installed on some servers, so it was not easy to check them.
    • It was easy to overlook errors in the shell scripts, leading to skipped checks, e.g. from the current SNMP feature PR, where this happens (but test does not fail because of it, see below)

To fix these weaknesses of my early prototype tests, I implemented pytest fixtures which automatically run xen-bugtool in a test container (using Linux namespaces, so it works on any Linux system that would otherwise be capable to run docker, podman, etc), but not requiring any configuration, group or root privileges to install and use.

Add:

  • The test framework using automatically always-used pytest fixtures
  • A comprehensive README-pytest.md
  • An example test case:
    • Check the output expected of xen-bugtool -y --entries=xenserver-logs (including the /etc/systemd.tar archive) that we just merged recently.
    • More tests to be added and converted with other PRs

An example, why the pytest framework instead of shell scripts is beneficial are for example these undetected test errors in this #22 which is currently in review:
https://github.com/xenserver/status-report/actions/runs/6938095957/job/18873299352?pr=22

+ grep -q 'snmpv2cpublic\|0xsnmpv3authenticationkey\|0xsnmpv3privacykey' /code/status-report/.tmp/tests/snmp/tar/snmp_xs_conf.out
grep: /code/status-report/.tmp/tests/snmp/tar/snmp_xs_conf.out: No such file or directory
+ grep -q snmpv2cpublic /code/status-report/.tmp/tests/snmp/tar/snmpd_xs_conf.out
grep: /code/status-report/.tmp/tests/snmp/tar/snmpd_xs_conf.out: No such file or directory
+ grep -q '0xsnmpv3authenticationkey\|0xsnmpv3privacykey' /code/status-report/.tmp/tests/snmp/tar/snmpd_conf.out
grep: /code/status-report/.tmp/tests/snmp/tar/snmpd_conf.out: No such file or directory

With the test implemented in Python, possibilities for silent test errors will be less difficult to spot!

And, with it, we can replace the use of grep with content matching in Python, also using re, when needed.

See the README-pytest.md for further information.

@bernhardkaindl bernhardkaindl force-pushed the private/bernhardk/pytest-integration-testframework branch from 7f8c997 to 7de527d Compare November 22, 2023 14:08
ashwin9390
ashwin9390 previously approved these changes Nov 22, 2023
@bernhardkaindl bernhardkaindl force-pushed the private/bernhardk/pytest-integration-testframework branch from 7de527d to ec1779b Compare November 22, 2023 16:05
@bernhardkaindl bernhardkaindl force-pushed the private/bernhardk/pytest-integration-testframework branch 3 times, most recently from a16fcdb to a83a126 Compare November 22, 2023 17:22
@bernhardkaindl bernhardkaindl force-pushed the private/bernhardk/pytest-integration-testframework branch 4 times, most recently from c6c0c1c to 14e98a1 Compare November 23, 2023 01:06
ashwin9390
ashwin9390 previously approved these changes Nov 23, 2023
GeraldEV
GeraldEV previously approved these changes Nov 23, 2023
Copy link
Collaborator

@GeraldEV GeraldEV left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with two minor comments

tests/unit/test_xapidb_filter.py Outdated Show resolved Hide resolved
tests/integration/test_xenserver_config.py Outdated Show resolved Hide resolved
@bernhardkaindl bernhardkaindl dismissed stale reviews from GeraldEV and ashwin9390 via 300b2cf November 23, 2023 13:07
@bernhardkaindl bernhardkaindl force-pushed the private/bernhardk/pytest-integration-testframework branch from 14e98a1 to 300b2cf Compare November 23, 2023 13:07
@bernhardkaindl bernhardkaindl merged commit 147dac5 into xenserver:master Nov 23, 2023
2 checks passed
@bernhardkaindl bernhardkaindl deleted the private/bernhardk/pytest-integration-testframework branch January 8, 2024 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants