tests: adding group and importance markers

[danlavu]

No description provided.

[jakub-vavra-cz]

Please do not add these to high/critical:
FAILED ../sssd/src/tests/system/tests/test_autofs.py::test_autofs__propagate_offline__single_domain (ad)
FAILED ../sssd/src/tests/system/tests/test_sudo.py::test_sudo__sudonotbefore_shorttime (ldap)
I think that there was a bug in sudo rules that affects the test (but bug reference is missing from the failing test for some reason).

As for test_autofs__propagate_offline__single_domain, it just blocks ldap port but does nothing about other AD ports and it does not look correct to me as it is for AD provider.

I would generally not consider tests for backends offline as critical apart from some specific ones. (like caching works when the backend goes offline).

The critical set should contain only the most important tests (smoke tests) to represent the the important features.

[danlavu]

@jakub-vavra-cz Ack, removed from critical/high and moved to medium/low.

[jakub-vavra-cz]

LGTM.

[pbrezina]

@jakub-vavra-cz we need to create some guidance how to assign the importance level and how does it map to downstream test execution. Can you update https://tests.sssd.io/en/latest/marks.html#pytest-mark-tier?

From my point of view, if there is a failure, we can not release new version. So every test counts. In this sense, every test is critical. So having guidance and use case documented would help a lot.

[pbrezina]

@sidecontrol I agree with the markers, but not with the documentation. It needs to be clearly documented how we want to use them - like schema and cache for example.

pytest.ini is only suitable for short description, https://tests.sssd.io/en/latest/marks.html#additional-markers is a good place for a thorough guidance.

If you agree, I think we should add comment to pytest.ini # see explanation of the markers at xyz and even remove the short description as it just repeates the marker name but does not provide more information. Then I think we can merge this one a you will follow with pull requests to sssd-test-framework/docs

[danlavu]

@pbrezina Yup, I'll create another pull request with the doc explaining the markers.

[pbrezina]

I marked this as blocked, because I think that we need the documentation first. Because until we have documentation, there will still be new tests that may miss this and you will need to update them here.

[danlavu]

@pbrezina

SSSD/sssd-test-framework#43

doc update.

[patriki01]

Thanks for your contribution. I approve.

[jakub-vavra-cz]

Pushed PR: #6902

• master
  • f05d4ec - tests: adding group and importance markers
• sssd-2-9
  • 674ee26 - tests: adding group and importance markers