better logging of fam failures #16967

evgeni · 2024-11-20T15:13:43Z

Problem Statement

The output of FAM tests is mangled due to how pytest tries to print the big result object.
There are no linebreaks and it's super hard to find the failure.

Additionally the double assert is pointless, as it asserts twice the same thing: the result is successful.

Solution

Stop asserting PASSED in stdout (additionally saves the case where there is the string PASSED in the output for some reason, but the test actually failed).
Pass a custom error output to the status == 0 assertion, including stdout, stderr and the status code.

Related Issues

evgeni · 2024-11-20T15:16:34Z

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py

Satellite-QE · 2024-11-20T15:42:03Z

PRT Result

Build Number: 9416
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 108 failed, 2 passed, 336 warnings in 1228.67s (0:20:28) ===========

evgeni · 2024-11-20T16:04:44Z

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py

Satellite-QE · 2024-11-20T16:30:00Z

PRT Result

Build Number: 9421
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 108 failed, 2 passed, 328 warnings in 1120.24s (0:18:40) ===========

evgeni · 2024-11-20T17:07:44Z

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py

Satellite-QE · 2024-11-20T17:35:41Z

PRT Result

Build Number: 9423
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 108 failed, 2 passed, 337 warnings in 1113.46s (0:18:33) ===========

evgeni · 2024-11-20T17:57:46Z

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py

Satellite-QE · 2024-11-20T18:27:29Z

PRT Result

Build Number: 9424
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 108 failed, 2 passed, 339 warnings in 1260.80s (0:21:00) ===========

evgeni · 2024-11-20T19:04:42Z

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py

Satellite-QE · 2024-11-20T20:29:37Z

PRT Result

Build Number: 9425
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 16 failed, 94 passed, 341 warnings in 4650.37s (1:17:30) ===========

evgeni · 2024-11-21T09:52:32Z

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py

Satellite-QE · 2024-11-21T11:21:05Z

PRT Result

Build Number: 9431
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 16 failed, 94 passed, 333 warnings in 4732.56s (1:18:52) ===========

evgeni · 2024-11-21T12:50:45Z

tests/foreman/sys/test_fam.py

+    module_target_sat.get('/tmp/pytest.stdout', 'pytest.stdout')
+    module_target_sat.get('/tmp/pytest.stderr', 'pytest.stderr')


this can't be the right approach :(

evgeni · 2024-11-22T07:13:59Z

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py -vv

Satellite-QE · 2024-11-22T08:46:17Z

PRT Result

Build Number: 9445
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py -vv --external-logging
Test Result : =========== 16 failed, 94 passed, 356 warnings in 5042.30s (1:24:02) ===========

evgeni · 2024-11-22T08:51:24Z

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles[smart_class_parameter_override_value]

Satellite-QE · 2024-11-22T09:15:44Z

PRT Result

Build Number: 9447
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles[smart_class_parameter_override_value] --external-logging
Test Result : ================= 1 failed, 11 warnings in 1118.29s (0:18:38) ==================

rmynar

Can't be easier :)

rmynar · 2024-11-25T15:02:49Z

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles[activation_key]

rmynar · 2024-11-25T15:13:14Z

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py::test_positive_import_run_roles

Satellite-QE · 2024-11-25T15:28:07Z

PRT Result

Build Number: 9455
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles[activation_key] --external-logging
Test Result : ================== 1 passed, 11 warnings in 986.83s (0:16:26) ==================

Satellite-QE · 2024-11-25T15:31:08Z

PRT Result

Build Number: 9456
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/sys/test_fam.py::test_positive_import_run_roles --external-logging
Test Result : ================== 1 passed, 9 warnings in 599.90s (0:09:59) ===================

(cherry picked from commit 04976cc)

better logging of fam failures (#16967) (cherry picked from commit 04976cc) Co-authored-by: Evgeni Golov <[email protected]>

(cherry picked from commit 04976cc)

* disable pytest plugin autoloading when running FAM tests (#16975) pytest by default tries to load all plugins it finds on a system. pulpcore ships with own pytest plugins, but we do not install their dependencies (as we don't want to run pulpcore tests), which leads to pytest failing to load those plugins and breaking overall execution: Traceback (most recent call last): File "/usr/bin/pytest-3.11", line 33, in <module> sys.exit(load_entry_point('pytest==7.2.0', 'console_scripts', 'pytest')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/_pytest/config/__init__.py", line 190, in console_main code = main() ^^^^^^ File "/usr/lib/python3.11/site-packages/_pytest/config/__init__.py", line 148, in main config = _prepareconfig(args, plugins) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/_pytest/config/__init__.py", line 329, in _prepareconfig config = pluginmanager.hook.pytest_cmdline_parse( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 55, in _multicall gen.send(outcome) File "/usr/lib/python3.11/site-packages/_pytest/helpconfig.py", line 103, in pytest_cmdline_parse config: Config = outcome.get_result() ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result raise ex[1].with_traceback(ex[2]) File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall res = hook_impl.function(*args) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1058, in pytest_cmdline_parse self.parse(args) File "/usr/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1346, in parse self._preparse(args, addopts=addopts) File "/usr/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1229, in _preparse self.pluginmanager.load_setuptools_entrypoints("pytest11") File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints plugin = ep.load() ^^^^^^^^^ File "/usr/lib64/python3.11/importlib/metadata/__init__.py", line 202, in load module = import_module(match.group('module')) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1204, in _gcd_import File "<frozen importlib._bootstrap>", line 1176, in _find_and_load File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 690, in _load_unlocked File "/usr/lib/python3.11/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/usr/lib/python3.11/site-packages/pulp_ansible/pytest_plugin.py", line 3, in <module> import numpy as np ModuleNotFoundError: No module named 'numpy' Disable the autoloading by setting the PYTEST_DISABLE_PLUGIN_AUTOLOAD environment variable. (cherry picked from commit 6a91dd9) * simplify FAM test command (#16980) - we don't need to export vars, setting them should be sufficient - make can change directories for us, no need to cd (cherry picked from commit fefe1ea) * create a fake ntp module for FAM tests (#16997) * use sat.put() instead of sat.execute(echo) to place puppet files * allow passing in custom module code * create a fake ntp module the one in the repo is too old (cherry picked from commit 39769a5) * better logging of fam failures (#16967) (cherry picked from commit 04976cc)

* Create fake galaxy.yml to make the FAM Makefile happy (#16936) The data in the file is unused, but not being able to load it produces errors in the logs and is confusing when searching for an actual problem during testing. (cherry picked from commit c45e681) * disable pytest plugin autoloading when running FAM tests (#16975) pytest by default tries to load all plugins it finds on a system. pulpcore ships with own pytest plugins, but we do not install their dependencies (as we don't want to run pulpcore tests), which leads to pytest failing to load those plugins and breaking overall execution: Traceback (most recent call last): File "/usr/bin/pytest-3.11", line 33, in <module> sys.exit(load_entry_point('pytest==7.2.0', 'console_scripts', 'pytest')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/_pytest/config/__init__.py", line 190, in console_main code = main() ^^^^^^ File "/usr/lib/python3.11/site-packages/_pytest/config/__init__.py", line 148, in main config = _prepareconfig(args, plugins) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/_pytest/config/__init__.py", line 329, in _prepareconfig config = pluginmanager.hook.pytest_cmdline_parse( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 55, in _multicall gen.send(outcome) File "/usr/lib/python3.11/site-packages/_pytest/helpconfig.py", line 103, in pytest_cmdline_parse config: Config = outcome.get_result() ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result raise ex[1].with_traceback(ex[2]) File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall res = hook_impl.function(*args) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1058, in pytest_cmdline_parse self.parse(args) File "/usr/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1346, in parse self._preparse(args, addopts=addopts) File "/usr/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1229, in _preparse self.pluginmanager.load_setuptools_entrypoints("pytest11") File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints plugin = ep.load() ^^^^^^^^^ File "/usr/lib64/python3.11/importlib/metadata/__init__.py", line 202, in load module = import_module(match.group('module')) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1204, in _gcd_import File "<frozen importlib._bootstrap>", line 1176, in _find_and_load File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 690, in _load_unlocked File "/usr/lib/python3.11/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/usr/lib/python3.11/site-packages/pulp_ansible/pytest_plugin.py", line 3, in <module> import numpy as np ModuleNotFoundError: No module named 'numpy' Disable the autoloading by setting the PYTEST_DISABLE_PLUGIN_AUTOLOAD environment variable. (cherry picked from commit 6a91dd9) * simplify FAM test command (#16980) - we don't need to export vars, setting them should be sufficient - make can change directories for us, no need to cd (cherry picked from commit fefe1ea) * create a fake ntp module for FAM tests (#16997) * use sat.put() instead of sat.execute(echo) to place puppet files * allow passing in custom module code * create a fake ntp module the one in the repo is too old (cherry picked from commit 39769a5) * better logging of fam failures (#16967) (cherry picked from commit 04976cc)

Satellite-QE added the PRT-Failed Indicates that latest PRT run is failed for the PR label Nov 20, 2024

evgeni force-pushed the better-fam-log branch from ad2bd00 to 67efb89 Compare November 20, 2024 16:01

evgeni added CherryPick PR needs CherryPick to previous branches 6.14.z Introduced in or relating directly to Satellite 6.14 6.15.z Introduced in or relating directly to Satellite 6.15 6.16.z Introduced in or relating directly to Satellite 6.16 labels Nov 20, 2024

evgeni force-pushed the better-fam-log branch 2 times, most recently from 4221c48 to 7c562af Compare November 21, 2024 09:52

evgeni removed 6.14.z Introduced in or relating directly to Satellite 6.14 6.15.z Introduced in or relating directly to Satellite 6.15 labels Nov 21, 2024

evgeni commented Nov 21, 2024

View reviewed changes

evgeni marked this pull request as ready for review November 21, 2024 13:20

evgeni requested a review from a team as a code owner November 21, 2024 13:20

github-actions bot assigned evgeni Nov 21, 2024

evgeni force-pushed the better-fam-log branch from 7c562af to 8059901 Compare November 22, 2024 07:12

better logging of fam failures

9d792a5

evgeni force-pushed the better-fam-log branch from 8059901 to 9d792a5 Compare November 25, 2024 10:30

rmynar approved these changes Nov 25, 2024

View reviewed changes

Satellite-QE added PRT-Passed Indicates that latest PRT run is passed for the PR and removed PRT-Failed Indicates that latest PRT run is failed for the PR labels Nov 25, 2024

evgeni merged commit 04976cc into SatelliteQE:master Nov 26, 2024
11 checks passed

evgeni deleted the better-fam-log branch November 26, 2024 12:43

github-actions bot pushed a commit that referenced this pull request Nov 26, 2024

better logging of fam failures (#16967)

fb7e207

(cherry picked from commit 04976cc)

Satellite-QE mentioned this pull request Nov 26, 2024

[6.16.z] better logging of fam failures #17001

Merged

pondrejk pushed a commit that referenced this pull request Nov 29, 2024

[6.16.z] better logging of fam failures (#17001)

8d533ec

better logging of fam failures (#16967) (cherry picked from commit 04976cc) Co-authored-by: Evgeni Golov <[email protected]>

evgeni added a commit to evgeni/robottelo that referenced this pull request Dec 13, 2024

better logging of fam failures (SatelliteQE#16967)

6a04ccf

(cherry picked from commit 04976cc)

evgeni added a commit to evgeni/robottelo that referenced this pull request Dec 13, 2024

better logging of fam failures (SatelliteQE#16967)

2d466d7

(cherry picked from commit 04976cc)

evgeni added a commit to evgeni/robottelo that referenced this pull request Dec 13, 2024

better logging of fam failures (SatelliteQE#16967)

10cf33d

(cherry picked from commit 04976cc)

evgeni added a commit to evgeni/robottelo that referenced this pull request Dec 13, 2024

better logging of fam failures (SatelliteQE#16967)

446e1c8

(cherry picked from commit 04976cc)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

better logging of fam failures #16967

better logging of fam failures #16967

evgeni commented Nov 20, 2024 •

edited

Loading

evgeni commented Nov 20, 2024

Satellite-QE commented Nov 20, 2024

evgeni commented Nov 20, 2024

Satellite-QE commented Nov 20, 2024

evgeni commented Nov 20, 2024

Satellite-QE commented Nov 20, 2024

evgeni commented Nov 20, 2024

Satellite-QE commented Nov 20, 2024

evgeni commented Nov 20, 2024

Satellite-QE commented Nov 20, 2024

evgeni commented Nov 21, 2024

Satellite-QE commented Nov 21, 2024

evgeni Nov 21, 2024

evgeni commented Nov 22, 2024

Satellite-QE commented Nov 22, 2024

evgeni commented Nov 22, 2024

Satellite-QE commented Nov 22, 2024

rmynar left a comment

rmynar commented Nov 25, 2024

rmynar commented Nov 25, 2024

Satellite-QE commented Nov 25, 2024

Satellite-QE commented Nov 25, 2024

		module_target_sat.get('/tmp/pytest.stdout', 'pytest.stdout')
		module_target_sat.get('/tmp/pytest.stderr', 'pytest.stderr')

better logging of fam failures #16967

better logging of fam failures #16967

Conversation

evgeni commented Nov 20, 2024 • edited Loading

Problem Statement

Solution

Related Issues

evgeni commented Nov 20, 2024

Satellite-QE commented Nov 20, 2024

evgeni commented Nov 20, 2024

Satellite-QE commented Nov 20, 2024

evgeni commented Nov 20, 2024

Satellite-QE commented Nov 20, 2024

evgeni commented Nov 20, 2024

Satellite-QE commented Nov 20, 2024

evgeni commented Nov 20, 2024

Satellite-QE commented Nov 20, 2024

evgeni commented Nov 21, 2024

Satellite-QE commented Nov 21, 2024

evgeni Nov 21, 2024

Choose a reason for hiding this comment

evgeni commented Nov 22, 2024

Satellite-QE commented Nov 22, 2024

evgeni commented Nov 22, 2024

Satellite-QE commented Nov 22, 2024

rmynar left a comment

Choose a reason for hiding this comment

rmynar commented Nov 25, 2024

rmynar commented Nov 25, 2024

Satellite-QE commented Nov 25, 2024

Satellite-QE commented Nov 25, 2024

evgeni commented Nov 20, 2024 •

edited

Loading