Assertion Error, time delta/format in 'wait_for_sync()', of capsule content 'last_sync_time' #14066

damoore044 · 2024-02-13T17:46:27Z

Problem Statement

4 Failures in recent runs of API::CapsuleContent, for AssertionError:

tests/foreman/api/test_capsulecontent.py:190: in test_positive_checksum_sync
    module_capsule_configured.wait_for_sync()
robottelo/host_helpers/capsule_mixins.py:75: in wait_for_sync
    or datetime.strptime(sync_status['last_sync_time'], '%Y-%m-%d %H:%M:%S UTC')
E   AssertionError

Locally I can see the last_sync_time format from logs as:

nailgun.client - DEBUG - Received HTTP 200 response: {"last_sync_time":"2024-02-13 15:05:10 UTC", ...

Solution

Split the assertions checking last_sync_time (check if None /Falsey first), then checking length of any in-progress sync tasks, avoiding possibly ambiguous assertion- where last_sync_time has not updated with a recent task, and is still an old sync time/None.
The prior Assertions with AND, OR, etc fail without showing the compared values. Provided error messages with assertion failure details.
Apply a one second margin of safety to start_time, due to rounding. Often the assertion fails for one second delta, which is start time recorded and sync done at pretty much the same time, but the seconds from sync_status were rounded to one second before start_time. This is inconsistent, depending on the milliseconds when last_sync_time is recorded.
From dateutil.parser, use parse(date-time string) method on the start_time, and any found last_sync_time. So when checking the last sync time is newer, parse will account for different formats of the same time. Similar solution to PR Fix Date assertion failure in CI errata:e2e #13121

Related Issue >> From 'sat-6.15-rhel8-Capsule-Content'

4 Failed api scenarios due to the AssertionError in this method. Inconsistently, others will fail locally for the same error.
6.14.z and 6.15.0 (most recent Build 13, Mar 7 2024, 12:43 AM)

PRT Case

trigger: test-robottelo
pytest: tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement

damoore044 · 2024-02-13T17:46:41Z

trigger: test-robottelo
pytest: tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement

Satellite-QE · 2024-02-13T19:34:06Z

PRT Result

Build Number: 5706
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement
Test Result : =========== 6 failed, 15 passed, 1357 warnings in 6107.27s (1:41:47) ===========

damoore044 · 2024-02-13T21:23:24Z

trigger: test-robottelo
pytest: tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement

Satellite-QE · 2024-02-13T23:11:17Z

PRT Result

Build Number: 5708
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement
Test Result : =========== 2 failed, 19 passed, 1404 warnings in 5961.97s (1:39:21) ===========

damoore044 · 2024-02-14T02:21:37Z

trigger: test-robottelo
pytest: tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement

Satellite-QE · 2024-02-14T04:09:19Z

PRT Result

Build Number: 5709
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement
Test Result : =========== 2 failed, 19 passed, 1423 warnings in 6162.18s (1:42:42) ===========

lhellebr

Left two comments, those things seem weird to me.
Conceptually this LGTM.
PRT needs to pass.

robottelo/host_helpers/capsule_mixins.py

damoore044 · 2024-02-14T14:20:01Z

trigger: test-robottelo
pytest: tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement

Satellite-QE · 2024-02-14T16:06:41Z

PRT Result

Build Number: 5719
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement
Test Result : =========== 2 failed, 19 passed, 1390 warnings in 5992.92s (1:39:52) ===========

damoore044 · 2024-02-14T21:48:19Z

trigger: test-robottelo
pytest: tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement

Satellite-QE · 2024-02-14T23:50:07Z

PRT Result

Build Number: 5727
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement
Test Result : ================ 21 passed, 1521 warnings in 6806.82s (1:53:26) ================

damoore044 · 2024-03-11T13:25:27Z

trigger: test-robottelo
pytest: tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement

Satellite-QE · 2024-03-11T15:09:13Z

PRT Result

Build Number: 6018
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement --external-logging
Test Result : ========== 20 passed, 1 skipped, 1398 warnings in 5808.92s (1:36:48) ===========

damoore044 · 2024-03-11T21:27:43Z

trigger: test-robottelo
pytest: tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement

damoore044 · 2024-03-12T13:16:40Z

trigger: test-robottelo
pytest: tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement

damoore044 · 2024-03-13T13:43:15Z

trigger: test-robottelo
pytest: tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement

change to sca-only orgs, entitlement orgs are failing assert capsule sync task(s) when invoked, and sync status pre-commit fix, refactor wait_for_sync() into concise steps

Satellite-QE · 2024-03-13T15:38:43Z

PRT Result

Build Number: 6051
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement --external-logging
Test Result : =========== 1 failed, 20 passed, 1428 warnings in 6410.09s (1:46:50) ===========

damoore044 · 2024-03-13T16:00:05Z

trigger: test-robottelo
pytest: tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement

Satellite-QE · 2024-03-13T17:43:15Z

PRT Result

Build Number: 6056
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/api/test_capsulecontent.py::TestCapsuleContentManagement --external-logging
Test Result : ================ 21 passed, 1431 warnings in 5734.36s (1:35:34) ================

jyejare

ACK , looks great but seems like we are verifying some things twice or more which is done by synchronous operation.

If you think its not, feel free to merge the PR.

jyejare · 2024-03-14T08:51:54Z

robottelo/host_helpers/capsule_mixins.py

        )
+        # No failed or active tasks remaining
+        assert len(updated_status['last_failed_sync_tasks']) == 0
+        assert len(updated_status['active_sync_tasks']) == 0


I think we dont need to verify this as maybe
self.nailgun_capsule.content_get_sync(timeout=timeout, synchronous=True) being synchronus activity verifies that ?

And anything similar assertions we do above are not needed ?

It appears content_get_sync does not run synchronously with the active sync task(s).
It seems to return any sync tasks in progress for the capsule, not matter if run synchronously w/ timeout or not.

Here, it is possible for a task to have been active, then finished just before querying content_get_sync. If that task failed and did so before querying, we would want to check that there are no failed tasks, otherwise we could have a recently failed sync that was not checked.

@damoore044 I think if synchronus is not behaving as it suppose to be we should raise a bug ! immediately !!

…ontent 'last_sync_time' (#14066) Assertion Error for datetime format of 'last sync time' change to sca-only orgs, entitlement orgs are failing assert capsule sync task(s) when invoked, and sync status pre-commit fix, refactor wait_for_sync() into concise steps (cherry picked from commit c10982f)

… in 'wait_for_sync()' (#14412) Assertion Error, time delta/format in 'wait_for_sync()', of capsule content 'last_sync_time' (#14066) Assertion Error for datetime format of 'last sync time' change to sca-only orgs, entitlement orgs are failing assert capsule sync task(s) when invoked, and sync status pre-commit fix, refactor wait_for_sync() into concise steps (cherry picked from commit c10982f) Co-authored-by: David Moore <[email protected]>

…ontent 'last_sync_time' (SatelliteQE#14066) Assertion Error for datetime format of 'last sync time' change to sca-only orgs, entitlement orgs are failing assert capsule sync task(s) when invoked, and sync status pre-commit fix, refactor wait_for_sync() into concise steps

damoore044 requested a review from a team as a code owner February 13, 2024 17:46

github-actions bot assigned damoore044 Feb 13, 2024

damoore044 added CherryPick PR needs CherryPick to previous branches Stream Introduced in or relating directly to Satellite Stream/Master 6.14.z Introduced in or relating directly to Satellite 6.14 6.15.z Introduced in or relating directly to Satellite 6.15 labels Feb 13, 2024

damoore044 requested a review from a team February 13, 2024 17:47

Satellite-QE added the PRT-Failed Indicates that latest PRT run is failed for the PR label Feb 13, 2024

damoore044 force-pushed the wait_for_sync_fix branch from 783323d to 7dc583c Compare February 13, 2024 21:23

damoore044 force-pushed the wait_for_sync_fix branch from 7dc583c to 9949945 Compare February 13, 2024 22:26

damoore044 changed the title ~~Assertion Error, datetime format in 'wait_for_sync()', of capsule content 'last_sync_time'~~ Assertion Error, time delta/format in 'wait_for_sync()', of capsule content 'last_sync_time' Feb 13, 2024

damoore044 force-pushed the wait_for_sync_fix branch 3 times, most recently from 940efbe to 1b83c35 Compare February 14, 2024 02:18

lhellebr reviewed Feb 14, 2024

View reviewed changes

robottelo/host_helpers/capsule_mixins.py Outdated Show resolved Hide resolved

robottelo/host_helpers/capsule_mixins.py Show resolved Hide resolved

damoore044 force-pushed the wait_for_sync_fix branch from 1b83c35 to 76ec135 Compare February 14, 2024 13:32

damoore044 force-pushed the wait_for_sync_fix branch from 76ec135 to 9e3ed99 Compare February 14, 2024 19:28

Satellite-QE added PRT-Passed Indicates that latest PRT run is passed for the PR and removed PRT-Failed Indicates that latest PRT run is failed for the PR labels Feb 14, 2024

damoore044 force-pushed the wait_for_sync_fix branch from 9e3ed99 to 1b863a9 Compare February 15, 2024 14:03

damoore044 removed the PRT-Passed Indicates that latest PRT run is passed for the PR label Mar 11, 2024

damoore044 force-pushed the wait_for_sync_fix branch from f8db636 to f5b8cc0 Compare March 11, 2024 14:07

Satellite-QE added the PRT-Passed Indicates that latest PRT run is passed for the PR label Mar 11, 2024

damoore044 force-pushed the wait_for_sync_fix branch 2 times, most recently from c50991a to 17a1cd8 Compare March 11, 2024 21:27

damoore044 removed the PRT-Passed Indicates that latest PRT run is passed for the PR label Mar 13, 2024

Assertion Error for datetime format of 'last sync time'

e717963

change to sca-only orgs, entitlement orgs are failing assert capsule sync task(s) when invoked, and sync status pre-commit fix, refactor wait_for_sync() into concise steps

damoore044 force-pushed the wait_for_sync_fix branch from 17a1cd8 to e717963 Compare March 13, 2024 14:46

Satellite-QE added the PRT-Failed Indicates that latest PRT run is failed for the PR label Mar 13, 2024

Satellite-QE added PRT-Passed Indicates that latest PRT run is passed for the PR and removed PRT-Failed Indicates that latest PRT run is failed for the PR labels Mar 13, 2024

ColeHiggins2 approved these changes Mar 13, 2024

View reviewed changes

jyejare approved these changes Mar 14, 2024

View reviewed changes

vsedmik merged commit c10982f into SatelliteQE:master Mar 14, 2024
11 checks passed

Satellite-QE mentioned this pull request Mar 14, 2024

[Failed-AutoCherryPick] - Assertion Error, time delta/format in 'wait_for_sync()', of capsule content 'last_sync_time' #14411

Closed

Satellite-QE mentioned this pull request Mar 14, 2024

[6.15.z] Eval CapsuleContent::API Assertion Errors, time delta/format in 'wait_for_sync()' #14412

Merged

damoore044 mentioned this pull request Mar 14, 2024

[6.14.z] Eval CapsuleContent::API Assertion Errors, time delta/format in 'wait_for_sync()' #14413

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Assertion Error, time delta/format in 'wait_for_sync()', of capsule content 'last_sync_time' #14066

Assertion Error, time delta/format in 'wait_for_sync()', of capsule content 'last_sync_time' #14066

damoore044 commented Feb 13, 2024 •

edited

Loading

damoore044 commented Feb 13, 2024

Satellite-QE commented Feb 13, 2024

damoore044 commented Feb 13, 2024

Satellite-QE commented Feb 13, 2024

damoore044 commented Feb 14, 2024

Satellite-QE commented Feb 14, 2024

lhellebr left a comment

damoore044 commented Feb 14, 2024

Satellite-QE commented Feb 14, 2024

damoore044 commented Feb 14, 2024

Satellite-QE commented Feb 14, 2024

damoore044 commented Mar 11, 2024

Satellite-QE commented Mar 11, 2024

damoore044 commented Mar 11, 2024

damoore044 commented Mar 12, 2024

damoore044 commented Mar 13, 2024

Satellite-QE commented Mar 13, 2024

damoore044 commented Mar 13, 2024

Satellite-QE commented Mar 13, 2024

jyejare left a comment

jyejare Mar 14, 2024

jyejare Mar 14, 2024

damoore044 Mar 14, 2024 •

edited

Loading

jyejare Mar 14, 2024

Assertion Error, time delta/format in 'wait_for_sync()', of capsule content 'last_sync_time' #14066

Assertion Error, time delta/format in 'wait_for_sync()', of capsule content 'last_sync_time' #14066

Conversation

damoore044 commented Feb 13, 2024 • edited Loading

Problem Statement

Solution

Related Issue >> From 'sat-6.15-rhel8-Capsule-Content'

PRT Case

damoore044 commented Feb 13, 2024

Satellite-QE commented Feb 13, 2024

damoore044 commented Feb 13, 2024

Satellite-QE commented Feb 13, 2024

damoore044 commented Feb 14, 2024

Satellite-QE commented Feb 14, 2024

lhellebr left a comment

Choose a reason for hiding this comment

damoore044 commented Feb 14, 2024

Satellite-QE commented Feb 14, 2024

damoore044 commented Feb 14, 2024

Satellite-QE commented Feb 14, 2024

damoore044 commented Mar 11, 2024

Satellite-QE commented Mar 11, 2024

damoore044 commented Mar 11, 2024

damoore044 commented Mar 12, 2024

damoore044 commented Mar 13, 2024

Satellite-QE commented Mar 13, 2024

damoore044 commented Mar 13, 2024

Satellite-QE commented Mar 13, 2024

jyejare left a comment

Choose a reason for hiding this comment

jyejare Mar 14, 2024

Choose a reason for hiding this comment

jyejare Mar 14, 2024

Choose a reason for hiding this comment

damoore044 Mar 14, 2024 • edited Loading

Choose a reason for hiding this comment

jyejare Mar 14, 2024

Choose a reason for hiding this comment

damoore044 commented Feb 13, 2024 •

edited

Loading

damoore044 Mar 14, 2024 •

edited

Loading