fix(operator.py): Increased timeout for get_mgr_backup_task method

[grzywin]

Some of the longevity tests for the Operator (e.g., longevity-scylla-operator-3h-eks-backup-test) are failing because they exceed the 300 second timeout set in the get_mgr_backup_task method. After investigation, I noticed that this operation can take up to approximately 8 minutes, so I increased the timeout to 10 minutes. Link to successful run after increasing timeout here

[soyacz]

Isn't it something to raise in the operator side - why creating a backup task takes 8 minutes?

[grzywin]

This is something they are already aware of as they were running the same tests for 1.13 release and had this timeout.
@zimnx put a comment about it here ("Discussion" tab) and marked tests as passed.

Isn't that disrupt_mgmt_backup nemesis is causing the backup taking more time?

[mikliapko]

LGTM

[soyacz]

This is something they are already aware of as they were running the same tests for 1.13 release and had this timeout. @zimnx put a comment about it here ("Discussion" tab) and marked tests as passed.

Isn't that disrupt_mgmt_backup nemesis is causing the backup taking more time?

@zimnx might not be aware of the timeout value.
I believe status should show up earlier than 5 minutes (verified every 2 secs), worth to investigate what is the bottleneck here.

[zimnx]

it's a known issue that creating tasks takes long time. We have multiple issues with improvements (scylladb/scylla-operator#1939). Recent Manager 3.3.1 brought important feature which will help us resolve subset of them.
I'm fine with bumping the timeout until we fix stuff on our end.

[vponomaryov]

LGTM

[soyacz]

ugh, I didn't see missing backport labels.
@grzywinski where we need that to be fixed?

[vponomaryov]

ugh, I didn't see missing backport labels. @grzywinski where we need that to be fixed?

We don't have dedicated operator branches. It gets run with SCT master branch only.