Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

disrupt_mgmt_backup nemesis failed during creating backup #4113

Open
timtimb0t opened this issue Nov 14, 2024 · 1 comment
Open

disrupt_mgmt_backup nemesis failed during creating backup #4113

timtimb0t opened this issue Nov 14, 2024 · 1 comment
Assignees
@Michal-Leszczynski
Labels
question Further information is requested

Comments

@timtimb0t
Copy link

Packages

Scylla version: 6.3.0~dev-20241108.aebb5329068e with build-id f25ba153fbf85f1e556539e48f980dd93e3ab285

Kernel Version: 6.8.0-1018-aws

Issue description

New issue

nemesis failed during running the command:
"sudo sctool backup -c 6e230973-a439-466f-9132-0e372132adce --location s3:manager-backup-tests-us-east-1 "

with error:

< t:2024-11-10 02:55:33,695 f:base.py         l:231  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.4.10.64>: Error: create backup target: create cluster session: no connections were made when creating the session
< t:2024-11-10 02:55:33,696 f:base.py         l:231  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.4.10.64>: Trace ID: OMG1PCX8RCmPV44b76Ys8w (grep in scylla-manager logs)
< t:2024-11-10 02:55:33,696 f:base.py         l:231  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.4.10.64>: 
< t:2024-11-10 02:55:33,696 f:base.py         l:147  c:RemoteLibSSH2CmdRunner p:ERROR > <10.4.10.64>: Error executing command: "sudo sctool backup -c 6e230973-a439-466f-9132-0e372132adce --location s3:manager-backup-tests-us-east-1 "; Exit status: 1
< t:2024-11-10 02:55:33,696 f:base.py         l:152  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.4.10.64>: STDERR: Error: create backup target: create cluster session: no connections were made when creating the session

Describe your issue in detail and steps it took to produce it.

Impact

Not sure, scylla is alive no more fails

How frequently does it reproduce?

Describe the frequency with how this issue can be reproduced.

Installation details

Cluster size: 6 nodes (i4i.4xlarge)

Scylla Nodes used in this run:

  • longevity-tls-50gb-3d-master-db-node-280a872a-9 (52.211.75.139 | 10.4.22.144) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-8 (34.241.137.84 | 10.4.22.242) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-7 (54.228.184.219 | 10.4.22.173) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-6 (54.229.77.250 | 10.4.22.37) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-5 (34.255.201.196 | 10.4.20.167) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-4 (52.210.248.129 | 10.4.23.185) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-3 (34.251.82.170 | 10.4.23.201) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-24 (54.216.199.82 | 10.4.21.123) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-23 (108.129.55.89 | 10.4.22.43) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-22 (52.211.161.91 | 10.4.20.26) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-21 (108.128.175.4 | 10.4.22.25) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-20 (52.17.57.33 | 10.4.21.205) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-2 (52.48.45.76 | 10.4.20.129) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-19 (34.253.75.51 | 10.4.23.33) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-18 (52.209.160.34 | 10.4.22.224) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-17 (54.154.93.28 | 10.4.23.144) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-16 (52.51.224.155 | 10.4.21.233) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-15 (54.228.250.240 | 10.4.22.29) (shards: -1)
  • longevity-tls-50gb-3d-master-db-node-280a872a-14 (18.200.53.77 | 10.4.22.40) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-13 (34.251.73.68 | 10.4.21.12) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-12 (52.16.77.104 | 10.4.22.222) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-11 (54.220.214.122 | 10.4.20.158) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-10 (18.203.63.179 | 10.4.22.147) (shards: 14)
  • longevity-tls-50gb-3d-master-db-node-280a872a-1 (52.30.131.206 | 10.4.22.226) (shards: 14)

OS / Image: ami-07f847bea92dccb9a (aws: undefined_region)

Test: longevity-50gb-3days-test
Test id: 280a872a-2bbb-41c4-a09d-cbfe9f7a866b
Test name: scylla-master/tier1/longevity-50gb-3days-test
Test method: longevity_test.LongevityTest.test_custom_time
Test config file(s):

Logs and commands
  • Restore Monitor Stack command: $ hydra investigate show-monitor 280a872a-2bbb-41c4-a09d-cbfe9f7a866b
  • Restore monitor on AWS instance using Jenkins job
  • Show all stored logs command: $ hydra investigate show-logs 280a872a-2bbb-41c4-a09d-cbfe9f7a866b

Logs:

Jenkins job URL
Argus
@timtimb0t timtimb0t added the question Further information is requested label Nov 14, 2024
@timtimb0t
Copy link
Author

Not sure about the manager behavior and its root cause

@Michal-Leszczynski Michal-Leszczynski self-assigned this Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Michal-Leszczynski Michal-Leszczynski

Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Michal-Leszczynski @timtimb0t