Cassandra S3 Backup returns malformed authorization header on uploading to S3

[ignstv]

When taking backup with dcos cassandra backup start --backup_name=test-snapshot --s3_access_key=<access-key> --s3_secret_key=<secret-key> --external_location=s3://<uri>, the snapshot node on dcos returns The authorization header is malformed; a non-empty Access Key (AKID) must be provided in the credential and S3 upload just fails. I have passed the access key and secret key and I have confirmed that I can put objects to S3 with those keys with AWS CLI locally. Is this a bug?

Here is the log from the snapshot node:

INFO  [2017-02-03 19:39:43,411] com.mesosphere.dcos.cassandra.executor.backup.S3StorageDriver: Valid backup directories. Keyspace: /var/lib/mesos/slave/slaves/58896702-d56b-4a89-906f-b2b9fd800273-S0/frameworks/58896702-d56b-4a89-906f-b2b9fd800273-0002/executors/node-0_executor__e8f659b0-2b67-4ff1-9e83-74594b584474/runs/1a3c41af-91a8-4d78-9252-e348e24aa8bc/volume/data/system_traces | ColumnFamily: /var/lib/mesos/slave/slaves/58896702-d56b-4a89-906f-b2b9fd800273-S0/frameworks/58896702-d56b-4a89-906f-b2b9fd800273-0002/executors/node-0_executor__e8f659b0-2b67-4ff1-9e83-74594b584474/runs/1a3c41af-91a8-4d78-9252-e348e24aa8bc/volume/data/system_traces/events-8826e8e9e16a372887533bc1fc713c25 | Snapshot: /var/lib/mesos/slave/slaves/58896702-d56b-4a89-906f-b2b9fd800273-S0/frameworks/58896702-d56b-4a89-906f-b2b9fd800273-0002/executors/node-0_executor__e8f659b0-2b67-4ff1-9e83-74594b584474/runs/1a3c41af-91a8-4d78-9252-e348e24aa8bc/volume/data/system_traces/events-8826e8e9e16a372887533bc1fc713c25/snapshots | BackupName: test-snapshot
INFO  [2017-02-03 19:39:43,411] com.mesosphere.dcos.cassandra.executor.backup.S3StorageDriver: Valid snapshot directory: true
INFO  [2017-02-03 19:39:43,411] com.mesosphere.dcos.cassandra.executor.backup.S3StorageDriver: Going to upload directory: /var/lib/mesos/slave/slaves/58896702-d56b-4a89-906f-b2b9fd800273-S0/frameworks/58896702-d56b-4a89-906f-b2b9fd800273-0002/executors/node-0_executor__e8f659b0-2b67-4ff1-9e83-74594b584474/runs/1a3c41af-91a8-4d78-9252-e348e24aa8bc/volume/data/system_traces/events-8826e8e9e16a372887533bc1fc713c25/snapshots/test-snapshot
INFO  [2017-02-03 19:39:43,411] com.mesosphere.dcos.cassandra.executor.backup.S3StorageDriver: URI: <s3-uri>
ERROR [2017-02-03 19:39:45,135] com.mesosphere.dcos.cassandra.executor.backup.S3StorageDriver: Error occurred on uploading directory test-snapshot : {}
! com.amazonaws.services.s3.model.AmazonS3Exception: The authorization header is malformed; a non-empty Access Key (AKID) must be provided in the credential. (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: <request-id>)
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1530) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1168) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:949) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:662) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:636) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:619) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:587) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:574) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:446) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4035) ~[aws-java-sdk-s3-1.11.49.jar:na]
! at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1585) ~[aws-java-sdk-s3-1.11.49.jar:na]
! at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:131) ~[aws-java-sdk-s3-1.11.49.jar:na]
! at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:123) ~[aws-java-sdk-s3-1.11.49.jar:na]
! at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:139) ~[aws-java-sdk-s3-1.11.49.jar:na]
! at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:47) ~[aws-java-sdk-s3-1.11.49.jar:na]
! at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_74]
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_74]
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_74]
! at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
INFO  [2017-02-03 19:39:45,135] com.mesosphere.dcos.cassandra.executor.backup.S3StorageDriver: Failed uploading snapshots for backup: test-snapshot, error: {}
! com.amazonaws.services.s3.model.AmazonS3Exception: The authorization header is malformed; a non-empty Access Key (AKID) must be provided in the credential. (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: <request-id>)
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1530) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1168) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:949) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:662) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:636) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:619) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:587) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:574) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:446) ~[aws-java-sdk-core-1.11.49.jar:na]
! at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4035) ~[aws-java-sdk-s3-1.11.49.jar:na]
! at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1585) ~[aws-java-sdk-s3-1.11.49.jar:na]
! at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:131) ~[aws-java-sdk-s3-1.11.49.jar:na]
! at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:123) ~[aws-java-sdk-s3-1.11.49.jar:na]
! at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:139) ~[aws-java-sdk-s3-1.11.49.jar:na]
! at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:47) ~[aws-java-sdk-s3-1.11.49.jar:na]
! ... 4 common frames omitted
! Causing: java.lang.Exception: com.amazonaws.services.s3.model.AmazonS3Exception: The authorization header is malformed; a non-empty Access Key (AKID) must be provided in the credential. (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: <request-id> S3 Extended Request ID: <ex-request-id> at com.mesosphere.dcos.cassandra.executor.backup.S3StorageDriver.uploadDirectory(S3StorageDriver.java:213) ~[cassandra-executor.jar:na]
! at com.mesosphere.dcos.cassandra.executor.backup.S3StorageDriver.upload(S3StorageDriver.java:178) ~[cassandra-executor.jar:na]
! at com.mesosphere.dcos.cassandra.executor.tasks.UploadSnapshot.run(UploadSnapshot.java:82) [cassandra-executor.jar:na]
! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_74]
! at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_74]
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_74]
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_74]
! at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]

[triclambert]

This repo is deprecated and will be archived in one week. Please see the latest version of Cassandra or DSE for DC/OS:

https://docs.mesosphere.com/service-docs/cassandra/
https://docs.mesosphere.com/service-docs/dse/ (enterprise-only)