SAM deploy requires CAPABILITY_AUTO_EXPAND for the aws-sam-cli-managed-default stack in eu-south-1 region

[cmogeni]

Description:

When running the SAM deploy command in the eu-south-1 region, it requires CAPABILITY_AUTO_EXPAND for the "aws-sam-cli-managed-default". Currently the SAM CLI creates this stack with the "CAPABILITY_IAM" capability but in this case it requires CAPABILITY_AUTO_EXPAND

Steps to reproduce:

The only difference with the normal flow is that the stack was "aws-sam-cli-managed-default" stack has been deleted, but it should still be possible to run "sam deploy" because that should auto-detect and recreate the stack, which it does but it runs into the capabilities issue described above

Observed result:

sam deploy --stack-name mystack --region eu-south-1 --resolve-s3 --s3-prefix redirect-lambda --parameter-overrides MyParam=param Myparam2=param2 --debug

2023-11-10 11:51:03,053 | Config file location: /myproject/samconfig.toml
2023-11-10 11:51:03,057 | Loading configuration values from [default.['deploy'].parameters] (env.command_name.section) inconfig file at'/myproject/samconfig.toml'...
2023-11-10 11:51:03,059 | Configuration values successfully loaded.
2023-11-10 11:51:03,060 | Configuration values are: {'stack_name': 'mystack', 'capabilities':'CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND', 'confirm_changeset': False, 'resolve_s3': False}
2023-11-10 11:51:03,074 | Using SAM Template at /myproject/.aws-sam/build/template.yaml
2023-11-10 11:51:03,144 | Using config file: samconfig.toml, config environment: default
2023-11-10 11:51:03,146 | Expand command line arguments to:
2023-11-10 11:51:03,148 | --template_file=/myproject/.aws-sam/build/template.yaml --stack_name=mystack --resolve_s3 --s3_prefix=redirect-lambda --fail_on_empty_changeset --on_failure=ROLLBACK --capabilities=['CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND']
2023-11-10 11:51:04,217 | Managed S3 stack [aws-sam-cli-managed-default] not found. Creating a new one. Creating the required resources...
2023-11-10 11:51:04,448 | Failed to create managed resources
Traceback (most recent call last):
  File "samcli/lib/utils/managed_cloudformation_stack.py", line 166, in _create_or_get_stack
  File "samcli/lib/utils/managed_cloudformation_stack.py", line 250, in _create_stack
  File "botocore/client.py", line 535, in _api_call
  File "botocore/client.py", line 980, in _make_api_call
botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the CreateChangeSet operation: Insufficient
capabilities: [CAPABILITY_IAM], Required capabilities are: [CAPABILITY_AUTO_EXPAND]
2023-11-10 11:51:04,453 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-11-10 11:51:04,470 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-11-10 11:51:04,472 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'xx',
'installationId': 'xx', 'sessionId': 'xx',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.3', 'samcliVersion': '1.98.0', 'awsProfileProvided': True,
'debugFlagProvided': True, 'region': 'eu-south-1', 'commandName': 'sam deploy', 'metricSpecificAttributes': {'projectType':
'CFN', 'gitOrigin': None, 'projectName': 'xx', 'initialCommit':
None}, 'duration': 1308, 'exitReason': 'ManagedStackError', 'exitCode': 1}}]}
2023-11-10 11:51:04,472 | Unable to find Click Context for getting session_id.
2023-11-10 11:51:04,479 | Sending Telemetry: {'metrics': [{'events': {'requestId': 'xx',
'installationId': 'xx', 'sessionId': 'xx',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.3', 'samcliVersion': '1.98.0', 'commandName': 'sam deploy',
'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id':
'12380f0e6e90403597f57f92a5435e92', 'time_stamp': '2023-11-10 10:51:03.053', 'exception_name': None}, {'event_name':
'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': 'xx', 'time_stamp': '2023-11-10
10:51:03.131', 'exception_name': None}]}}}]}
2023-11-10 11:51:05,602 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
2023-11-10 11:51:05,604 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
Error: Failed to create managed resources: An error occurred (ValidationError) when calling the CreateChangeSet operation: Insufficient capabilities: [CAPABILITY_IAM], Required capabilities are: [CAPABILITY_AUTO_EXPAND]

Expected result:

It should recreate the stack "aws-sam-cli-managed-default" stack successfully

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: linux + codebuild
2. sam --version: tested both 1.98.0 and 1.101.0
3. AWS region: eu-south-1

# Paste the output of `sam --info` here

Add --debug flag to command you are running

[hawflau]

Hi @cmogeni, thanks for raising the issue. I could reproduce the issue. Seems like "eu-south-1" is one of the regions with this error. The managed stack was created successfully in other regions like "np-northeast-1"

[jysheng123]

Hi, added a fix for now that will be incorporated in the next SAM-CLI release, however this is quite an interesting bug due to this being a region focused issue. The team will do a deeper dive in the future to figure out the root cause of this but this error won't happen after the next release. Thanks for bring it up

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[jysheng123]

Hi, after digging into this, this is a bug on CloudFormation, we are in communication with them and they are working on a fix, will update this issue again when it is fixed on their side.

[hawflau]

Hi @cmogeni, we've just confirmed CloudFormation has deployed their fix, and our test in creating a managed stack in eu-south-1 worked.

Closing the issue. Feel free to reopen if you still see the same problem.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.