Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to delete Cloudformation stacks #106

Open
andreamaruccia opened this issue Jun 29, 2020 · 13 comments
Open

Unable to delete Cloudformation stacks #106

andreamaruccia opened this issue Jun 29, 2020 · 13 comments

Comments

@andreamaruccia
Copy link

I've this config:

aws_cloudformation_stack:
  - id: awsweeper-del-me

running it I get these logs:

awsweeper --region eu-west-1 --parallel 1 --debug --force config.yaml

   • downloaded and installed provider                  name=aws version=2.59.0
   • configured provider                                name=aws version=2.59.0
   • SHOWING RESOURCES THAT WOULD BE DELETED (DRY RUN)
<snipping>
      • failed to import resource; trying to read resource without import error=Throttling: Rate exceeded
        status code: 400, request id: 18399ea7-5742-409a-9e62-14cf0a14cb51 id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72 type=aws_cloudformation_stack
Error: failed to read current state of resource: Throttling: Rate exceeded
        status code: 400, request id: dca88124-cc18-4375-8aaf-39bcd0bbd6dd
Error: failed to read current state of resource: Throttling: Rate exceeded
        status code: 400, request id: cc2e00e1-1554-4bd3-9ca9-f235fd738e37
Error: failed to read current state of resource: Throttling: Rate exceeded
        status code: 400, request id: f78f05c8-a354-4465-92d1-5e1099bae9b5
<snipping>
      • failed to get tags        error=state is nil: <nil> id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72 type=aws_cloudformation_stack
<snipping>
        ---
        Type: aws_cloudformation_stack
        Found: 1

                Id:             arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72
                Created:        2020-06-29 09:40:07.952 +0000 UTC

        ---

   • TOTAL NUMBER OF RESOURCES THAT WOULD BE DELETED: 1
   • USER WILL NOT BE ASKED FOR CONFIRMATION (FORCE MODE)
   • STARTING TO DELETE RESOURCES
      • start distributing resources to workers for this run
      • unable to delete resource                          error=resource state is nil; need to call update first resource_id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72 type=aws_cloudformation_stack
   • TOTAL NUMBER OF DELETED RESOURCES: 0

and I can still see the cfn stack in the aws console :(

I've tried it with version 0.7 and it seems to work:

 awsweeper --region eu-west-1 --debug --force config.yaml        
   • downloaded and installed provider                  name=aws version=2.43.0
   • configured provider                                name=aws version=2.43.0
   • using region: eu-west-1  
   • SHOWING RESOURCES THAT WOULD BE DELETED (DRY RUN)

        ---
        Type: aws_cloudformation_stack
        Found: 1

                Id:             arn:aws:cloudformation:eu-west-1:<redacted>/:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72
        Created:        2020-06-29 09:40:07.952 +0000 UTC

        ---

   • TOTAL NUMBER OF RESOURCES THAT WOULD BE DELETED: 1
   • USER WILL NOT BE ASKED FOR CONFIRMATION (FORCE MODE)
   • STARTING TO DELETE RESOURCES
      • start distributing resources to workers for this run
      • worker starts deleting resource                    id=arn:aws:cloudformation:eu-west-1:<redacted>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72 type=aws_cloudformation_stack worker_id=5
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72
   • TOTAL NUMBER OF DELETED RESOURCES: 1
@jckuester
Copy link
Owner

jckuester commented Jun 30, 2020 •
edited
Loading

Hi @andreamaruccia, I've also noticed the throttling exception that appears somehow if the AWS API is called too often. I added some retry logic that should fix the problem you are seeing: https://github.com/jckuester/awsweeper/releases/tag/v0.10.1

And yep, in 0.7 it worked because I didn't use the Terraform AWS Provider back then to support so many resources.

@andreamaruccia
Copy link
Author

hi @jckuester it seems this helps already as I can delete cloudformation stacks that I've created recently.
But when it comes to delete an older one, it seems that it doesn't delete it somehow.

This is what works well:

  1. create a cfn stack with this yaml and name it awsweeper-test
Description: >
  Stack deployment test for awsweeper

Parameters:
  ManagedPolicyName:
    Type: String
    Default: default-managed-policy-name

Resources:
  noCloudTrail:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Sub "${ManagedPolicyName}-${AWS::Region}"
      Description: desc
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: MoleculeTest
            Effect: Deny
            Action:
              - cloudtrail:StopLogging
            Resource: "*"
Outputs:
  ExampleOutput:
    Description: This is an example output
    Value: !Ref noCloudTrail

use this configuration:

aws_cloudformation_stack:
  - id: awsweeper-test

This doesn't:

  1. Take an old cloudformation template named old-rotten-poc created on 2019-04-18 14:56:06 UTC+0200
  2. use this configuration:
aws_cloudformation_stack:
  - id: old-rotten-poc

I wonder if this is due to pagination or such now that the rate limiting is out of the way

@jckuester
Copy link
Owner

Is there any useful debug output of awsweeper for old-rotten-poc?

@andreamaruccia
Copy link
Author

that's the weired thing it just says it's being deleted:

    • downloaded and installed provider                  name=aws version=2.68.0
    • configured provider                                name=aws version=2.68.0
    • SHOWING RESOURCES THAT WOULD BE DELETED (DRY RUN)
       • retrying to read current state of resource error=Throttling: Rate exceeded
 	status code: 400, request id: 11699a25-0a7e-48b2-8bef-135e11ba896b
       • retrying to read current state of resource error=Throttling: Rate exceeded
 	status code: 400, request id: a73faa95-4a90-41d9-957e-e283627b0f7e
<snip>
       • retrying to read current state of resource error=Throttling: Rate exceeded
 	status code: 400, request id: dbe3094a-7635-457e-8069-524b87829467
       • failed to import resource; trying to read resource without import error=Throttling: Rate exceeded
 	status code: 400, request id: dbe3094a-7635-457e-8069-524b87829467 id=arn:aws:cloudformation:eu-west-1:<redacted>:stack/<redacted-stack-name>/57d1c6c0-1da0-11ea-be65-0ace5b4cd800 type=aws_cloudformation_stack
    • TOTAL NUMBER OF RESOURCES THAT WOULD BE DELETED: 14
    • USER WILL NOT BE ASKED FOR CONFIRMATION (FORCE MODE)
    • STARTING TO DELETE RESOURCES
       • start distributing resources to workers for this run
 	---
 	Type: aws_cloudformation_stack
 	Found: 14
 		Id:		arn:aws:cloudformation:eu-west-1:<redacted>:stack/<redacted-stackname>/6261ff90-9e85-11ea-998c-0224adcab7c8
 		Created:	2020-05-25 12:43:50.775 +0000 UTC
 		Id:		arn:aws:cloudformation:eu-west-1:<redacted>:stack/<redacted-stackname>/d7b48310-99a0-11ea-ac4f-0a0e6fcb5aa0
 		Created:	2020-05-19 07:17:48.315 +0000 UTC
 		Id:		arn:aws:cloudformation:eu-west-1:<redacted>:stack/<redacted-stackname>/d7ae6890-99a0-11ea-9143-0a0af0d573f8
 		Created:	2020-05-19 07:17:48.186 +0000 UTC
 		Id:		arn:aws:cloudformation:eu-west-1:<redacted>:stack/<redacted-stackname>/d7a95f80-99a0-11ea-8769-02fa7ab4594c
 		Created:	2020-05-19 07:17:48.155 +0000 UTC
 		Id:		arn:aws:cloudformation:eu-west-1:<redacted>:stack/<redacted-stackname>/d1bd4280-99a0-11ea-8474-0ae0d664c880
<snip>
 	---
       ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:473444336670:stack/<redacted-stackname>/3348cbf0-6dd2-11ea-9d48-0a9bdcf5c20a
       ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:473444336670:stack/<redacted-stackname>/6261ff90-9e85-11ea-998c-0224adcab7c8
       ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:473444336670:stack/<redacted-stackname>/d7a95f80-99a0-11ea-8769-02fa7ab4594c
<snip>
    • TOTAL NUMBER OF DELETED RESOURCES: 14

@jckuester
Copy link
Owner

jckuester commented Jul 1, 2020 •
edited
Loading

@andreamaruccia, hmm, hard to say from here what's wrong. Is it just a single resource that doesn't get deleted or many? If you run awsweeper again, the resource(s) still stay?

@andreamaruccia
Copy link
Author

andreamaruccia commented Jul 1, 2020 •
edited
Loading

@jckuester all the resources remain, not a single one was deleted (14 in my case)
Is there a way to increase the log level somehow on the resource deletion commands?

edit: yes I ran it 2/3 times in a row, and also at night (in a schedule)

@jckuester
Copy link
Owner

jckuester commented Jul 2, 2020 •
edited
Loading

I created a test myself and deleting of aws_cloudformation_stack resources works fine. But you said it just doesn't work with old stacks, right? Deletion also doesn't work with 0.7 then, I assume?

Is there a way to increase the log level somehow on the resource deletion commands?

Not officially, but I can make a code change to enable TRACE output of the Terraform AWS provider, which performs the destroy. I created a branch for you where I enabled TRACE output: debug/terraform-provider-trace-output

Are you able to checkout that branch and go build it or do you want me to send you a binary?

@andreamaruccia
Copy link
Author

andreamaruccia commented Jul 2, 2020 •
edited
Loading

I created a test myself and deleting of aws_cloudformation_stack resources works fine. But you said it just doesn't work with old stacks, right? Deletion also doesn't work with 0.7 then, I assume?

Yes I confirm 0.7 doesn't work either. My test with 0.7 was deleting the newly created test resource

Are you able to checkout that branch and go build it or do you want me to send you a binary?

yes I will do it, thanks for enabling me to test it

@andreamaruccia
Copy link
Author

Here are the logs with trace:

<snip>
        Enter a value: YES
   • STARTING TO DELETE RESOURCES
      • start distributing resources to workers for this run
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/f275c070-0a0c-11ea-a4cc-0a7ffa9bc424
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/d7a95f80-99a0-11ea-8769-02fa7ab4594c
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/3891f230-6dd2-11ea-9f24-06e58f87e324
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/62fb7cd0-0a0e-11ea-b12b-0aa024806772
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/38b3ab00-6dd2-11ea-b898-0aac29486886
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/3348cbf0-6dd2-11ea-9d48-0a9bdcf5c20a
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/d7b48310-99a0-11ea-ac4f-0a0e6fcb5aa0
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/d1bd4280-99a0-11ea-8474-0ae0d664c880
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>-S3Stack-XY1NP9J0ZM1E/6261ff90-9e85-11ea-998c-0224adcab7c8
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>-SQSStack-W5K1OU0J2JJ3/d7ae6890-99a0-11ea-9143-0a0af0d573f8
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/bd4b7460-ca3f-11e9-8977-069eaf5ad9a0
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/d48eb560-ca44-11e9-be29-0677a5c9174c
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/d7692af0-ca57-11e9-b63b-0a4044c481ac
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/54209610-61d9-11e9-88f7-0a0c5b3ce4f4
   • TOTAL NUMBER OF DELETED RESOURCES: 14

I don't see anything super bad :/

@jckuester
Copy link
Owner

Yeah, nothing bad really to see in the TRACE logs. If Terraform says nothing, the resource should actually be deleted. Don't know what's going on there, honestly. Can you manually delete the resources in the AWS console?

@andreamaruccia
Copy link
Author

@jckuester I solved the mistery, these resource have termination protection on this is why it wouldn't delete them. By removing it it was able to delete it.

I wonder what the behavior should be in this case to be honest.
I can see two options to improve this:

  1. Add a log when there is a protection to make it clear that it wont be deleted
  2. Add a flag to remove the protection before deleting (like a force)

Anyway I am already happy to understand this

@jckuester
Copy link
Owner

I am glad that you could find the issue, @andreamaruccia. Let's keep this issue open, so that we don't forget about it until I have found a way to deal with deletion protection.

@andreamaruccia
Copy link
Author

thank you @jckuester, sounds good

I am glad that you could find the issue, @andreamaruccia. Let's keep this issue open, so that we don't forget about it until I have found a way to deal with deletion protection.

thank you @jckuester, sounds good

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jckuester @andreamaruccia