GC AWS Old Images (from packer)

[smerle33]

Service(s)

AWS

Summary

handle a Garbage Collector to remove OLD AMI version

Should be split in 3 steps :

• dev (done by feat(GC AWS AMI): Garbage collector for DEV images packer-images#201 )
• staging
• prod

Reproduction steps

No response

[dduportal]

Reminder of the "rules" for the GC of AMIs (I can't find my previous comment so I'm adding it here as asked :) ).

Rule of thumb I propose is "1d, 1m, 1y":

• "Dev" images (e.g. created by a PR build or a developer local build): if an AMI tagged as "dev" is older than 24h, then 🗑️
  • Rationale: these images are short lived, only to demonstrate that a PR does not break the build and have the intented purpose.
• "Staging" images (e.g. created by a build of the main branch, which purpose it to ensure that "it just works"): if an AMI tagged as "staging is older than 1 week, then 🗑️
  • Rationale: the principal branch is built once per week, hence no need to keep older versions
• "Production" images (e.g. created by a tag build and used in production): if an AMI tagged as "production" is older than 1 year, then 🗑️
  • Rationale: 1 year is a laaaaaareg margin to really ensure that we clean things up. I personnaly would try a "3 months" rules so if we do not update an AMI soemwhere, then we'll see it fail in 3 month, but given the low amount of tagged images, no need to worry.

[dduportal]

Addon: we also have to delete the snapshot along with AMI.

The AWS documentation at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/deregister-ami.html describe the process: when you want to "delete an AMI", it's a 4 steps task:

1. Get the AMI ID (already done by our script)
2. Get the snapshot ID associated with this AMI
3. Deregister the AMI (already done by our script)
4. Delete the snapshot

[dduportal]

• Implemented a snapshots cleanup for 1 year-old snapshots as per Spring 2023: Decrease AWS costs #3502
  • chore(cleanup AWS) garbage collect EC2 snapshots older than 1 year on us-east-2 packer-images#587
  • Fixup with chore(cleanup AWS snapshots) fixup of #587 packer-images#589
• Implemented cleaning up of staging images inhttps://github.com/chore(cleanup AWS images) ensure staging images older than 7 days are removed packer-images#590

Checking the result

[dduportal]

• Result is promising. We are down to 512 AMIS (instead of more than 3000 before).
• Updating the "Rule of thumb" to:
  • dev => 1 day
  • staging => 1 week
  • prod => 2 months
• implement prod cleanup - chore(cleanup AWS images) remove AMI with build_type=prod older than 2 months packer-images#594
  • Went from ~512 AMIs to only 163 with this change
• change the snapshots threshold to 2 month instead of 1 year (and see the result before decreasing it to 1 week and then 1 day) - chore(cleanup AWS snapshots) decrease threshold to 2 month packer-images#595

[dduportal]

Manually deregistered the folloiwng AMIs:

• Ubuntu 20.x (ref. Ubuntu 22.04 upgrade campaign #2982)
• AMIs in us-east-2 with no tags at all
• AMIS oldere than 2 month

=> we're down to 72 AMIs (instead of 3000s).

Time to clean up the snapshots now!

[dduportal]

• Result of running chore(cleanup AWS snapshots) decrease threshold to 2 month packer-images#595 in production: snapshots amount went from ~ 4300 to 1001 now 👍

[dduportal]

• chore(cleanup AWS snapshots) try to delete when older than 1 day packer-images#597 result is: 85 snapshots left. That should help!