Upgrade AWS SDK Go to v2 #191
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Although aws-sdk-go-v2 was released long ago, historically, until now, tfmigrate has used v1 as well as the old Terraform. aws-sdk-go and aws-sdk-go-v2 have different priorities for loading credentials. Specifically, aws-sdk-go v1 prioritizes reading environment variables over profiles. This is a problem when switching between multiple accounts in a profile while OIDC authenticating to AWS from GitHub Actions. Up to Terraform v1.5, aws-sdk-go v1 was used for S3 backend authentication via hashicorp/aws-sdk-go-base, but aws-sdk-go-v2 is used for Terraform / OpenTofu v1.6 and later. Up to tfmigrate v0.3, we have been using aws-sdk-go v1 via hashicorp/aws-sdk-go-base for history s3 storage authentication but will be using aws-sdk-go-v2 after the next tfmigrate v0.4. This is a breaking change, but the goal is to align with the behavior of Terraform / OpenTofu v1.6 and later, so if you are affected, please adjust your AWS authentication settings. At the time of this writing, v2 of hashicorp/aws-sdk-go-base, which uses aws-sdk-go-v2, is still beta, but it is already used in the stable releases of Terraform and OpenTofu. Since there are multiple beta versions of hashicorp/aws-sdk-go-base and slightly different versions in use, the results of my investigation are shown below. - tfmigrate: v0.3.24: v1.1.0 - terraform v1.5.7: v0.7.1 - terraform v1.6.0: v2.0.0-beta.35 - terraform v1.7.0: v2.0.0-beta.43 - terraform v1.8.0: v2.0.0-beta.45 - terraform v1.9.8 (latest): v2.0.0-beta.45 - opentofu v1.6.0: v2.0.0-beta.43 - opentofu v1.8.5 (latest): v2.0.0-beta.43 - terraform-provider-aws v3.76.1: v1.0.0 - terraform-provider-aws v4.0.0: v2.0.0-beta.5 - terraform-provider-aws v5.0.0: v2.0.0-beta.27 - terraform-provider-aws v5.29.0: v2.0.0-beta.43 - terraform-provider-aws v5.31.0: v2.0.0-beta.45 - terraform-provider-aws v5.74.0 (latest): v2.0.0-beta.59 As a special note, use_legacy_workflow has been removed in Terraform v1.8.0 and OpenTofu v1.8.0, but UseLegacyWorkflow has been removed in aws-sdk-go-base in v2.0.0-beta.49. Considering the above, I think v2.0.0-beta.43 or v2.0.0-beta.45 would be reasonable. However, at this point, I don't see any solid reason for choosing v2.0.0-beta.45, so I will select v2.0.0-beta.43 to align with OpenTofu. I rewrote the s3 implementation for history storage, referring to the aws-sdk-go-v2 migration guide. https://aws.github.io/aws-sdk-go-v2/docs/migrating/ From the tfmigrate user's perspective, there are no breaking changes at the configuration file level. Still, it should be noted that AWS credentials have higher precedence in profiles than in environment variables. Also, as a notable side effect of the upgrade to v2, the path to the file configuring the AWS profile is ~/.aws/config by default, but it can now be specified in the environment variable AWS_CONFIG_FILE. This will be useful for customizing the configuration in a CI/CD environment.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Although aws-sdk-go-v2 was released long ago, historically, until now, tfmigrate has used v1 as well as the old Terraform.
aws-sdk-go and aws-sdk-go-v2 have different priorities for loading credentials. Specifically, aws-sdk-go v1 prioritizes reading environment variables over profiles. This is a problem when switching between multiple accounts in a profile while OIDC authenticating to AWS from GitHub Actions.
Up to Terraform v1.5, aws-sdk-go v1 was used for S3 backend authentication via hashicorp/aws-sdk-go-base, but aws-sdk-go-v2 is used for Terraform / OpenTofu v1.6 and later.
Up to tfmigrate v0.3, we have been using aws-sdk-go v1 via hashicorp/aws-sdk-go-base for history s3 storage authentication but will be using aws-sdk-go-v2 after the next tfmigrate v0.4. This is a breaking change, but the goal is to align with the behavior of Terraform / OpenTofu v1.6 and later, so if you are affected, please adjust your AWS authentication settings.
At the time of this writing, v2 of hashicorp/aws-sdk-go-base, which uses aws-sdk-go-v2, is still beta, but it is already used in the stable releases of Terraform and OpenTofu. Since there are multiple beta versions of hashicorp/aws-sdk-go-base and slightly different versions in use, the results of my investigation are shown below.
As a special note, use_legacy_workflow has been removed in Terraform v1.8.0 and OpenTofu v1.8.0, but UseLegacyWorkflow has been removed in aws-sdk-go-base in v2.0.0-beta.49.
Considering the above, I think v2.0.0-beta.43 or v2.0.0-beta.45 would be reasonable. However, at this point, I don't see any solid reason for choosing v2.0.0-beta.45, so I will select v2.0.0-beta.43 to align with OpenTofu.
I rewrote the s3 implementation for history storage, referring to the aws-sdk-go-v2 migration guide.
https://aws.github.io/aws-sdk-go-v2/docs/migrating/
From the tfmigrate user's perspective, there are no breaking changes at the configuration file level. Still, it should be noted that AWS credentials have higher precedence in profiles than in environment variables.
Also, as a notable side effect of the upgrade to v2, the path to the file configuring the AWS profile is ~/.aws/config by default, but it can now be specified in the environment variable AWS_CONFIG_FILE. This will be useful for customizing the configuration in a CI/CD environment.