chore: update Python runtime and AWS CDK #38

ceholden · 2024-10-17T22:52:46Z

What I am changing

This PR addresses issue #37 by updating,

Updates the (now deprecated) Python 3.8 runtime to Python ~~3.12~~ 3.11
- I wanted to update to Python 3.12, but alas there is a CDK bug with Python 3.12 runtimes and pipenv, (aws-lambda-python-alpha): I'm failing to bundle a Python 3.12 function using pipenv. aws/aws-cdk#30170. The recommendation is to downgrade to Python 3.11. Once this is fixed we should have a clear upgrade path to 3.12
Updates the (now deprecated) AWS CDK v1 to CDK v2. This is required to support Python versions >=3.10
- NB - This requires that our AWS accounts be bootstrapped for v2 of CDK, which should already be done because hls-orchestration is using v2 of CDK

I'm also hoping that some of the version bumps will prune down on the 82 Dependabot security warnings.

How I did it

Bumping Python versions

I bumped to the current latest Python runtime (3.12), but the CDK has a bug with that version so we're stuck on 3.11 for now. This required updating a few package dependencies. I saw that most of the Pipfiles used hard pinned versions for dependencies even though the versions are pinned in the lockfile, so I tried to keep to that convention. It would be easy to update this if we want to do that.

Updating psycopg2 to >=2.9.9 to support Python 3.12
- See CHANGELOG entry
Remove now deprecated 3rd party psycopg2-layer from AWS Lambda layers. The db layer installs the same psycopg2-binary as the psycopg2-layer so we should be able to get that dependency from our own db layer
- This only exists up until Python 3.9 and is now no longer maintained, https://github.com/jetbridge/psycopg2-lambda-layer?tab=readme-ov-file#no-longer-maintained
Update boto3 to >=1.29.0 to support Python 3.12 for Boto3 and the Python 3.12 Lambda runtime
- See CHANGELOG
One of the packages didn't pin moto and I hit a breaking change in v5, so I went ahead and updated all of the packages that use Moto to the latest v5
- From v4 to v5 they had a breaking change that migrated all mock_[service] decorator functions to just be mock_aws.
- There was one other hangup - one of the link_fetcher tests failed because the mocked SQS queues apparently share the same state. I resolved this by naming each queue based on the test name.
- See CHANGELOG for v5
Switched from black/isort/flake8 to ruff as one linter/formatter to rule them all that is super fast

Bumping AWS CDK

See migration guide for reference,
https://docs.aws.amazon.com/cdk/v2/guide/migrating-v2.html

The changes and related changes include,

AWS consolidated all the non-beta functionality into just one package, simplifying our management of CDK dependencies
Update imports from previously named aws_cdk.core (now in aws_cdk module)
Remove CDK flags that have been removed in V2. These flags in V2 now behave as if the variable were set to "true" in v1
- core:enableStackNameDuplicates
- aws-secretsmanager:parseOwnedSecretName

How you can test it

Run the very nice test suite!

Running from the repo root,

# make sure to have latest dependencies installed
$ make install
# with dummy postgres variables
$ PG_DB=foo PG_PASSWORD=foo PG_USER=foo make unit-tests

We can also help test that the CDK update is working without deploying by checking the CDK diff,

npx cdk diff

ciaransweet · 2024-10-18T09:19:35Z

@ceholden gimme a ping when this is ready for review!

Always happy this little project is still alive and kicking 😍

ceholden · 2024-10-18T13:38:09Z

@ceholden gimme a ping when this is ready for review!

Always happy this little project is still alive and kicking 😍

Thanks @ciaransweet! Appreciate the good work you did especially with docs, tests, and dev tooling. This sort of maintenance work feels a lot safer with all of that in place 💯 🚀

I'm hoping to wrap this up ahead of adding the code I've written to feed the granule downloader from ESA's notification system. From preliminary testing, moving to an event driven "link fetcher" should help us reduce download latency to ~3 minutes from new data publication time and also allow us to catch old granules that have been reprocessed by ESA. Not sure if you can access but I have a writeup of the experiment findings and plan for next steps in this ticket, https://github.com/NASA-IMPACT/hls_development/issues/300

I think I'll be fighting with some of the package version bumps some more, but would love your help reviewing when it's in a good place 😸

ceholden · 2024-10-21T21:55:11Z

cdk.json

-    "@aws-cdk/core:enableStackNameDuplicates": "true",
    "aws-cdk:enableDiffNoFail": "true",
    "@aws-cdk/core:stackRelativeExports": "true",
    "@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true,
-    "@aws-cdk/aws-secretsmanager:parseOwnedSecretName": true,


These are removed in CDK v2 and act as if they were all true

ceholden · 2024-10-21T21:58:31Z

lambdas/link_fetcher/tests/conftest.py

+def mock_sqs_queue(request, sqs_resource, monkeysession, sqs_client):
+    queue = sqs_resource.create_queue(QueueName=f"mock-queue-{request.node.name}"[:80])


In Moto v5 these queues apparently share state, so tests were failing because the queues from one test were impacting other tests. This change resolves the issue by giving unique names per test

ceholden · 2024-10-21T22:00:57Z

setup.py

+aws_cdk_version = "2.162.1"
 inst_reqs = [
-    *[f"aws_cdk.{x}=={aws_cdk_version}" for x in aws_cdk_reqs],
+    f"aws-cdk-lib=={aws_cdk_version}",
+    f"aws-cdk.aws-lambda-python-alpha=={aws_cdk_version}a0",


CDK v2 only has 1 main package but splits anything in "alpha" or "beta" off into separate packages. We're using PythonFunction from aws-cdk.aws-lambda-python-alpha so we need this additional package

ciaransweet · 2024-10-25T12:07:44Z

README.md

@@ -21,9 +21,9 @@ This project aims to provide a serverless implementation of the current [HLS S2

 To develop on this project, you should install:

-* NVM [Node Version Manager](https://github.com/nvm-sh/nvm) / Node 12
+* NVM [Node Version Manager](https://github.com/nvm-sh/nvm) / Node 18


Weeeew 6 node versions!

README.md

ciaransweet

Awesome job here @ceholden just a minor comment but looks gucci to me otherwise!

README.md

ceholden · 2024-10-25T16:12:55Z

Thanks to the help of @chuckwondo we're now past the hurdle of setting up CDK v2 bootstrap in us-west-2 for the staging account 🎉. The deploy user we're using for dev requires some additional IAM permission grants that I'm working through before we can deploy the integration test stack. I think I've gotten past that hurdle just now.

It looks like we also need to add a permissions boundary to the stacks because our deploy user's permissions to do things (e.g., create roles) depends on the permissions boundary being set for those deployed resources. Chuck's most recent PR has a nice example of what we need to do for this

chuckwondo

Looks good to me! Fingers crossed on getting integration tests to deploy and run.

ceholden · 2024-10-28T14:22:23Z

@chuckwondo thanks for your review and help on Friday! I did get the integration tests to run and succeed, but had a lot of trouble with misconfiguration in the staging account permissions boundaries. I'll ask around on Slack about next steps to fix this issue, but no code change should be required on this end

chuckwondo · 2024-10-28T15:26:05Z

@chuckwondo thanks for your review and help on Friday! I did get the integration tests to run and succeed, but had a lot of trouble with misconfiguration in the staging account permissions boundaries. I'll ask around on Slack about next steps to fix this issue, but no code change should be required on this end

Can you elaborate on the misconfiguration? I'm not following, especially since int. tests have succeeded.

ceholden · 2024-10-28T18:18:50Z

@chuckwondo I'll followup on Slack with more info, but FYI I re-ran the integration tests to reproduce the issue after updating the Github environment. I'm going to edit the "dev" Github environment to remove the permissions boundary and re-run to ensure the integration tests are still passing

ceholden added 10 commits October 17, 2024 17:05

Bump db layer to Py3.12

804a868

Bump root devtools to Py3.12

c69c24f

Bump date_generator to Py3.12

bc7486d

Bump downloader to Py3.12

64a77de

Bump link_fetcher to Py3.12

9a02ce9

Bump mock_scihub_product_api to Py3.12

601b185

Bump mock_scihub_search_api to Py3.12

b615df5

Bump requeuer to Py3.12

19905aa

Bump Lambda runtimes to Py3.12

1494ffe

Install Py3.12 in GH Actions

18f848c

ceholden added 3 commits October 18, 2024 17:01

Bump and relock flake8 & isort

4ccac64

Bump boto3 and psycopg2 to supported versions

a962208

Bump moto and update decorator usage for v5 (only mock_aws now)

5144598

ceholden force-pushed the ceh/issue37 branch from 878feb1 to 6411d14 Compare October 18, 2024 21:35

Upgrade alembic_migration

4265a12

ceholden force-pushed the ceh/issue37 branch from 6411d14 to 4265a12 Compare October 18, 2024 21:41

ceholden added 3 commits October 18, 2024 17:54

Mocked SQS queue name is unique per test

cbe422f

Update alembic_migration use of moto^=5

9fa7da8

Try to fix missing urllib3 despite it being in lockfile

dc928c8

ceholden had a problem deploying to dev October 18, 2024 22:22 — with GitHub Actions Failure

ceholden added 3 commits October 21, 2024 16:21

Support CDK v2

d82a2b1

sigh downgrade to 3.11 because of aws/aws-cdk#30170

c2a9387

Bump boto3/botocore + requests

d1476f7

ceholden force-pushed the ceh/issue37 branch from 899f455 to d1476f7 Compare October 21, 2024 21:19

fix lint

88517e5

ceholden had a problem deploying to dev October 21, 2024 21:38 — with GitHub Actions Failure

ceholden commented Oct 21, 2024

View reviewed changes

ceholden changed the title ~~[DRAFT] chore: update Python runtime and AWS CDK~~ chore: update Python runtime and AWS CDK Oct 21, 2024

replace black/isort/flake8 with ruff

3fa51a8

ceholden force-pushed the ceh/issue37 branch from ba1ea4f to 3fa51a8 Compare October 24, 2024 17:17

ceholden had a problem deploying to dev October 24, 2024 17:25 — with GitHub Actions Failure

ciaransweet reviewed Oct 25, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

ciaransweet suggested changes Oct 25, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

ceholden had a problem deploying to dev October 25, 2024 15:02 — with GitHub Actions Failure

ceholden had a problem deploying to dev October 25, 2024 15:37 — with GitHub Actions Failure

ceholden added 2 commits October 25, 2024 17:45

use package.json as source of truth for cdk install

728c905

Deploy stack w/ permissions boundary on resources

0cf181e

ceholden had a problem deploying to dev October 25, 2024 21:54 — with GitHub Actions Failure

Specify permissions boundary via full ARN

0afe820

ceholden had a problem deploying to dev October 25, 2024 22:15 — with GitHub Actions Failure

ceholden had a problem deploying to dev October 25, 2024 22:43 — with GitHub Actions Failure

Ensure permissions boundary is passed via .env

aef6c72

ceholden force-pushed the ceh/issue37 branch from 0800cf3 to aef6c72 Compare October 25, 2024 22:54

ceholden had a problem deploying to dev October 25, 2024 22:57 — with GitHub Actions Failure

ceholden temporarily deployed to dev October 25, 2024 23:14 — with GitHub Actions Inactive

cleanup references to isort/flake8/black

7c8b3b6

ceholden temporarily deployed to dev October 25, 2024 23:40 — with GitHub Actions Inactive

ciaransweet approved these changes Oct 28, 2024

View reviewed changes

chuckwondo approved these changes Oct 28, 2024

View reviewed changes

ceholden had a problem deploying to dev October 28, 2024 18:10 — with GitHub Actions Failure

ceholden temporarily deployed to dev October 28, 2024 18:19 — with GitHub Actions Inactive

ceholden merged commit 97096cf into main Oct 30, 2024
3 checks passed

ceholden deleted the ceh/issue37 branch October 30, 2024 20:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: update Python runtime and AWS CDK #38

chore: update Python runtime and AWS CDK #38

ceholden commented Oct 17, 2024 •

edited

Loading

ciaransweet commented Oct 18, 2024

ceholden commented Oct 18, 2024

ceholden Oct 21, 2024

ceholden Oct 21, 2024

ceholden Oct 21, 2024

ciaransweet Oct 25, 2024

ciaransweet left a comment

ceholden commented Oct 25, 2024

chuckwondo left a comment

ceholden commented Oct 28, 2024

chuckwondo commented Oct 28, 2024

ceholden commented Oct 28, 2024

		def mock_sqs_queue(request, sqs_resource, monkeysession, sqs_client):
		queue = sqs_resource.create_queue(QueueName=f"mock-queue-{request.node.name}"[:80])

chore: update Python runtime and AWS CDK #38

chore: update Python runtime and AWS CDK #38

Conversation

ceholden commented Oct 17, 2024 • edited Loading

What I am changing

How I did it

Bumping Python versions

Bumping AWS CDK

How you can test it

ciaransweet commented Oct 18, 2024

ceholden commented Oct 18, 2024

ceholden Oct 21, 2024

Choose a reason for hiding this comment

ceholden Oct 21, 2024

Choose a reason for hiding this comment

ceholden Oct 21, 2024

Choose a reason for hiding this comment

ciaransweet Oct 25, 2024

Choose a reason for hiding this comment

ciaransweet left a comment

Choose a reason for hiding this comment

ceholden commented Oct 25, 2024

chuckwondo left a comment

Choose a reason for hiding this comment

ceholden commented Oct 28, 2024

chuckwondo commented Oct 28, 2024

ceholden commented Oct 28, 2024

ceholden commented Oct 17, 2024 •

edited

Loading