From 90d3d155a8347e139a217a02d577373b16c4ceff Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 13 May 2024 19:38:42 +0000 Subject: [PATCH 01/45] Bump cfn-lint from 0.87.1 to 0.87.2 Bumps [cfn-lint](https://github.com/aws-cloudformation/cfn-python-lint) from 0.87.1 to 0.87.2. - [Release notes](https://github.com/aws-cloudformation/cfn-python-lint/releases) - [Changelog](https://github.com/aws-cloudformation/cfn-lint/blob/v0.87.2/CHANGELOG.md) - [Commits](https://github.com/aws-cloudformation/cfn-python-lint/compare/v0.87.1...v0.87.2) --- updated-dependencies: - dependency-name: cfn-lint dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- requirements-all.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements-all.txt b/requirements-all.txt index f0f028a19..62c9c4736 100644 --- a/requirements-all.txt +++ b/requirements-all.txt @@ -17,4 +17,4 @@ flake8-blind-except==0.2.1 flake8-builtins==2.5.0 setuptools==69.5.1 openapi-spec-validator==0.7.1 -cfn-lint==0.87.1 +cfn-lint==0.87.2 From 748b0340f2408faabbaeb18d41996edeaf5ce5cf Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 13 May 2024 19:38:47 +0000 Subject: [PATCH 02/45] Bump cryptography from 42.0.5 to 42.0.7 Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.5 to 42.0.7. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pyca/cryptography/compare/42.0.5...42.0.7) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- requirements-apps-api-binary.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements-apps-api-binary.txt b/requirements-apps-api-binary.txt index 3beaa30e8..ca19e20b4 100644 --- a/requirements-apps-api-binary.txt +++ b/requirements-apps-api-binary.txt @@ -1 +1 @@ -cryptography==42.0.5 +cryptography==42.0.7 From 726d6c9e3f6969a2a1e3fa1f1490cb4c55a1b2eb Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 13 May 2024 19:38:53 +0000 Subject: [PATCH 03/45] Bump moto[dynamodb] from 5.0.6 to 5.0.7 Bumps [moto[dynamodb]](https://github.com/getmoto/moto) from 5.0.6 to 5.0.7. - [Release notes](https://github.com/getmoto/moto/releases) - [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md) - [Commits](https://github.com/getmoto/moto/compare/5.0.6...5.0.7) --- updated-dependencies: - dependency-name: moto[dynamodb] dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- requirements-all.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements-all.txt b/requirements-all.txt index f0f028a19..511f7b727 100644 --- a/requirements-all.txt +++ b/requirements-all.txt @@ -7,7 +7,7 @@ -r requirements-apps-update-db.txt boto3==1.34.100 jinja2==3.1.4 -moto[dynamodb]==5.0.6 +moto[dynamodb]==5.0.7 pytest==8.2.0 PyYAML==6.0.1 responses==0.25.0 From b976853cd65f7bda58fcc0f6cb6daad08637c723 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 13 May 2024 19:48:39 +0000 Subject: [PATCH 04/45] Bump actions/checkout from 4.1.2 to 4.1.5 Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.2 to 4.1.5. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4.1.2...v4.1.5) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- .github/workflows/deploy-whitelisting-sandbox.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/deploy-whitelisting-sandbox.yml b/.github/workflows/deploy-whitelisting-sandbox.yml index 3001fbdc1..a9671a51d 100644 --- a/.github/workflows/deploy-whitelisting-sandbox.yml +++ b/.github/workflows/deploy-whitelisting-sandbox.yml @@ -41,7 +41,7 @@ jobs: url: https://${{ matrix.domain }} steps: - - uses: actions/checkout@v4.1.2 + - uses: actions/checkout@v4.1.5 - uses: aws-actions/configure-aws-credentials@v4 with: From 7e993de6c886255ebe9849546ee8c70eff4cccfb Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Mon, 13 May 2024 15:44:59 -0800 Subject: [PATCH 05/45] binary media types for RestAPI --- CHANGELOG.md | 5 +++++ apps/api/api-cf.yml.j2 | 2 ++ 2 files changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d8b3f5205..d02ea06da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,11 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [7.2.2] + +### Fixed +- Set `BinaryMediaTypes` to `*/*` for the `AWS::ApiGateway::RestApi` resource to allow images to be served via the HyP3 API. Fixes https://github.com/ASFHyP3/hyp3/issues/2278 + ## [7.2.1] ### Fixed diff --git a/apps/api/api-cf.yml.j2 b/apps/api/api-cf.yml.j2 index 975bbf3be..420197645 100644 --- a/apps/api/api-cf.yml.j2 +++ b/apps/api/api-cf.yml.j2 @@ -49,6 +49,8 @@ Resources: RestApi: Type: AWS::ApiGateway::RestApi Properties: + BinaryMediaTypes: + - '*/*' EndpointConfiguration: Types: - {{ 'PRIVATE' if security_environment == 'EDC' else 'REGIONAL' }} From 2ae98fb609c5491f87653631e57c02984b627b9c Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Wed, 15 May 2024 12:30:07 -0800 Subject: [PATCH 06/45] Revert "binary media types for RestAPI" --- CHANGELOG.md | 5 ----- apps/api/api-cf.yml.j2 | 2 -- 2 files changed, 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d02ea06da..d8b3f5205 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,11 +4,6 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [7.2.2] - -### Fixed -- Set `BinaryMediaTypes` to `*/*` for the `AWS::ApiGateway::RestApi` resource to allow images to be served via the HyP3 API. Fixes https://github.com/ASFHyP3/hyp3/issues/2278 - ## [7.2.1] ### Fixed diff --git a/apps/api/api-cf.yml.j2 b/apps/api/api-cf.yml.j2 index 420197645..975bbf3be 100644 --- a/apps/api/api-cf.yml.j2 +++ b/apps/api/api-cf.yml.j2 @@ -49,8 +49,6 @@ Resources: RestApi: Type: AWS::ApiGateway::RestApi Properties: - BinaryMediaTypes: - - '*/*' EndpointConfiguration: Types: - {{ 'PRIVATE' if security_environment == 'EDC' else 'REGIONAL' }} From 2d275abc0ead32eb298cd0d6aa520da425c41869 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 16 May 2024 19:07:12 +0000 Subject: [PATCH 07/45] Bump boto3 from 1.34.100 to 1.34.106 Bumps [boto3](https://github.com/boto/boto3) from 1.34.100 to 1.34.106. - [Release notes](https://github.com/boto/boto3/releases) - [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst) - [Commits](https://github.com/boto/boto3/compare/1.34.100...1.34.106) --- updated-dependencies: - dependency-name: boto3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- requirements-all.txt | 2 +- requirements-apps-disable-private-dns.txt | 2 +- requirements-apps-start-execution-manager.txt | 2 +- requirements-apps-start-execution-worker.txt | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/requirements-all.txt b/requirements-all.txt index 511f7b727..31589084b 100644 --- a/requirements-all.txt +++ b/requirements-all.txt @@ -5,7 +5,7 @@ -r requirements-apps-start-execution-worker.txt -r requirements-apps-disable-private-dns.txt -r requirements-apps-update-db.txt -boto3==1.34.100 +boto3==1.34.106 jinja2==3.1.4 moto[dynamodb]==5.0.7 pytest==8.2.0 diff --git a/requirements-apps-disable-private-dns.txt b/requirements-apps-disable-private-dns.txt index 46c415b8b..334eadc0c 100644 --- a/requirements-apps-disable-private-dns.txt +++ b/requirements-apps-disable-private-dns.txt @@ -1 +1 @@ -boto3==1.34.100 +boto3==1.34.106 diff --git a/requirements-apps-start-execution-manager.txt b/requirements-apps-start-execution-manager.txt index ff011fc59..39a9ac915 100644 --- a/requirements-apps-start-execution-manager.txt +++ b/requirements-apps-start-execution-manager.txt @@ -1,3 +1,3 @@ -boto3==1.34.100 +boto3==1.34.106 ./lib/dynamo/ ./lib/lambda_logging/ diff --git a/requirements-apps-start-execution-worker.txt b/requirements-apps-start-execution-worker.txt index 8ff916d39..bf742632d 100644 --- a/requirements-apps-start-execution-worker.txt +++ b/requirements-apps-start-execution-worker.txt @@ -1,2 +1,2 @@ -boto3==1.34.100 +boto3==1.34.106 ./lib/lambda_logging/ From 01a028b809775e5f2ce8bc41155b5a5cf6ff2bdf Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Mon, 20 May 2024 10:57:35 -0800 Subject: [PATCH 08/45] first draft of access codes implementation --- .../src/hyp3_api/api-spec/openapi-spec.yml.j2 | 8 +++++ apps/api/src/hyp3_api/handlers.py | 4 ++- apps/main-cf.yml.j2 | 12 +++++++ lib/dynamo/dynamo/exceptions.py | 4 +++ lib/dynamo/dynamo/user.py | 35 ++++++++++++++++--- 5 files changed, 58 insertions(+), 5 deletions(-) diff --git a/apps/api/src/hyp3_api/api-spec/openapi-spec.yml.j2 b/apps/api/src/hyp3_api/api-spec/openapi-spec.yml.j2 index 98d5ea13f..ace466fe1 100644 --- a/apps/api/src/hyp3_api/api-spec/openapi-spec.yml.j2 +++ b/apps/api/src/hyp3_api/api-spec/openapi-spec.yml.j2 @@ -197,6 +197,8 @@ components: properties: use_case: $ref: "#/components/schemas/use_case" + access_code: + $ref: "#/components/schemas/access_code" user: description: Information about a user @@ -315,6 +317,12 @@ components: type: string example: I want to process data. + access_code: + description: Grants automatic user approval while the code remains active. + type: string + format: uuid + example: 27836b79-e5b2-4d8f-932f-659724ea02c3 + user_id: description: Username from Earthdata Login. type: string diff --git a/apps/api/src/hyp3_api/handlers.py b/apps/api/src/hyp3_api/handlers.py index 81c3afd13..393fc4ac6 100644 --- a/apps/api/src/hyp3_api/handlers.py +++ b/apps/api/src/hyp3_api/handlers.py @@ -4,7 +4,7 @@ from flask import abort, jsonify, request import dynamo -from dynamo.exceptions import InsufficientCreditsError, UnexpectedApplicationStatusError +from dynamo.exceptions import AccessCodeError, InsufficientCreditsError, UnexpectedApplicationStatusError from hyp3_api import util from hyp3_api.validation import GranuleValidationError, validate_jobs @@ -65,6 +65,8 @@ def patch_user(body: dict, user: str, edl_access_token: str) -> dict: print(body) try: user_record = dynamo.user.update_user(user, edl_access_token, body) + except AccessCodeError as e: + abort(problem_format(403, str(e))) except UnexpectedApplicationStatusError as e: abort(problem_format(403, str(e))) return _user_response(user_record) diff --git a/apps/main-cf.yml.j2 b/apps/main-cf.yml.j2 index 46a17191b..16bdac36f 100644 --- a/apps/main-cf.yml.j2 +++ b/apps/main-cf.yml.j2 @@ -356,6 +356,18 @@ Resources: - AttributeName: user_id KeyType: HASH + # TODO: do we like this name? + AccessCodesTable: + Type: AWS::DynamoDB::Table + Properties: + BillingMode: PAY_PER_REQUEST + AttributeDefinitions: + - AttributeName: access_code + AttributeType: S + KeySchema: + - AttributeName: access_code + KeyType: HASH + {% if security_environment == 'EDC' %} DisablePrivateDNS: Type: AWS::CloudFormation::Stack diff --git a/lib/dynamo/dynamo/exceptions.py b/lib/dynamo/dynamo/exceptions.py index 29f386ec7..6460d21f8 100644 --- a/lib/dynamo/dynamo/exceptions.py +++ b/lib/dynamo/dynamo/exceptions.py @@ -5,6 +5,10 @@ class DatabaseConditionException(Exception): """Raised when a DynamoDB condition expression check fails.""" +class AccessCodeError(Exception): + """Raised when a user application includes an invalid or expired access code.""" + + class InsufficientCreditsError(Exception): """Raised when trying to submit jobs whose total cost exceeds the user's remaining credits.""" diff --git a/lib/dynamo/dynamo/user.py b/lib/dynamo/dynamo/user.py index bfe5fcfea..4070aec09 100644 --- a/lib/dynamo/dynamo/user.py +++ b/lib/dynamo/dynamo/user.py @@ -2,14 +2,19 @@ from datetime import datetime, timezone from decimal import Decimal from os import environ +from typing import Optional import botocore.exceptions import requests from dynamo.exceptions import ( - ApprovedApplicationError, DatabaseConditionException, InvalidApplicationStatusError, RejectedApplicationError + AccessCodeError, + ApprovedApplicationError, + DatabaseConditionException, + InvalidApplicationStatusError, + RejectedApplicationError, ) -from dynamo.util import DYNAMODB_RESOURCE +from dynamo.util import DYNAMODB_RESOURCE, format_time APPLICATION_NOT_STARTED = 'NOT_STARTED' APPLICATION_PENDING = 'PENDING' @@ -21,19 +26,25 @@ def update_user(user_id: str, edl_access_token: str, body: dict) -> dict: user = get_or_create_user(user_id) application_status = user['application_status'] if application_status in (APPLICATION_NOT_STARTED, APPLICATION_PENDING): + updated_application_status = _get_updated_application_status(body.get('access_code')) edl_profile = _get_edl_profile(user_id, edl_access_token) users_table = DYNAMODB_RESOURCE.Table(environ['USERS_TABLE_NAME']) try: user = users_table.update_item( Key={'user_id': user_id}, - UpdateExpression='SET #edl_profile = :edl_profile, use_case = :use_case, application_status = :pending', + UpdateExpression=( + 'SET #edl_profile = :edl_profile,' + ' use_case = :use_case,' + ' application_status = :updated_application_status' + ), ConditionExpression='application_status IN (:not_started, :pending)', ExpressionAttributeNames={'#edl_profile': '_edl_profile'}, ExpressionAttributeValues={ ':edl_profile': edl_profile, ':use_case': body['use_case'], ':not_started': APPLICATION_NOT_STARTED, - ':pending': APPLICATION_PENDING + ':pending': APPLICATION_PENDING, + ':updated_application_status': updated_application_status, }, ReturnValues='ALL_NEW', )['Attributes'] @@ -49,6 +60,22 @@ def update_user(user_id: str, edl_access_token: str, body: dict) -> dict: raise InvalidApplicationStatusError(user_id, application_status) +def _get_updated_application_status(access_code: Optional[str]) -> str: + if access_code is None: + return APPLICATION_PENDING + + access_codes_table = DYNAMODB_RESOURCE.Table(environ['ACCESS_CODES_TABLE_NAME']) + item = access_codes_table.get_item(Key={'access_code': access_code}).get('Item') + + if item is None: + raise AccessCodeError(f'{access_code} is not a valid access code') + + if format_time(datetime.now(timezone.utc)) >= item['expires']: + raise AccessCodeError(f'Access code {access_code} expired on {item["expires"]}') + + return APPLICATION_APPROVED + + def _get_edl_profile(user_id: str, edl_access_token: str) -> dict: url = f'https://urs.earthdata.nasa.gov/api/users/{user_id}' response = requests.get(url, headers={'Authorization': f'Bearer {edl_access_token}'}) From 9b73490b8a886955256d952d2768699a728d80cd Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Mon, 20 May 2024 11:11:33 -0800 Subject: [PATCH 09/45] pass AccessCodesTable name to API stack, add a TODO --- apps/api/api-cf.yml.j2 | 8 ++++++++ apps/main-cf.yml.j2 | 1 + lib/dynamo/dynamo/user.py | 1 + 3 files changed, 10 insertions(+) diff --git a/apps/api/api-cf.yml.j2 b/apps/api/api-cf.yml.j2 index 975bbf3be..338ff01ec 100644 --- a/apps/api/api-cf.yml.j2 +++ b/apps/api/api-cf.yml.j2 @@ -6,6 +6,9 @@ Parameters: UsersTable: Type: String + AccessCodesTable: + Type: String + AuthPublicKey: Type: String @@ -171,6 +174,10 @@ Resources: - dynamodb:PutItem - dynamodb:UpdateItem Resource: !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${UsersTable}*" + - Effect: Allow + Action: + - dynamodb:GetItem + Resource: !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${AccessCodesTable}*" Lambda: Type: AWS::Lambda::Function @@ -179,6 +186,7 @@ Resources: Variables: JOBS_TABLE_NAME: !Ref JobsTable USERS_TABLE_NAME: !Ref UsersTable + ACCESS_CODES_TABLE_NAME: !Ref AccessCodesTable AUTH_PUBLIC_KEY: !Ref AuthPublicKey AUTH_ALGORITHM: !Ref AuthAlgorithm DEFAULT_CREDITS_PER_USER: !Ref DefaultCreditsPerUser diff --git a/apps/main-cf.yml.j2 b/apps/main-cf.yml.j2 index 16bdac36f..e47dc3f97 100644 --- a/apps/main-cf.yml.j2 +++ b/apps/main-cf.yml.j2 @@ -120,6 +120,7 @@ Resources: Parameters: JobsTable: !Ref JobsTable UsersTable: !Ref UsersTable + AccessCodesTable: !Ref AccessCodesTable AuthPublicKey: !Ref AuthPublicKey AuthAlgorithm: !Ref AuthAlgorithm DefaultCreditsPerUser: !Ref DefaultCreditsPerUser diff --git a/lib/dynamo/dynamo/user.py b/lib/dynamo/dynamo/user.py index 4070aec09..68f98923f 100644 --- a/lib/dynamo/dynamo/user.py +++ b/lib/dynamo/dynamo/user.py @@ -70,6 +70,7 @@ def _get_updated_application_status(access_code: Optional[str]) -> str: if item is None: raise AccessCodeError(f'{access_code} is not a valid access code') + # TODO is this our preferred expiration time format? if format_time(datetime.now(timezone.utc)) >= item['expires']: raise AccessCodeError(f'Access code {access_code} expired on {item["expires"]}') From 80fd79f07808ddaf8b52bc15476ec5565a7aadd1 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Mon, 20 May 2024 11:15:33 -0800 Subject: [PATCH 10/45] add access codes table to cfg.env, add a TODO --- tests/cfg.env | 1 + tests/conftest.py | 2 ++ 2 files changed, 3 insertions(+) diff --git a/tests/cfg.env b/tests/cfg.env index 31bf731a5..a8e781711 100644 --- a/tests/cfg.env +++ b/tests/cfg.env @@ -1,6 +1,7 @@ FLASK_DEBUG=true JOBS_TABLE_NAME=hyp3-db-table-job USERS_TABLE_NAME=hyp3-db-table-user +ACCESS_CODES_TABLE_NAME=hyp3-db-table-access-codes AUTH_PUBLIC_KEY=123456789 AUTH_ALGORITHM=HS256 DEFAULT_CREDITS_PER_USER=25 diff --git a/tests/conftest.py b/tests/conftest.py index 555c08616..f0308e8ee 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -7,6 +7,8 @@ from dynamo.user import APPLICATION_APPROVED +# TODO: mock AccessCodesTable + @pytest.fixture def table_properties(): From c758ea01bf25afff5bb0eeec66b15ed960b80215 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Mon, 20 May 2024 11:19:27 -0800 Subject: [PATCH 11/45] add a TODO --- lib/dynamo/dynamo/user.py | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/dynamo/dynamo/user.py b/lib/dynamo/dynamo/user.py index 68f98923f..6e2c9ed26 100644 --- a/lib/dynamo/dynamo/user.py +++ b/lib/dynamo/dynamo/user.py @@ -26,6 +26,7 @@ def update_user(user_id: str, edl_access_token: str, body: dict) -> dict: user = get_or_create_user(user_id) application_status = user['application_status'] if application_status in (APPLICATION_NOT_STARTED, APPLICATION_PENDING): + # TODO: add access_code to user record updated_application_status = _get_updated_application_status(body.get('access_code')) edl_profile = _get_edl_profile(user_id, edl_access_token) users_table = DYNAMODB_RESOURCE.Table(environ['USERS_TABLE_NAME']) From 7bc7dc956c260548ccc44329744a4a913bfa6453 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 20 May 2024 19:21:47 +0000 Subject: [PATCH 12/45] --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- .github/workflows/deploy-daac.yml | 2 +- .github/workflows/deploy-enterprise-test.yml | 2 +- .github/workflows/deploy-enterprise.yml | 2 +- .github/workflows/deploy-whitelisting-sandbox.yml | 2 +- .github/workflows/static-analysis.yml | 10 +++++----- .github/workflows/tests.yml | 2 +- 6 files changed, 10 insertions(+), 10 deletions(-) diff --git a/.github/workflows/deploy-daac.yml b/.github/workflows/deploy-daac.yml index be75bc6ed..73aaf2ba9 100644 --- a/.github/workflows/deploy-daac.yml +++ b/.github/workflows/deploy-daac.yml @@ -62,7 +62,7 @@ jobs: url: https://${{ matrix.domain }} steps: - - uses: actions/checkout@v4.1.5 + - uses: actions/checkout@v4.1.6 - uses: aws-actions/configure-aws-credentials@v4 with: diff --git a/.github/workflows/deploy-enterprise-test.yml b/.github/workflows/deploy-enterprise-test.yml index e60494da3..1e9bdce3d 100644 --- a/.github/workflows/deploy-enterprise-test.yml +++ b/.github/workflows/deploy-enterprise-test.yml @@ -82,7 +82,7 @@ jobs: url: https://${{ matrix.domain }} steps: - - uses: actions/checkout@v4.1.5 + - uses: actions/checkout@v4.1.6 - uses: aws-actions/configure-aws-credentials@v4 with: diff --git a/.github/workflows/deploy-enterprise.yml b/.github/workflows/deploy-enterprise.yml index 09a58aaf8..d71172d58 100644 --- a/.github/workflows/deploy-enterprise.yml +++ b/.github/workflows/deploy-enterprise.yml @@ -229,7 +229,7 @@ jobs: url: https://${{ matrix.domain }} steps: - - uses: actions/checkout@v4.1.5 + - uses: actions/checkout@v4.1.6 - uses: aws-actions/configure-aws-credentials@v4 with: diff --git a/.github/workflows/deploy-whitelisting-sandbox.yml b/.github/workflows/deploy-whitelisting-sandbox.yml index a9671a51d..7ffe281bf 100644 --- a/.github/workflows/deploy-whitelisting-sandbox.yml +++ b/.github/workflows/deploy-whitelisting-sandbox.yml @@ -41,7 +41,7 @@ jobs: url: https://${{ matrix.domain }} steps: - - uses: actions/checkout@v4.1.5 + - uses: actions/checkout@v4.1.6 - uses: aws-actions/configure-aws-credentials@v4 with: diff --git a/.github/workflows/static-analysis.yml b/.github/workflows/static-analysis.yml index ee35bf19f..c986d1feb 100644 --- a/.github/workflows/static-analysis.yml +++ b/.github/workflows/static-analysis.yml @@ -6,7 +6,7 @@ jobs: flake8: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4.1.5 + - uses: actions/checkout@v4.1.6 - uses: actions/setup-python@v5 with: python-version: 3.9 @@ -23,7 +23,7 @@ jobs: matrix: security_environment: [ASF, EDC, JPL, JPL-public] steps: - - uses: actions/checkout@v4.1.5 + - uses: actions/checkout@v4.1.6 - uses: actions/setup-python@v5 with: python-version: 3.9 @@ -37,7 +37,7 @@ jobs: openapi-spec-validator: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4.1.5 + - uses: actions/checkout@v4.1.6 - uses: actions/setup-python@v5 with: python-version: 3.9 @@ -50,7 +50,7 @@ jobs: statelint: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4.1.5 + - uses: actions/checkout@v4.1.6 - uses: ruby/setup-ruby@v1 with: ruby-version: 2.7 @@ -70,7 +70,7 @@ jobs: snyk: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4.1.5 + - uses: actions/checkout@v4.1.6 - uses: snyk/actions/setup@0.4.0 - uses: actions/setup-python@v5 with: diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index a48a05477..dd480b0a9 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -7,7 +7,7 @@ jobs: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4.1.5 + - uses: actions/checkout@v4.1.6 - uses: actions/setup-python@v5 with: From 79c6308a08411f16570194d22df9e1ad3455b8ee Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 20 May 2024 19:34:00 +0000 Subject: [PATCH 13/45] --- updated-dependencies: - dependency-name: boto3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- requirements-all.txt | 2 +- requirements-apps-disable-private-dns.txt | 2 +- requirements-apps-start-execution-manager.txt | 2 +- requirements-apps-start-execution-worker.txt | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/requirements-all.txt b/requirements-all.txt index 343c4fa19..d9421f47f 100644 --- a/requirements-all.txt +++ b/requirements-all.txt @@ -5,7 +5,7 @@ -r requirements-apps-start-execution-worker.txt -r requirements-apps-disable-private-dns.txt -r requirements-apps-update-db.txt -boto3==1.34.106 +boto3==1.34.109 jinja2==3.1.4 moto[dynamodb]==5.0.7 pytest==8.2.0 diff --git a/requirements-apps-disable-private-dns.txt b/requirements-apps-disable-private-dns.txt index 334eadc0c..e69671eed 100644 --- a/requirements-apps-disable-private-dns.txt +++ b/requirements-apps-disable-private-dns.txt @@ -1 +1 @@ -boto3==1.34.106 +boto3==1.34.109 diff --git a/requirements-apps-start-execution-manager.txt b/requirements-apps-start-execution-manager.txt index 39a9ac915..ab2543465 100644 --- a/requirements-apps-start-execution-manager.txt +++ b/requirements-apps-start-execution-manager.txt @@ -1,3 +1,3 @@ -boto3==1.34.106 +boto3==1.34.109 ./lib/dynamo/ ./lib/lambda_logging/ diff --git a/requirements-apps-start-execution-worker.txt b/requirements-apps-start-execution-worker.txt index bf742632d..639ea1469 100644 --- a/requirements-apps-start-execution-worker.txt +++ b/requirements-apps-start-execution-worker.txt @@ -1,2 +1,2 @@ -boto3==1.34.106 +boto3==1.34.109 ./lib/lambda_logging/ From 70bb9c48f9b30cd14491abff65ba5c3afc091847 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 20 May 2024 19:34:11 +0000 Subject: [PATCH 14/45] --- updated-dependencies: - dependency-name: cfn-lint dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- requirements-all.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements-all.txt b/requirements-all.txt index 343c4fa19..3c5fd1029 100644 --- a/requirements-all.txt +++ b/requirements-all.txt @@ -17,4 +17,4 @@ flake8-blind-except==0.2.1 flake8-builtins==2.5.0 setuptools==69.5.1 openapi-spec-validator==0.7.1 -cfn-lint==0.87.2 +cfn-lint==0.87.3 From 7c006d2fcaf83298435111c439fbe51b320349ff Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 20 May 2024 19:34:15 +0000 Subject: [PATCH 15/45] --- updated-dependencies: - dependency-name: serverless-wsgi dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- requirements-apps-api.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements-apps-api.txt b/requirements-apps-api.txt index b67720ab1..5899d889b 100644 --- a/requirements-apps-api.txt +++ b/requirements-apps-api.txt @@ -5,7 +5,7 @@ openapi-core==0.19.1 prance==23.6.21.0 PyJWT==2.8.0 requests==2.31.0 -serverless_wsgi==3.0.3 +serverless_wsgi==3.0.4 shapely==2.0.4 strict-rfc3339==0.7 ./lib/dynamo/ From e3a75ba2648d43fce9cc8a72c54f11604f190d97 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Mon, 20 May 2024 14:42:39 -0800 Subject: [PATCH 16/45] add access_code to user record --- lib/dynamo/dynamo/user.py | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/lib/dynamo/dynamo/user.py b/lib/dynamo/dynamo/user.py index 6e2c9ed26..2c44cc2a1 100644 --- a/lib/dynamo/dynamo/user.py +++ b/lib/dynamo/dynamo/user.py @@ -2,7 +2,6 @@ from datetime import datetime, timezone from decimal import Decimal from os import environ -from typing import Optional import botocore.exceptions import requests @@ -26,8 +25,16 @@ def update_user(user_id: str, edl_access_token: str, body: dict) -> dict: user = get_or_create_user(user_id) application_status = user['application_status'] if application_status in (APPLICATION_NOT_STARTED, APPLICATION_PENDING): - # TODO: add access_code to user record - updated_application_status = _get_updated_application_status(body.get('access_code')) + access_code = body.get('access_code') + if access_code: + _validate_access_code(access_code) + updated_application_status = APPLICATION_APPROVED + access_code_expression = ', access_code = :access_code' + access_code_value = {':access_code': access_code} + else: + updated_application_status = APPLICATION_PENDING + access_code_expression = '' + access_code_value = {} edl_profile = _get_edl_profile(user_id, edl_access_token) users_table = DYNAMODB_RESOURCE.Table(environ['USERS_TABLE_NAME']) try: @@ -37,6 +44,7 @@ def update_user(user_id: str, edl_access_token: str, body: dict) -> dict: 'SET #edl_profile = :edl_profile,' ' use_case = :use_case,' ' application_status = :updated_application_status' + f'{access_code_expression}' ), ConditionExpression='application_status IN (:not_started, :pending)', ExpressionAttributeNames={'#edl_profile': '_edl_profile'}, @@ -46,6 +54,7 @@ def update_user(user_id: str, edl_access_token: str, body: dict) -> dict: ':not_started': APPLICATION_NOT_STARTED, ':pending': APPLICATION_PENDING, ':updated_application_status': updated_application_status, + **access_code_value }, ReturnValues='ALL_NEW', )['Attributes'] @@ -61,10 +70,7 @@ def update_user(user_id: str, edl_access_token: str, body: dict) -> dict: raise InvalidApplicationStatusError(user_id, application_status) -def _get_updated_application_status(access_code: Optional[str]) -> str: - if access_code is None: - return APPLICATION_PENDING - +def _validate_access_code(access_code: str) -> None: access_codes_table = DYNAMODB_RESOURCE.Table(environ['ACCESS_CODES_TABLE_NAME']) item = access_codes_table.get_item(Key={'access_code': access_code}).get('Item') @@ -75,8 +81,6 @@ def _get_updated_application_status(access_code: Optional[str]) -> str: if format_time(datetime.now(timezone.utc)) >= item['expires']: raise AccessCodeError(f'Access code {access_code} expired on {item["expires"]}') - return APPLICATION_APPROVED - def _get_edl_profile(user_id: str, edl_access_token: str) -> dict: url = f'https://urs.earthdata.nasa.gov/api/users/{user_id}' From 28680406764b867203f6083daad71c4896f80535 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 21 May 2024 17:10:11 +0000 Subject: [PATCH 17/45] --- updated-dependencies: - dependency-name: requests dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- requirements-apps-api.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements-apps-api.txt b/requirements-apps-api.txt index 5899d889b..750fcfffb 100644 --- a/requirements-apps-api.txt +++ b/requirements-apps-api.txt @@ -4,7 +4,7 @@ jsonschema==4.22.0 openapi-core==0.19.1 prance==23.6.21.0 PyJWT==2.8.0 -requests==2.31.0 +requests==2.32.1 serverless_wsgi==3.0.4 shapely==2.0.4 strict-rfc3339==0.7 From 2cd6619306d2ffd91592a9f25f38b6cb73fffbb1 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 21 May 2024 17:12:42 +0000 Subject: [PATCH 18/45] --- updated-dependencies: - dependency-name: pytest dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- requirements-all.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements-all.txt b/requirements-all.txt index 55add1717..3bd625237 100644 --- a/requirements-all.txt +++ b/requirements-all.txt @@ -8,7 +8,7 @@ boto3==1.34.109 jinja2==3.1.4 moto[dynamodb]==5.0.7 -pytest==8.2.0 +pytest==8.2.1 PyYAML==6.0.1 responses==0.25.0 flake8==7.0.0 From 66b53285c5aa5b79b370f58e38ecda35feb94999 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Tue, 21 May 2024 09:36:43 -0800 Subject: [PATCH 19/45] remove a TODO --- apps/main-cf.yml.j2 | 1 - 1 file changed, 1 deletion(-) diff --git a/apps/main-cf.yml.j2 b/apps/main-cf.yml.j2 index e47dc3f97..bc5ad6135 100644 --- a/apps/main-cf.yml.j2 +++ b/apps/main-cf.yml.j2 @@ -357,7 +357,6 @@ Resources: - AttributeName: user_id KeyType: HASH - # TODO: do we like this name? AccessCodesTable: Type: AWS::DynamoDB::Table Properties: From f366a10d0627ee42b1a47d6851e8ce510089fc57 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Tue, 21 May 2024 09:40:00 -0800 Subject: [PATCH 20/45] mock access codes table --- tests/conftest.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/tests/conftest.py b/tests/conftest.py index f0308e8ee..be67e315b 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -7,14 +7,13 @@ from dynamo.user import APPLICATION_APPROVED -# TODO: mock AccessCodesTable - @pytest.fixture def table_properties(): class TableProperties: jobs_table = get_table_properties_from_template('JobsTable') users_table = get_table_properties_from_template('UsersTable') + access_codes_table = get_table_properties_from_template('AccessCodesTable') return TableProperties() @@ -42,6 +41,10 @@ class Tables: TableName=environ['USERS_TABLE_NAME'], **table_properties.users_table, ) + access_codes_table = DYNAMODB_RESOURCE.create_table( + TableName=environ['ACCESS_CODES_TABLE_NAME'], + **table_properties.access_codes_table, + ) tables = Tables() yield tables From 31580c0ba4bf890f0d23bfbdb02673f98aeb8107 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Tue, 21 May 2024 12:03:22 -0800 Subject: [PATCH 21/45] factor out a dynamo.util.current_time function --- lib/dynamo/dynamo/jobs.py | 4 ++-- lib/dynamo/dynamo/user.py | 5 ++--- lib/dynamo/dynamo/util.py | 6 +++++- tests/test_api/test_get_user.py | 5 ++--- tests/test_api/test_submit_job.py | 5 ++--- tests/test_dynamo/test_jobs.py | 4 ++-- 6 files changed, 15 insertions(+), 14 deletions(-) diff --git a/lib/dynamo/dynamo/jobs.py b/lib/dynamo/dynamo/jobs.py index f95e26aca..664b32b64 100644 --- a/lib/dynamo/dynamo/jobs.py +++ b/lib/dynamo/dynamo/jobs.py @@ -17,7 +17,7 @@ RejectedApplicationError, ) from dynamo.user import APPLICATION_APPROVED, APPLICATION_NOT_STARTED, APPLICATION_PENDING, APPLICATION_REJECTED -from dynamo.util import DYNAMODB_RESOURCE, convert_floats_to_decimals, format_time, get_request_time_expression +from dynamo.util import DYNAMODB_RESOURCE, convert_floats_to_decimals, current_time, get_request_time_expression costs_file = Path(__file__).parent / 'costs.json' COSTS = convert_floats_to_decimals(json.loads(costs_file.read_text())) @@ -32,7 +32,7 @@ def put_jobs(user_id: str, jobs: List[dict], dry_run=False) -> List[dict]: table = DYNAMODB_RESOURCE.Table(environ['JOBS_TABLE_NAME']) - request_time = format_time(datetime.now(timezone.utc)) + request_time = current_time() user_record = dynamo.user.get_or_create_user(user_id) diff --git a/lib/dynamo/dynamo/user.py b/lib/dynamo/dynamo/user.py index 2c44cc2a1..26bd7499d 100644 --- a/lib/dynamo/dynamo/user.py +++ b/lib/dynamo/dynamo/user.py @@ -13,7 +13,7 @@ InvalidApplicationStatusError, RejectedApplicationError, ) -from dynamo.util import DYNAMODB_RESOURCE, format_time +from dynamo.util import DYNAMODB_RESOURCE, current_time APPLICATION_NOT_STARTED = 'NOT_STARTED' APPLICATION_PENDING = 'PENDING' @@ -77,8 +77,7 @@ def _validate_access_code(access_code: str) -> None: if item is None: raise AccessCodeError(f'{access_code} is not a valid access code') - # TODO is this our preferred expiration time format? - if format_time(datetime.now(timezone.utc)) >= item['expires']: + if current_time() >= item['expires']: raise AccessCodeError(f'Access code {access_code} expired on {item["expires"]}') diff --git a/lib/dynamo/dynamo/util.py b/lib/dynamo/dynamo/util.py index 4d2561a2b..f7d09650e 100644 --- a/lib/dynamo/dynamo/util.py +++ b/lib/dynamo/dynamo/util.py @@ -21,13 +21,17 @@ def get_request_time_expression(start, end): return key.lte(formatted_end) -def format_time(time: datetime): +def format_time(time: datetime) -> str: if time.tzinfo is None: raise ValueError(f'missing tzinfo for datetime {time}') utc_time = time.astimezone(timezone.utc) return utc_time.isoformat(timespec='seconds') +def current_time() -> str: + return format_time(datetime.now(timezone.utc)) + + def convert_floats_to_decimals(element): if type(element) is float: return Decimal(str(element)) diff --git a/tests/test_api/test_get_user.py b/tests/test_api/test_get_user.py index fe21ea5ed..f08d7d817 100644 --- a/tests/test_api/test_get_user.py +++ b/tests/test_api/test_get_user.py @@ -1,10 +1,9 @@ -from datetime import datetime, timezone from http import HTTPStatus from test_api.conftest import USER_URI, login, make_db_record from dynamo.user import APPLICATION_APPROVED, APPLICATION_NOT_STARTED, APPLICATION_REJECTED -from dynamo.util import format_time +from dynamo.util import current_time def test_get_new_user(client, tables, monkeypatch): @@ -46,7 +45,7 @@ def test_get_user_with_jobs(client, tables): } tables.users_table.put_item(Item=user) - request_time = format_time(datetime.now(timezone.utc)) + request_time = current_time() items = [ make_db_record('job1', user_id=user_id, request_time=request_time, status_code='PENDING', name='job1'), make_db_record('job2', user_id=user_id, request_time=request_time, status_code='RUNNING', name='job1'), diff --git a/tests/test_api/test_submit_job.py b/tests/test_api/test_submit_job.py index 962b544f6..3faec48d5 100644 --- a/tests/test_api/test_submit_job.py +++ b/tests/test_api/test_submit_job.py @@ -1,11 +1,10 @@ -from datetime import datetime, timezone from decimal import Decimal from http import HTTPStatus from test_api.conftest import login, make_job, setup_requests_mock, submit_batch from dynamo.user import APPLICATION_PENDING -from dynamo.util import format_time +from dynamo.util import current_time def test_submit_one_job(client, approved_user): @@ -17,7 +16,7 @@ def test_submit_one_job(client, approved_user): jobs = response.json['jobs'] assert len(jobs) == 1 assert jobs[0]['status_code'] == 'PENDING' - assert jobs[0]['request_time'] <= format_time(datetime.now(timezone.utc)) + assert jobs[0]['request_time'] <= current_time() assert jobs[0]['user_id'] == approved_user diff --git a/tests/test_dynamo/test_jobs.py b/tests/test_dynamo/test_jobs.py index 2b5d1b748..aa5539017 100644 --- a/tests/test_dynamo/test_jobs.py +++ b/tests/test_dynamo/test_jobs.py @@ -1,5 +1,4 @@ import unittest.mock -from datetime import datetime, timezone from decimal import Decimal import pytest @@ -14,6 +13,7 @@ RejectedApplicationError, ) from dynamo.user import APPLICATION_APPROVED +from dynamo.util import current_time def test_query_jobs_by_user(tables): @@ -281,7 +281,7 @@ def test_put_jobs(tables, monkeypatch, approved_user): assert set(job.keys()) == { 'name', 'job_id', 'user_id', 'status_code', 'execution_started', 'request_time', 'priority', 'credit_cost' } - assert job['request_time'] <= dynamo.util.format_time(datetime.now(timezone.utc)) + assert job['request_time'] <= current_time() assert job['user_id'] == approved_user assert job['status_code'] == 'PENDING' assert job['execution_started'] is False From 9edf91c0dded200f7d2d7f78b4ae84a51a39c4a6 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Tue, 21 May 2024 12:47:47 -0800 Subject: [PATCH 22/45] add test_update_user_access_code --- lib/dynamo/dynamo/user.py | 9 +++-- tests/test_dynamo/test_user.py | 73 ++++++++++++++++++++++++++++++++++ 2 files changed, 79 insertions(+), 3 deletions(-) diff --git a/lib/dynamo/dynamo/user.py b/lib/dynamo/dynamo/user.py index 26bd7499d..041c086fa 100644 --- a/lib/dynamo/dynamo/user.py +++ b/lib/dynamo/dynamo/user.py @@ -6,6 +6,7 @@ import botocore.exceptions import requests +import dynamo.util from dynamo.exceptions import ( AccessCodeError, ApprovedApplicationError, @@ -13,7 +14,7 @@ InvalidApplicationStatusError, RejectedApplicationError, ) -from dynamo.util import DYNAMODB_RESOURCE, current_time +from dynamo.util import DYNAMODB_RESOURCE APPLICATION_NOT_STARTED = 'NOT_STARTED' APPLICATION_PENDING = 'PENDING' @@ -62,7 +63,9 @@ def update_user(user_id: str, edl_access_token: str, body: dict) -> dict: if e.response['Error']['Code'] == 'ConditionalCheckFailedException': raise DatabaseConditionException(f'Failed to update record for user {user_id}') raise - return user + # TODO: better way to reset credits for Approved users? + #return user + return get_or_create_user(user_id) if application_status == APPLICATION_REJECTED: raise RejectedApplicationError(user_id) if application_status == APPLICATION_APPROVED: @@ -77,7 +80,7 @@ def _validate_access_code(access_code: str) -> None: if item is None: raise AccessCodeError(f'{access_code} is not a valid access code') - if current_time() >= item['expires']: + if dynamo.util.current_time() >= item['expires']: raise AccessCodeError(f'Access code {access_code} expired on {item["expires"]}') diff --git a/tests/test_dynamo/test_user.py b/tests/test_dynamo/test_user.py index ee12c0299..558449336 100644 --- a/tests/test_dynamo/test_user.py +++ b/tests/test_dynamo/test_user.py @@ -6,6 +6,7 @@ import dynamo.user from dynamo.exceptions import ( + AccessCodeError, ApprovedApplicationError, DatabaseConditionException, InvalidApplicationStatusError, @@ -188,6 +189,78 @@ def test_update_user_failed_application_status(tables): }] +def test_update_user_access_code(tables): + tables.access_codes_table.put_item(Item={'access_code': '123', 'expires': '2024-05-21T20:01:04+00:00'}) + + with pytest.raises(AccessCodeError, match=r'.*not a valid access code.*'): + dynamo.user.update_user( + 'foo', + 'test-edl-access-token', + {'use_case': 'I want data.', 'access_code': '456'} + ) + + assert tables.users_table.scan()['Items'] == [ + {'user_id': 'foo', 'remaining_credits': Decimal(0), 'application_status': APPLICATION_NOT_STARTED} + ] + + with unittest.mock.patch('dynamo.util.current_time') as mock_current_time: + mock_current_time.return_value = '2024-05-21T20:01:05+00:00' + with pytest.raises(AccessCodeError, match=r'.*expired.*'): + dynamo.user.update_user( + 'foo', + 'test-edl-access-token', + {'use_case': 'I want data.', 'access_code': '123'} + ) + mock_current_time.assert_called_once_with() + + assert tables.users_table.scan()['Items'] == [ + {'user_id': 'foo', 'remaining_credits': Decimal(0), 'application_status': APPLICATION_NOT_STARTED} + ] + + with unittest.mock.patch('dynamo.util.current_time') as mock_current_time: + mock_current_time.return_value = '2024-05-21T20:01:04+00:00' + with pytest.raises(AccessCodeError, match=r'.*expired.*'): + dynamo.user.update_user( + 'foo', + 'test-edl-access-token', + {'use_case': 'I want data.', 'access_code': '123'} + ) + mock_current_time.assert_called_once_with() + + assert tables.users_table.scan()['Items'] == [ + {'user_id': 'foo', 'remaining_credits': Decimal(0), 'application_status': APPLICATION_NOT_STARTED} + ] + + with unittest.mock.patch('dynamo.util.current_time') as mock_current_time, \ + unittest.mock.patch('dynamo.user._get_current_month') as mock_get_current_month, \ + unittest.mock.patch('dynamo.user._get_edl_profile') as mock_get_edl_profile: + + mock_current_time.return_value = '2024-05-21T20:01:03+00:00' + mock_get_current_month.return_value = '2024-05' + mock_get_edl_profile.return_value = {'key': 'value'} + + user = dynamo.user.update_user( + 'foo', + 'test-edl-access-token', + {'use_case': 'I want data.', 'access_code': '123'} + ) + + mock_current_time.assert_called_once_with() + assert mock_get_current_month.mock_calls == [unittest.mock.call()] * 2 + mock_get_edl_profile.assert_called_once_with('foo', 'test-edl-access-token') + + assert user == { + 'user_id': 'foo', + 'remaining_credits': Decimal(25), + '_month_of_last_credit_reset': '2024-05', + 'application_status': APPLICATION_APPROVED, + '_edl_profile': {'key': 'value'}, + 'use_case': 'I want data.', + 'access_code': '123', + } + assert tables.users_table.scan()['Items'] == [user] + + def test_get_or_create_user_existing_user(tables): tables.users_table.put_item( Item={ From c49a2177bd46d5f526b402bf1404aca7ce42a6e2 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Tue, 21 May 2024 12:52:09 -0800 Subject: [PATCH 23/45] add a todo --- tests/test_dynamo/test_user.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/test_dynamo/test_user.py b/tests/test_dynamo/test_user.py index 558449336..7eb82a063 100644 --- a/tests/test_dynamo/test_user.py +++ b/tests/test_dynamo/test_user.py @@ -189,6 +189,7 @@ def test_update_user_failed_application_status(tables): }] +# TODO: split into multiple tests def test_update_user_access_code(tables): tables.access_codes_table.put_item(Item={'access_code': '123', 'expires': '2024-05-21T20:01:04+00:00'}) From e99a071329d44d32597281462501743c1759de09 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Wed, 22 May 2024 11:01:58 -0800 Subject: [PATCH 24/45] split into multiple tests --- tests/test_dynamo/test_user.py | 65 +++++++++++++++++++--------------- 1 file changed, 36 insertions(+), 29 deletions(-) diff --git a/tests/test_dynamo/test_user.py b/tests/test_dynamo/test_user.py index 7eb82a063..bab627f9f 100644 --- a/tests/test_dynamo/test_user.py +++ b/tests/test_dynamo/test_user.py @@ -189,20 +189,41 @@ def test_update_user_failed_application_status(tables): }] -# TODO: split into multiple tests def test_update_user_access_code(tables): tables.access_codes_table.put_item(Item={'access_code': '123', 'expires': '2024-05-21T20:01:04+00:00'}) - with pytest.raises(AccessCodeError, match=r'.*not a valid access code.*'): - dynamo.user.update_user( + with unittest.mock.patch('dynamo.util.current_time') as mock_current_time, \ + unittest.mock.patch('dynamo.user._get_current_month') as mock_get_current_month, \ + unittest.mock.patch('dynamo.user._get_edl_profile') as mock_get_edl_profile: + + mock_current_time.return_value = '2024-05-21T20:01:03+00:00' + mock_get_current_month.return_value = '2024-05' + mock_get_edl_profile.return_value = {'key': 'value'} + + user = dynamo.user.update_user( 'foo', 'test-edl-access-token', - {'use_case': 'I want data.', 'access_code': '456'} + {'use_case': 'I want data.', 'access_code': '123'} ) - assert tables.users_table.scan()['Items'] == [ - {'user_id': 'foo', 'remaining_credits': Decimal(0), 'application_status': APPLICATION_NOT_STARTED} - ] + mock_current_time.assert_called_once_with() + assert mock_get_current_month.mock_calls == [unittest.mock.call()] * 2 + mock_get_edl_profile.assert_called_once_with('foo', 'test-edl-access-token') + + assert user == { + 'user_id': 'foo', + 'remaining_credits': Decimal(25), + '_month_of_last_credit_reset': '2024-05', + 'application_status': APPLICATION_APPROVED, + '_edl_profile': {'key': 'value'}, + 'use_case': 'I want data.', + 'access_code': '123', + } + assert tables.users_table.scan()['Items'] == [user] + + +def test_update_user_access_code_expired(tables): + tables.access_codes_table.put_item(Item={'access_code': '123', 'expires': '2024-05-21T20:01:04+00:00'}) with unittest.mock.patch('dynamo.util.current_time') as mock_current_time: mock_current_time.return_value = '2024-05-21T20:01:05+00:00' @@ -232,34 +253,20 @@ def test_update_user_access_code(tables): {'user_id': 'foo', 'remaining_credits': Decimal(0), 'application_status': APPLICATION_NOT_STARTED} ] - with unittest.mock.patch('dynamo.util.current_time') as mock_current_time, \ - unittest.mock.patch('dynamo.user._get_current_month') as mock_get_current_month, \ - unittest.mock.patch('dynamo.user._get_edl_profile') as mock_get_edl_profile: - mock_current_time.return_value = '2024-05-21T20:01:03+00:00' - mock_get_current_month.return_value = '2024-05' - mock_get_edl_profile.return_value = {'key': 'value'} +def test_update_user_access_code_invalid(tables): + tables.access_codes_table.put_item(Item={'access_code': '123', 'expires': ''}) - user = dynamo.user.update_user( + with pytest.raises(AccessCodeError, match=r'.*not a valid access code.*'): + dynamo.user.update_user( 'foo', 'test-edl-access-token', - {'use_case': 'I want data.', 'access_code': '123'} + {'use_case': 'I want data.', 'access_code': '456'} ) - mock_current_time.assert_called_once_with() - assert mock_get_current_month.mock_calls == [unittest.mock.call()] * 2 - mock_get_edl_profile.assert_called_once_with('foo', 'test-edl-access-token') - - assert user == { - 'user_id': 'foo', - 'remaining_credits': Decimal(25), - '_month_of_last_credit_reset': '2024-05', - 'application_status': APPLICATION_APPROVED, - '_edl_profile': {'key': 'value'}, - 'use_case': 'I want data.', - 'access_code': '123', - } - assert tables.users_table.scan()['Items'] == [user] + assert tables.users_table.scan()['Items'] == [ + {'user_id': 'foo', 'remaining_credits': Decimal(0), 'application_status': APPLICATION_NOT_STARTED} + ] def test_get_or_create_user_existing_user(tables): From 86934e5ec92d6414ce2cfdf55fc82a0298ec014d Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Wed, 22 May 2024 11:18:08 -0800 Subject: [PATCH 25/45] add test_patch_user_access_code --- tests/test_api/test_patch_user.py | 33 ++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/tests/test_api/test_patch_user.py b/tests/test_api/test_patch_user.py index 90d1c8d2d..3206369ed 100644 --- a/tests/test_api/test_patch_user.py +++ b/tests/test_api/test_patch_user.py @@ -4,7 +4,7 @@ from test_api.conftest import DEFAULT_ACCESS_TOKEN, USER_URI, login -from dynamo.user import APPLICATION_PENDING, APPLICATION_REJECTED +from dynamo.user import APPLICATION_APPROVED, APPLICATION_PENDING, APPLICATION_REJECTED def test_patch_new_user(client, tables): @@ -66,3 +66,34 @@ def test_patch_rejected_user(client, tables): assert response.status_code == HTTPStatus.FORBIDDEN assert 'has been rejected' in response.json['detail'] + + +def test_patch_user_access_code(client, tables): + tables.access_codes_table.put_item( + Item={'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3', 'expires': '2024-05-21T20:01:04+00:00'} + ) + login(client, 'foo') + + with unittest.mock.patch('dynamo.util.current_time') as mock_current_time, \ + unittest.mock.patch('dynamo.user._get_edl_profile') as mock_get_edl_profile: + + mock_current_time.return_value = '2024-05-21T20:01:03+00:00' + mock_get_edl_profile.return_value = {} + + response = client.patch( + USER_URI, + json={'use_case': 'I want data.', 'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3'} + ) + + mock_current_time.assert_called_once_with() + mock_get_edl_profile.assert_called_once_with('foo', DEFAULT_ACCESS_TOKEN) + + assert response.status_code == HTTPStatus.OK + assert response.json == { + 'user_id': 'foo', + 'application_status': APPLICATION_APPROVED, + 'remaining_credits': Decimal(25), + 'job_names': [], + 'use_case': 'I want data.', + 'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3', + } From 6fd2570873ebecb2222d4606c27e86daac55c5bc Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Wed, 22 May 2024 11:22:09 -0800 Subject: [PATCH 26/45] add test_patch_user_access_code_expired --- tests/test_api/test_patch_user.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/tests/test_api/test_patch_user.py b/tests/test_api/test_patch_user.py index 3206369ed..7391eeb23 100644 --- a/tests/test_api/test_patch_user.py +++ b/tests/test_api/test_patch_user.py @@ -97,3 +97,21 @@ def test_patch_user_access_code(client, tables): 'use_case': 'I want data.', 'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3', } + + +def test_patch_user_access_code_expired(client, tables): + tables.access_codes_table.put_item( + Item={'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3', 'expires': '2024-05-21T20:01:04+00:00'} + ) + login(client, 'foo') + + with unittest.mock.patch('dynamo.util.current_time') as mock_current_time: + mock_current_time.return_value = '2024-05-21T20:01:04+00:00' + response = client.patch( + USER_URI, + json={'use_case': 'I want data.', 'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3'} + ) + mock_current_time.assert_called_once_with() + + assert response.status_code == HTTPStatus.FORBIDDEN + assert 'expired' in response.json['detail'] From 30909b42a52db74b32ab8c3dab5d3612beed9018 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Wed, 22 May 2024 11:25:30 -0800 Subject: [PATCH 27/45] add test_patch_user_access_code_invalid --- tests/test_api/test_patch_user.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/tests/test_api/test_patch_user.py b/tests/test_api/test_patch_user.py index 7391eeb23..92d3c2e64 100644 --- a/tests/test_api/test_patch_user.py +++ b/tests/test_api/test_patch_user.py @@ -115,3 +115,18 @@ def test_patch_user_access_code_expired(client, tables): assert response.status_code == HTTPStatus.FORBIDDEN assert 'expired' in response.json['detail'] + + +def test_patch_user_access_code_invalid(client, tables): + tables.access_codes_table.put_item( + Item={'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3', 'expires': ''} + ) + login(client, 'foo') + + response = client.patch( + USER_URI, + json={'use_case': 'I want data.', 'access_code': '580ef99b-0e16-46b6-8902-c3586f2a8065'} + ) + + assert response.status_code == HTTPStatus.FORBIDDEN + assert 'not a valid access code' in response.json['detail'] From 4f6b71e69fc150348f027ffc32fd61de8bdccfae Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Wed, 22 May 2024 13:02:55 -0800 Subject: [PATCH 28/45] remove default_credits parameter from _reset_credits_if_needed --- lib/dynamo/dynamo/user.py | 6 ++---- tests/test_dynamo/test_user.py | 12 ------------ 2 files changed, 2 insertions(+), 16 deletions(-) diff --git a/lib/dynamo/dynamo/user.py b/lib/dynamo/dynamo/user.py index 041c086fa..bfdba15f4 100644 --- a/lib/dynamo/dynamo/user.py +++ b/lib/dynamo/dynamo/user.py @@ -93,7 +93,6 @@ def _get_edl_profile(user_id: str, edl_access_token: str) -> dict: def get_or_create_user(user_id: str) -> dict: current_month = _get_current_month() - default_credits = Decimal(os.environ['DEFAULT_CREDITS_PER_USER']) users_table = DYNAMODB_RESOURCE.Table(environ['USERS_TABLE_NAME']) user = users_table.get_item(Key={'user_id': user_id}).get('Item') @@ -103,7 +102,6 @@ def get_or_create_user(user_id: str) -> dict: return _reset_credits_if_needed( user=user, - default_credits=default_credits, current_month=current_month, users_table=users_table, ) @@ -128,7 +126,7 @@ def _create_user(user_id: str, users_table) -> dict: return user -def _reset_credits_if_needed(user: dict, default_credits: Decimal, current_month: str, users_table) -> dict: +def _reset_credits_if_needed(user: dict, current_month: str, users_table) -> dict: if ( user['application_status'] == APPLICATION_APPROVED and user.get('_month_of_last_credit_reset', '0') < current_month # noqa: W503 @@ -147,7 +145,7 @@ def _reset_credits_if_needed(user: dict, default_credits: Decimal, current_month ExpressionAttributeNames={'#month_of_last_credit_reset': '_month_of_last_credit_reset'}, ExpressionAttributeValues={ ':approved': APPLICATION_APPROVED, - ':credits': user.get('credits_per_month', default_credits), + ':credits': user.get('credits_per_month', Decimal(os.environ['DEFAULT_CREDITS_PER_USER'])), ':current_month': current_month, ':number': 'N', }, diff --git a/tests/test_dynamo/test_user.py b/tests/test_dynamo/test_user.py index bab627f9f..52a057817 100644 --- a/tests/test_dynamo/test_user.py +++ b/tests/test_dynamo/test_user.py @@ -339,7 +339,6 @@ def test_reset_credits(tables): user = dynamo.user._reset_credits_if_needed( user=original_user_record, - default_credits=Decimal(25), current_month='2024-02', users_table=tables.users_table, ) @@ -364,7 +363,6 @@ def test_reset_credits_month_exists(tables): user = dynamo.user._reset_credits_if_needed( user=original_user_record, - default_credits=Decimal(25), current_month='2024-02', users_table=tables.users_table, ) @@ -389,7 +387,6 @@ def test_reset_credits_override(tables): user = dynamo.user._reset_credits_if_needed( user=original_user_record, - default_credits=Decimal(25), current_month='2024-02', users_table=tables.users_table, ) @@ -415,7 +412,6 @@ def test_reset_credits_same_month(tables): user = dynamo.user._reset_credits_if_needed( user=original_user_record, - default_credits=Decimal(25), current_month='2024-02', users_table=tables.users_table, ) @@ -439,7 +435,6 @@ def test_reset_credits_infinite_credits(tables): user = dynamo.user._reset_credits_if_needed( user=original_user_record, - default_credits=Decimal(25), current_month='2024-02', users_table=tables.users_table, ) @@ -464,7 +459,6 @@ def test_reset_credits_to_zero(tables): user = dynamo.user._reset_credits_if_needed( user=original_user_record, - default_credits=Decimal(25), current_month='2024-02', users_table=tables.users_table, ) @@ -488,7 +482,6 @@ def test_reset_credits_already_at_zero(tables): user = dynamo.user._reset_credits_if_needed( user=original_user_record, - default_credits=Decimal(25), current_month='2024-02', users_table=tables.users_table, ) @@ -518,7 +511,6 @@ def test_reset_credits_failed_not_approved(tables): 'remaining_credits': Decimal(10), 'application_status': APPLICATION_APPROVED, }, - default_credits=Decimal(25), current_month='2024-02', users_table=tables.users_table, ) @@ -548,7 +540,6 @@ def test_reset_credits_failed_same_month(tables): '_month_of_last_credit_reset': '2024-01', 'application_status': APPLICATION_APPROVED, }, - default_credits=Decimal(25), current_month='2024-02', users_table=tables.users_table, ) @@ -577,7 +568,6 @@ def test_reset_credits_failed_infinite_credits(tables): 'remaining_credits': Decimal(10), 'application_status': APPLICATION_APPROVED, }, - default_credits=Decimal(25), current_month='2024-02', users_table=tables.users_table, ) @@ -607,7 +597,6 @@ def test_reset_credits_failed_approved(tables): '_month_of_last_credit_reset': '2024-02', 'application_status': 'bar', }, - default_credits=Decimal(25), current_month='2024-02', users_table=tables.users_table, ) @@ -638,7 +627,6 @@ def test_reset_credits_failed_zero_credits(tables): '_month_of_last_credit_reset': '2024-02', 'application_status': 'bar', }, - default_credits=Decimal(25), current_month='2024-02', users_table=tables.users_table, ) From 31231e62e9f25206cbe1dcfadafbcda4d95cd51b Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Wed, 22 May 2024 13:13:48 -0800 Subject: [PATCH 29/45] refactor update_user to call _reset_credits_if_needed --- lib/dynamo/dynamo/user.py | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/lib/dynamo/dynamo/user.py b/lib/dynamo/dynamo/user.py index bfdba15f4..4925acc43 100644 --- a/lib/dynamo/dynamo/user.py +++ b/lib/dynamo/dynamo/user.py @@ -63,9 +63,7 @@ def update_user(user_id: str, edl_access_token: str, body: dict) -> dict: if e.response['Error']['Code'] == 'ConditionalCheckFailedException': raise DatabaseConditionException(f'Failed to update record for user {user_id}') raise - # TODO: better way to reset credits for Approved users? - #return user - return get_or_create_user(user_id) + return _reset_credits_if_needed(user=user, current_month=_get_current_month(), users_table=users_table) if application_status == APPLICATION_REJECTED: raise RejectedApplicationError(user_id) if application_status == APPLICATION_APPROVED: @@ -92,19 +90,13 @@ def _get_edl_profile(user_id: str, edl_access_token: str) -> dict: def get_or_create_user(user_id: str) -> dict: - current_month = _get_current_month() - users_table = DYNAMODB_RESOURCE.Table(environ['USERS_TABLE_NAME']) user = users_table.get_item(Key={'user_id': user_id}).get('Item') if user is None: user = _create_user(user_id, users_table) - return _reset_credits_if_needed( - user=user, - current_month=current_month, - users_table=users_table, - ) + return _reset_credits_if_needed(user=user, current_month=_get_current_month(), users_table=users_table) def _get_current_month() -> str: From a9572516fbdc24a61de344297b8716241eed6d49 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Wed, 22 May 2024 13:59:31 -0800 Subject: [PATCH 30/45] changelog --- CHANGELOG.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d8b3f5205..ed7e3a546 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,15 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [7.3.0] + +This release adds support for access codes. If a user specifies a valid, unexpired access code when they apply for HyP3 access, they will be granted automatic approval without the need for a HyP3 operator to review their application. An access code is a UUID and expires at a particular datetime. + +If you operate a HyP3 deployment, you can create a new access code by adding an item to the `AccessCodesTable` DynamoDB table for your deployment, with a new UUID for the `access_code` attribute and an ISO-formatted UTC timestamp for the `expires` attribute, e.g. `2024-06-01T00:00:00+00:00` for an access code that expires on June 1, 2024. + +### Added +- The `PATCH /user` endpoint now includes an optional `access_code` parameter and returns a `403` response if given an invalid or expired access code. + ## [7.2.1] ### Fixed From 6c006f2eab448b67cf79d44138596d97c76db17b Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Wed, 22 May 2024 14:02:56 -0800 Subject: [PATCH 31/45] unused import --- lib/dynamo/dynamo/jobs.py | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/dynamo/dynamo/jobs.py b/lib/dynamo/dynamo/jobs.py index 664b32b64..1b0902b59 100644 --- a/lib/dynamo/dynamo/jobs.py +++ b/lib/dynamo/dynamo/jobs.py @@ -1,5 +1,4 @@ import json -from datetime import datetime, timezone from decimal import Decimal from os import environ from pathlib import Path From 32226028a4a95d1147eb9774ad7569d1db297a63 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Thu, 23 May 2024 12:08:40 -0800 Subject: [PATCH 32/45] rename `expires` field to `end_date` --- CHANGELOG.md | 2 +- lib/dynamo/dynamo/user.py | 4 ++-- tests/test_api/test_patch_user.py | 6 +++--- tests/test_dynamo/test_user.py | 6 +++--- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ed7e3a546..c28c9ecb2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 This release adds support for access codes. If a user specifies a valid, unexpired access code when they apply for HyP3 access, they will be granted automatic approval without the need for a HyP3 operator to review their application. An access code is a UUID and expires at a particular datetime. -If you operate a HyP3 deployment, you can create a new access code by adding an item to the `AccessCodesTable` DynamoDB table for your deployment, with a new UUID for the `access_code` attribute and an ISO-formatted UTC timestamp for the `expires` attribute, e.g. `2024-06-01T00:00:00+00:00` for an access code that expires on June 1, 2024. +If you operate a HyP3 deployment, you can create a new access code by adding an item to the `AccessCodesTable` DynamoDB table for your deployment, with a new UUID for the `access_code` attribute and an ISO-formatted UTC timestamp for the `end_date` attribute, e.g. `2024-06-01T00:00:00+00:00` for an access code that expires on June 1, 2024. ### Added - The `PATCH /user` endpoint now includes an optional `access_code` parameter and returns a `403` response if given an invalid or expired access code. diff --git a/lib/dynamo/dynamo/user.py b/lib/dynamo/dynamo/user.py index 4925acc43..b8c1f917f 100644 --- a/lib/dynamo/dynamo/user.py +++ b/lib/dynamo/dynamo/user.py @@ -78,8 +78,8 @@ def _validate_access_code(access_code: str) -> None: if item is None: raise AccessCodeError(f'{access_code} is not a valid access code') - if dynamo.util.current_time() >= item['expires']: - raise AccessCodeError(f'Access code {access_code} expired on {item["expires"]}') + if dynamo.util.current_time() >= item['end_date']: + raise AccessCodeError(f'Access code {access_code} expired on {item["end_date"]}') def _get_edl_profile(user_id: str, edl_access_token: str) -> dict: diff --git a/tests/test_api/test_patch_user.py b/tests/test_api/test_patch_user.py index 92d3c2e64..c480e79f8 100644 --- a/tests/test_api/test_patch_user.py +++ b/tests/test_api/test_patch_user.py @@ -70,7 +70,7 @@ def test_patch_rejected_user(client, tables): def test_patch_user_access_code(client, tables): tables.access_codes_table.put_item( - Item={'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3', 'expires': '2024-05-21T20:01:04+00:00'} + Item={'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3', 'end_date': '2024-05-21T20:01:04+00:00'} ) login(client, 'foo') @@ -101,7 +101,7 @@ def test_patch_user_access_code(client, tables): def test_patch_user_access_code_expired(client, tables): tables.access_codes_table.put_item( - Item={'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3', 'expires': '2024-05-21T20:01:04+00:00'} + Item={'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3', 'end_date': '2024-05-21T20:01:04+00:00'} ) login(client, 'foo') @@ -119,7 +119,7 @@ def test_patch_user_access_code_expired(client, tables): def test_patch_user_access_code_invalid(client, tables): tables.access_codes_table.put_item( - Item={'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3', 'expires': ''} + Item={'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3'} ) login(client, 'foo') diff --git a/tests/test_dynamo/test_user.py b/tests/test_dynamo/test_user.py index 52a057817..051c9a37e 100644 --- a/tests/test_dynamo/test_user.py +++ b/tests/test_dynamo/test_user.py @@ -190,7 +190,7 @@ def test_update_user_failed_application_status(tables): def test_update_user_access_code(tables): - tables.access_codes_table.put_item(Item={'access_code': '123', 'expires': '2024-05-21T20:01:04+00:00'}) + tables.access_codes_table.put_item(Item={'access_code': '123', 'end_date': '2024-05-21T20:01:04+00:00'}) with unittest.mock.patch('dynamo.util.current_time') as mock_current_time, \ unittest.mock.patch('dynamo.user._get_current_month') as mock_get_current_month, \ @@ -223,7 +223,7 @@ def test_update_user_access_code(tables): def test_update_user_access_code_expired(tables): - tables.access_codes_table.put_item(Item={'access_code': '123', 'expires': '2024-05-21T20:01:04+00:00'}) + tables.access_codes_table.put_item(Item={'access_code': '123', 'end_date': '2024-05-21T20:01:04+00:00'}) with unittest.mock.patch('dynamo.util.current_time') as mock_current_time: mock_current_time.return_value = '2024-05-21T20:01:05+00:00' @@ -255,7 +255,7 @@ def test_update_user_access_code_expired(tables): def test_update_user_access_code_invalid(tables): - tables.access_codes_table.put_item(Item={'access_code': '123', 'expires': ''}) + tables.access_codes_table.put_item(Item={'access_code': '123'}) with pytest.raises(AccessCodeError, match=r'.*not a valid access code.*'): dynamo.user.update_user( From ff9a7f68815e94e84a2e027a65bc4d1142974d65 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Thu, 23 May 2024 12:18:53 -0800 Subject: [PATCH 33/45] allow any string for access_code --- CHANGELOG.md | 4 ++-- apps/api/src/hyp3_api/api-spec/openapi-spec.yml.j2 | 3 +-- tests/test_api/test_patch_user.py | 14 +++++++------- 3 files changed, 10 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c28c9ecb2..b0ae739e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,9 +6,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [7.3.0] -This release adds support for access codes. If a user specifies a valid, unexpired access code when they apply for HyP3 access, they will be granted automatic approval without the need for a HyP3 operator to review their application. An access code is a UUID and expires at a particular datetime. +This release adds support for access codes. If a user specifies a valid, unexpired access code when they apply for HyP3 access, they will be granted automatic approval without the need for a HyP3 operator to review their application. -If you operate a HyP3 deployment, you can create a new access code by adding an item to the `AccessCodesTable` DynamoDB table for your deployment, with a new UUID for the `access_code` attribute and an ISO-formatted UTC timestamp for the `end_date` attribute, e.g. `2024-06-01T00:00:00+00:00` for an access code that expires on June 1, 2024. +If you operate a HyP3 deployment, you can create a new access code by adding an item to the `AccessCodesTable` DynamoDB table for your deployment, with any string for the `access_code` attribute and an ISO-formatted UTC timestamp for the `end_date` attribute, e.g. `2024-06-01T00:00:00+00:00` for an access code that expires on June 1, 2024. ### Added - The `PATCH /user` endpoint now includes an optional `access_code` parameter and returns a `403` response if given an invalid or expired access code. diff --git a/apps/api/src/hyp3_api/api-spec/openapi-spec.yml.j2 b/apps/api/src/hyp3_api/api-spec/openapi-spec.yml.j2 index ace466fe1..16c014488 100644 --- a/apps/api/src/hyp3_api/api-spec/openapi-spec.yml.j2 +++ b/apps/api/src/hyp3_api/api-spec/openapi-spec.yml.j2 @@ -320,8 +320,7 @@ components: access_code: description: Grants automatic user approval while the code remains active. type: string - format: uuid - example: 27836b79-e5b2-4d8f-932f-659724ea02c3 + example: 123 user_id: description: Username from Earthdata Login. diff --git a/tests/test_api/test_patch_user.py b/tests/test_api/test_patch_user.py index c480e79f8..80171de21 100644 --- a/tests/test_api/test_patch_user.py +++ b/tests/test_api/test_patch_user.py @@ -70,7 +70,7 @@ def test_patch_rejected_user(client, tables): def test_patch_user_access_code(client, tables): tables.access_codes_table.put_item( - Item={'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3', 'end_date': '2024-05-21T20:01:04+00:00'} + Item={'access_code': '123', 'end_date': '2024-05-21T20:01:04+00:00'} ) login(client, 'foo') @@ -82,7 +82,7 @@ def test_patch_user_access_code(client, tables): response = client.patch( USER_URI, - json={'use_case': 'I want data.', 'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3'} + json={'use_case': 'I want data.', 'access_code': '123'} ) mock_current_time.assert_called_once_with() @@ -95,13 +95,13 @@ def test_patch_user_access_code(client, tables): 'remaining_credits': Decimal(25), 'job_names': [], 'use_case': 'I want data.', - 'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3', + 'access_code': '123', } def test_patch_user_access_code_expired(client, tables): tables.access_codes_table.put_item( - Item={'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3', 'end_date': '2024-05-21T20:01:04+00:00'} + Item={'access_code': '123', 'end_date': '2024-05-21T20:01:04+00:00'} ) login(client, 'foo') @@ -109,7 +109,7 @@ def test_patch_user_access_code_expired(client, tables): mock_current_time.return_value = '2024-05-21T20:01:04+00:00' response = client.patch( USER_URI, - json={'use_case': 'I want data.', 'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3'} + json={'use_case': 'I want data.', 'access_code': '123'} ) mock_current_time.assert_called_once_with() @@ -119,13 +119,13 @@ def test_patch_user_access_code_expired(client, tables): def test_patch_user_access_code_invalid(client, tables): tables.access_codes_table.put_item( - Item={'access_code': '27836b79-e5b2-4d8f-932f-659724ea02c3'} + Item={'access_code': '123'} ) login(client, 'foo') response = client.patch( USER_URI, - json={'use_case': 'I want data.', 'access_code': '580ef99b-0e16-46b6-8902-c3586f2a8065'} + json={'use_case': 'I want data.', 'access_code': '456'} ) assert response.status_code == HTTPStatus.FORBIDDEN From 7992faca1bbda098ebd7360f4b7fbf16b1a8be97 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Thu, 23 May 2024 12:21:32 -0800 Subject: [PATCH 34/45] Update lib/dynamo/dynamo/user.py Co-authored-by: Forrest Williams <31411324+forrestfwilliams@users.noreply.github.com> --- lib/dynamo/dynamo/user.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/dynamo/dynamo/user.py b/lib/dynamo/dynamo/user.py index b8c1f917f..f39aab7ce 100644 --- a/lib/dynamo/dynamo/user.py +++ b/lib/dynamo/dynamo/user.py @@ -63,7 +63,8 @@ def update_user(user_id: str, edl_access_token: str, body: dict) -> dict: if e.response['Error']['Code'] == 'ConditionalCheckFailedException': raise DatabaseConditionException(f'Failed to update record for user {user_id}') raise - return _reset_credits_if_needed(user=user, current_month=_get_current_month(), users_table=users_table) + user = _reset_credits_if_needed(user=user, current_month=_get_current_month(), users_table=users_table) + return user if application_status == APPLICATION_REJECTED: raise RejectedApplicationError(user_id) if application_status == APPLICATION_APPROVED: From a53bce280fc95ef79b7a077bcb042cd5232061ca Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Thu, 23 May 2024 12:25:10 -0800 Subject: [PATCH 35/45] rename function --- lib/dynamo/dynamo/jobs.py | 4 ++-- lib/dynamo/dynamo/user.py | 2 +- lib/dynamo/dynamo/util.py | 2 +- tests/test_api/test_get_user.py | 4 ++-- tests/test_api/test_submit_job.py | 4 ++-- tests/test_dynamo/test_jobs.py | 4 ++-- 6 files changed, 10 insertions(+), 10 deletions(-) diff --git a/lib/dynamo/dynamo/jobs.py b/lib/dynamo/dynamo/jobs.py index 1b0902b59..ddeb579dd 100644 --- a/lib/dynamo/dynamo/jobs.py +++ b/lib/dynamo/dynamo/jobs.py @@ -16,7 +16,7 @@ RejectedApplicationError, ) from dynamo.user import APPLICATION_APPROVED, APPLICATION_NOT_STARTED, APPLICATION_PENDING, APPLICATION_REJECTED -from dynamo.util import DYNAMODB_RESOURCE, convert_floats_to_decimals, current_time, get_request_time_expression +from dynamo.util import DYNAMODB_RESOURCE, convert_floats_to_decimals, current_utc_time, get_request_time_expression costs_file = Path(__file__).parent / 'costs.json' COSTS = convert_floats_to_decimals(json.loads(costs_file.read_text())) @@ -31,7 +31,7 @@ def put_jobs(user_id: str, jobs: List[dict], dry_run=False) -> List[dict]: table = DYNAMODB_RESOURCE.Table(environ['JOBS_TABLE_NAME']) - request_time = current_time() + request_time = current_utc_time() user_record = dynamo.user.get_or_create_user(user_id) diff --git a/lib/dynamo/dynamo/user.py b/lib/dynamo/dynamo/user.py index f39aab7ce..b0e454904 100644 --- a/lib/dynamo/dynamo/user.py +++ b/lib/dynamo/dynamo/user.py @@ -79,7 +79,7 @@ def _validate_access_code(access_code: str) -> None: if item is None: raise AccessCodeError(f'{access_code} is not a valid access code') - if dynamo.util.current_time() >= item['end_date']: + if dynamo.util.current_utc_time() >= item['end_date']: raise AccessCodeError(f'Access code {access_code} expired on {item["end_date"]}') diff --git a/lib/dynamo/dynamo/util.py b/lib/dynamo/dynamo/util.py index f7d09650e..37a2f7f0e 100644 --- a/lib/dynamo/dynamo/util.py +++ b/lib/dynamo/dynamo/util.py @@ -28,7 +28,7 @@ def format_time(time: datetime) -> str: return utc_time.isoformat(timespec='seconds') -def current_time() -> str: +def current_utc_time() -> str: return format_time(datetime.now(timezone.utc)) diff --git a/tests/test_api/test_get_user.py b/tests/test_api/test_get_user.py index f08d7d817..23acef7be 100644 --- a/tests/test_api/test_get_user.py +++ b/tests/test_api/test_get_user.py @@ -3,7 +3,7 @@ from test_api.conftest import USER_URI, login, make_db_record from dynamo.user import APPLICATION_APPROVED, APPLICATION_NOT_STARTED, APPLICATION_REJECTED -from dynamo.util import current_time +from dynamo.util import current_utc_time def test_get_new_user(client, tables, monkeypatch): @@ -45,7 +45,7 @@ def test_get_user_with_jobs(client, tables): } tables.users_table.put_item(Item=user) - request_time = current_time() + request_time = current_utc_time() items = [ make_db_record('job1', user_id=user_id, request_time=request_time, status_code='PENDING', name='job1'), make_db_record('job2', user_id=user_id, request_time=request_time, status_code='RUNNING', name='job1'), diff --git a/tests/test_api/test_submit_job.py b/tests/test_api/test_submit_job.py index 3faec48d5..292b7e26e 100644 --- a/tests/test_api/test_submit_job.py +++ b/tests/test_api/test_submit_job.py @@ -4,7 +4,7 @@ from test_api.conftest import login, make_job, setup_requests_mock, submit_batch from dynamo.user import APPLICATION_PENDING -from dynamo.util import current_time +from dynamo.util import current_utc_time def test_submit_one_job(client, approved_user): @@ -16,7 +16,7 @@ def test_submit_one_job(client, approved_user): jobs = response.json['jobs'] assert len(jobs) == 1 assert jobs[0]['status_code'] == 'PENDING' - assert jobs[0]['request_time'] <= current_time() + assert jobs[0]['request_time'] <= current_utc_time() assert jobs[0]['user_id'] == approved_user diff --git a/tests/test_dynamo/test_jobs.py b/tests/test_dynamo/test_jobs.py index aa5539017..b4bafa4d5 100644 --- a/tests/test_dynamo/test_jobs.py +++ b/tests/test_dynamo/test_jobs.py @@ -13,7 +13,7 @@ RejectedApplicationError, ) from dynamo.user import APPLICATION_APPROVED -from dynamo.util import current_time +from dynamo.util import current_utc_time def test_query_jobs_by_user(tables): @@ -281,7 +281,7 @@ def test_put_jobs(tables, monkeypatch, approved_user): assert set(job.keys()) == { 'name', 'job_id', 'user_id', 'status_code', 'execution_started', 'request_time', 'priority', 'credit_cost' } - assert job['request_time'] <= current_time() + assert job['request_time'] <= current_utc_time() assert job['user_id'] == approved_user assert job['status_code'] == 'PENDING' assert job['execution_started'] is False From adf2e8ab20133e0674e1999ffea582d8aa33c397 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Thu, 23 May 2024 12:27:45 -0800 Subject: [PATCH 36/45] rename some tests --- tests/test_api/test_patch_user.py | 2 +- tests/test_dynamo/test_user.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_api/test_patch_user.py b/tests/test_api/test_patch_user.py index 80171de21..e5d8c58b7 100644 --- a/tests/test_api/test_patch_user.py +++ b/tests/test_api/test_patch_user.py @@ -99,7 +99,7 @@ def test_patch_user_access_code(client, tables): } -def test_patch_user_access_code_expired(client, tables): +def test_patch_user_access_code_end_date(client, tables): tables.access_codes_table.put_item( Item={'access_code': '123', 'end_date': '2024-05-21T20:01:04+00:00'} ) diff --git a/tests/test_dynamo/test_user.py b/tests/test_dynamo/test_user.py index 051c9a37e..b1a3786e6 100644 --- a/tests/test_dynamo/test_user.py +++ b/tests/test_dynamo/test_user.py @@ -222,7 +222,7 @@ def test_update_user_access_code(tables): assert tables.users_table.scan()['Items'] == [user] -def test_update_user_access_code_expired(tables): +def test_update_user_access_code_end_date(tables): tables.access_codes_table.put_item(Item={'access_code': '123', 'end_date': '2024-05-21T20:01:04+00:00'}) with unittest.mock.patch('dynamo.util.current_time') as mock_current_time: From b76294d2bed2ef576ef430d0efad0e40e1579bc3 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Thu, 23 May 2024 12:30:03 -0800 Subject: [PATCH 37/45] rename function in mocks --- tests/test_api/test_patch_user.py | 12 ++++++------ tests/test_dynamo/test_user.py | 18 +++++++++--------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/tests/test_api/test_patch_user.py b/tests/test_api/test_patch_user.py index e5d8c58b7..54b4dbec5 100644 --- a/tests/test_api/test_patch_user.py +++ b/tests/test_api/test_patch_user.py @@ -74,10 +74,10 @@ def test_patch_user_access_code(client, tables): ) login(client, 'foo') - with unittest.mock.patch('dynamo.util.current_time') as mock_current_time, \ + with unittest.mock.patch('dynamo.util.current_utc_time') as mock_current_utc_time, \ unittest.mock.patch('dynamo.user._get_edl_profile') as mock_get_edl_profile: - mock_current_time.return_value = '2024-05-21T20:01:03+00:00' + mock_current_utc_time.return_value = '2024-05-21T20:01:03+00:00' mock_get_edl_profile.return_value = {} response = client.patch( @@ -85,7 +85,7 @@ def test_patch_user_access_code(client, tables): json={'use_case': 'I want data.', 'access_code': '123'} ) - mock_current_time.assert_called_once_with() + mock_current_utc_time.assert_called_once_with() mock_get_edl_profile.assert_called_once_with('foo', DEFAULT_ACCESS_TOKEN) assert response.status_code == HTTPStatus.OK @@ -105,13 +105,13 @@ def test_patch_user_access_code_end_date(client, tables): ) login(client, 'foo') - with unittest.mock.patch('dynamo.util.current_time') as mock_current_time: - mock_current_time.return_value = '2024-05-21T20:01:04+00:00' + with unittest.mock.patch('dynamo.util.current_utc_time') as mock_current_utc_time: + mock_current_utc_time.return_value = '2024-05-21T20:01:04+00:00' response = client.patch( USER_URI, json={'use_case': 'I want data.', 'access_code': '123'} ) - mock_current_time.assert_called_once_with() + mock_current_utc_time.assert_called_once_with() assert response.status_code == HTTPStatus.FORBIDDEN assert 'expired' in response.json['detail'] diff --git a/tests/test_dynamo/test_user.py b/tests/test_dynamo/test_user.py index b1a3786e6..776680aca 100644 --- a/tests/test_dynamo/test_user.py +++ b/tests/test_dynamo/test_user.py @@ -192,11 +192,11 @@ def test_update_user_failed_application_status(tables): def test_update_user_access_code(tables): tables.access_codes_table.put_item(Item={'access_code': '123', 'end_date': '2024-05-21T20:01:04+00:00'}) - with unittest.mock.patch('dynamo.util.current_time') as mock_current_time, \ + with unittest.mock.patch('dynamo.util.current_utc_time') as mock_current_utc_time, \ unittest.mock.patch('dynamo.user._get_current_month') as mock_get_current_month, \ unittest.mock.patch('dynamo.user._get_edl_profile') as mock_get_edl_profile: - mock_current_time.return_value = '2024-05-21T20:01:03+00:00' + mock_current_utc_time.return_value = '2024-05-21T20:01:03+00:00' mock_get_current_month.return_value = '2024-05' mock_get_edl_profile.return_value = {'key': 'value'} @@ -206,7 +206,7 @@ def test_update_user_access_code(tables): {'use_case': 'I want data.', 'access_code': '123'} ) - mock_current_time.assert_called_once_with() + mock_current_utc_time.assert_called_once_with() assert mock_get_current_month.mock_calls == [unittest.mock.call()] * 2 mock_get_edl_profile.assert_called_once_with('foo', 'test-edl-access-token') @@ -225,29 +225,29 @@ def test_update_user_access_code(tables): def test_update_user_access_code_end_date(tables): tables.access_codes_table.put_item(Item={'access_code': '123', 'end_date': '2024-05-21T20:01:04+00:00'}) - with unittest.mock.patch('dynamo.util.current_time') as mock_current_time: - mock_current_time.return_value = '2024-05-21T20:01:05+00:00' + with unittest.mock.patch('dynamo.util.current_utc_time') as mock_current_utc_time: + mock_current_utc_time.return_value = '2024-05-21T20:01:05+00:00' with pytest.raises(AccessCodeError, match=r'.*expired.*'): dynamo.user.update_user( 'foo', 'test-edl-access-token', {'use_case': 'I want data.', 'access_code': '123'} ) - mock_current_time.assert_called_once_with() + mock_current_utc_time.assert_called_once_with() assert tables.users_table.scan()['Items'] == [ {'user_id': 'foo', 'remaining_credits': Decimal(0), 'application_status': APPLICATION_NOT_STARTED} ] - with unittest.mock.patch('dynamo.util.current_time') as mock_current_time: - mock_current_time.return_value = '2024-05-21T20:01:04+00:00' + with unittest.mock.patch('dynamo.util.current_utc_time') as mock_current_utc_time: + mock_current_utc_time.return_value = '2024-05-21T20:01:04+00:00' with pytest.raises(AccessCodeError, match=r'.*expired.*'): dynamo.user.update_user( 'foo', 'test-edl-access-token', {'use_case': 'I want data.', 'access_code': '123'} ) - mock_current_time.assert_called_once_with() + mock_current_utc_time.assert_called_once_with() assert tables.users_table.scan()['Items'] == [ {'user_id': 'foo', 'remaining_credits': Decimal(0), 'application_status': APPLICATION_NOT_STARTED} From 5e4edeecc0f6bec445060cd085f6c9645b29a20c Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Thu, 23 May 2024 12:58:33 -0800 Subject: [PATCH 38/45] add start_date for access codes --- lib/dynamo/dynamo/user.py | 6 +++++- tests/test_api/test_patch_user.py | 22 ++++++++++++++++++++-- tests/test_dynamo/test_user.py | 26 ++++++++++++++++++++++++-- 3 files changed, 49 insertions(+), 5 deletions(-) diff --git a/lib/dynamo/dynamo/user.py b/lib/dynamo/dynamo/user.py index b0e454904..df25c70d6 100644 --- a/lib/dynamo/dynamo/user.py +++ b/lib/dynamo/dynamo/user.py @@ -79,7 +79,11 @@ def _validate_access_code(access_code: str) -> None: if item is None: raise AccessCodeError(f'{access_code} is not a valid access code') - if dynamo.util.current_utc_time() >= item['end_date']: + now = dynamo.util.current_utc_time() + if now < item['start_date']: + raise AccessCodeError(f'Access code {access_code} will become active on {item["start_date"]}') + + if now >= item['end_date']: raise AccessCodeError(f'Access code {access_code} expired on {item["end_date"]}') diff --git a/tests/test_api/test_patch_user.py b/tests/test_api/test_patch_user.py index 54b4dbec5..a3086a63d 100644 --- a/tests/test_api/test_patch_user.py +++ b/tests/test_api/test_patch_user.py @@ -70,7 +70,7 @@ def test_patch_rejected_user(client, tables): def test_patch_user_access_code(client, tables): tables.access_codes_table.put_item( - Item={'access_code': '123', 'end_date': '2024-05-21T20:01:04+00:00'} + Item={'access_code': '123', 'start_date': '2024-05-21T20:01:03+00:00', 'end_date': '2024-05-21T20:01:04+00:00'} ) login(client, 'foo') @@ -99,9 +99,27 @@ def test_patch_user_access_code(client, tables): } +def test_patch_user_access_code_start_date(client, tables): + tables.access_codes_table.put_item( + Item={'access_code': '123', 'start_date': '2024-05-21T20:01:03+00:00'} + ) + login(client, 'foo') + + with unittest.mock.patch('dynamo.util.current_utc_time') as mock_current_utc_time: + mock_current_utc_time.return_value = '2024-05-21T20:01:02+00:00' + response = client.patch( + USER_URI, + json={'use_case': 'I want data.', 'access_code': '123'} + ) + mock_current_utc_time.assert_called_once_with() + + assert response.status_code == HTTPStatus.FORBIDDEN + assert 'will become active' in response.json['detail'] + + def test_patch_user_access_code_end_date(client, tables): tables.access_codes_table.put_item( - Item={'access_code': '123', 'end_date': '2024-05-21T20:01:04+00:00'} + Item={'access_code': '123', 'start_date': '2024-05-21T20:01:03+00:00', 'end_date': '2024-05-21T20:01:04+00:00'} ) login(client, 'foo') diff --git a/tests/test_dynamo/test_user.py b/tests/test_dynamo/test_user.py index 776680aca..6b1fed030 100644 --- a/tests/test_dynamo/test_user.py +++ b/tests/test_dynamo/test_user.py @@ -190,7 +190,9 @@ def test_update_user_failed_application_status(tables): def test_update_user_access_code(tables): - tables.access_codes_table.put_item(Item={'access_code': '123', 'end_date': '2024-05-21T20:01:04+00:00'}) + tables.access_codes_table.put_item( + Item={'access_code': '123', 'start_date': '2024-05-21T20:01:03+00:00', 'end_date': '2024-05-21T20:01:04+00:00'} + ) with unittest.mock.patch('dynamo.util.current_utc_time') as mock_current_utc_time, \ unittest.mock.patch('dynamo.user._get_current_month') as mock_get_current_month, \ @@ -222,8 +224,28 @@ def test_update_user_access_code(tables): assert tables.users_table.scan()['Items'] == [user] +def test_update_user_access_code_start_date(tables): + tables.access_codes_table.put_item(Item={'access_code': '123', 'start_date': '2024-05-21T20:01:03+00:00'}) + + with unittest.mock.patch('dynamo.util.current_utc_time') as mock_current_utc_time: + mock_current_utc_time.return_value = '2024-05-21T20:01:02+00:00' + with pytest.raises(AccessCodeError, match=r'.*will become active.*'): + dynamo.user.update_user( + 'foo', + 'test-edl-access-token', + {'use_case': 'I want data.', 'access_code': '123'} + ) + mock_current_utc_time.assert_called_once_with() + + assert tables.users_table.scan()['Items'] == [ + {'user_id': 'foo', 'remaining_credits': Decimal(0), 'application_status': APPLICATION_NOT_STARTED} + ] + + def test_update_user_access_code_end_date(tables): - tables.access_codes_table.put_item(Item={'access_code': '123', 'end_date': '2024-05-21T20:01:04+00:00'}) + tables.access_codes_table.put_item( + Item={'access_code': '123', 'start_date': '2024-05-21T20:01:03+00:00', 'end_date': '2024-05-21T20:01:04+00:00'} + ) with unittest.mock.patch('dynamo.util.current_utc_time') as mock_current_utc_time: mock_current_utc_time.return_value = '2024-05-21T20:01:05+00:00' From d3ac85c33f3756474199fd0ffffded68c1e54c59 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Thu, 23 May 2024 13:02:44 -0800 Subject: [PATCH 39/45] update changelog --- CHANGELOG.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b0ae739e9..526f658fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,12 +6,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [7.3.0] -This release adds support for access codes. If a user specifies a valid, unexpired access code when they apply for HyP3 access, they will be granted automatic approval without the need for a HyP3 operator to review their application. +This release adds support for access codes. If a user specifies an active access code when they apply for HyP3 access, they will be granted automatic approval without the need for a HyP3 operator to review their application. -If you operate a HyP3 deployment, you can create a new access code by adding an item to the `AccessCodesTable` DynamoDB table for your deployment, with any string for the `access_code` attribute and an ISO-formatted UTC timestamp for the `end_date` attribute, e.g. `2024-06-01T00:00:00+00:00` for an access code that expires on June 1, 2024. +If you operate a HyP3 deployment, you can create a new access code by adding an item to the `AccessCodesTable` DynamoDB table for your deployment, with any string for the `access_code` attribute and an ISO-formatted UTC timestamp for the `start_date` and `end_date` attributes, e.g. `2024-06-01T00:00:00+00:00` and `2024-06-02T00:00:00+00:00` for an access code that becomes active on June 1, 2024 and expires on June 2, 2024. ### Added -- The `PATCH /user` endpoint now includes an optional `access_code` parameter and returns a `403` response if given an invalid or expired access code. +- The `PATCH /user` endpoint now includes an optional `access_code` parameter and returns a `403` response if given an invalid or inactive access code. ## [7.2.1] From e2ddd8468e31008acf333fb0b9e39ece94f7bd51 Mon Sep 17 00:00:00 2001 From: Charlie Marshak Date: Thu, 23 May 2024 14:23:35 -0700 Subject: [PATCH 40/45] Update deploy-enterprise.yml --- .github/workflows/deploy-enterprise.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/deploy-enterprise.yml b/.github/workflows/deploy-enterprise.yml index d71172d58..0e3219eb7 100644 --- a/.github/workflows/deploy-enterprise.yml +++ b/.github/workflows/deploy-enterprise.yml @@ -37,7 +37,7 @@ jobs: domain: hyp3-a19-jpl.asf.alaska.edu template_bucket: cf-templates-v4pvone059de-us-west-2 image_tag: latest - product_lifetime_in_days: 180 + product_lifetime_in_days: 30 default_credits_per_user: 0 default_application_status: APPROVED cost_profile: DEFAULT @@ -45,8 +45,8 @@ jobs: job_spec/ARIA_RAIDER.yml job_spec/INSAR_ISCE.yml instance_types: c6id.xlarge,c6id.2xlarge,c6id.4xlarge,c6id.8xlarge - default_max_vcpus: 4000 - expanded_max_vcpus: 4000 + default_max_vcpus: 0 + expanded_max_vcpus: 0 required_surplus: 0 security_environment: JPL-public ami_id: /aws/service/ecs/optimized-ami/amazon-linux-2023/recommended/image_id From 1e2494806051a9a8b92ca853065a5c0a90425b16 Mon Sep 17 00:00:00 2001 From: Charlie Marshak Date: Thu, 23 May 2024 14:33:07 -0700 Subject: [PATCH 41/45] Update CHANGELOG.md --- CHANGELOG.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 526f658fd..4cae075cf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [7.3.0] +### Added +* Turn off hyp3 ACCESS spend by zeroing the max VCPUs in the associated deployment. +* Reduce product lifetime in hyp3 ACCESS deployment to 14 days. + +## [7.3.0] + This release adds support for access codes. If a user specifies an active access code when they apply for HyP3 access, they will be granted automatic approval without the need for a HyP3 operator to review their application. If you operate a HyP3 deployment, you can create a new access code by adding an item to the `AccessCodesTable` DynamoDB table for your deployment, with any string for the `access_code` attribute and an ISO-formatted UTC timestamp for the `start_date` and `end_date` attributes, e.g. `2024-06-01T00:00:00+00:00` and `2024-06-02T00:00:00+00:00` for an access code that becomes active on June 1, 2024 and expires on June 2, 2024. From dd505cb3337f01110dff44add7faf257b58c5fd2 Mon Sep 17 00:00:00 2001 From: Charlie Marshak Date: Thu, 23 May 2024 14:35:52 -0700 Subject: [PATCH 42/45] Update CHANGELOG.md --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4cae075cf..fda8636fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [7.3.0] +## [7.3.1] ### Added * Turn off hyp3 ACCESS spend by zeroing the max VCPUs in the associated deployment. From 654785ac09c89f2c468d84a6e3be0112a29cd6a1 Mon Sep 17 00:00:00 2001 From: Charlie Marshak Date: Thu, 23 May 2024 15:02:39 -0700 Subject: [PATCH 43/45] Update deploy-enterprise.yml --- .github/workflows/deploy-enterprise.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/deploy-enterprise.yml b/.github/workflows/deploy-enterprise.yml index 0e3219eb7..23669e144 100644 --- a/.github/workflows/deploy-enterprise.yml +++ b/.github/workflows/deploy-enterprise.yml @@ -37,7 +37,7 @@ jobs: domain: hyp3-a19-jpl.asf.alaska.edu template_bucket: cf-templates-v4pvone059de-us-west-2 image_tag: latest - product_lifetime_in_days: 30 + product_lifetime_in_days: 14 default_credits_per_user: 0 default_application_status: APPROVED cost_profile: DEFAULT From c9a604363c9215be94574549d38546b3c7ce3825 Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Thu, 23 May 2024 14:36:33 -0800 Subject: [PATCH 44/45] fix changelog --- CHANGELOG.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fda8636fb..012a2bd6b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,12 +4,6 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [7.3.1] - -### Added -* Turn off hyp3 ACCESS spend by zeroing the max VCPUs in the associated deployment. -* Reduce product lifetime in hyp3 ACCESS deployment to 14 days. - ## [7.3.0] This release adds support for access codes. If a user specifies an active access code when they apply for HyP3 access, they will be granted automatic approval without the need for a HyP3 operator to review their application. @@ -18,6 +12,8 @@ If you operate a HyP3 deployment, you can create a new access code by adding an ### Added - The `PATCH /user` endpoint now includes an optional `access_code` parameter and returns a `403` response if given an invalid or inactive access code. +* Turn off hyp3 ACCESS spend by zeroing the max VCPUs in the associated deployment. +* Reduce product lifetime in hyp3 ACCESS deployment to 14 days. ## [7.2.1] From decdc0cc54f204f80c161e270e7e25590787826b Mon Sep 17 00:00:00 2001 From: Jake Herrmann Date: Thu, 23 May 2024 14:38:00 -0800 Subject: [PATCH 45/45] fix changelog again --- CHANGELOG.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 012a2bd6b..c260abd71 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,8 +12,10 @@ If you operate a HyP3 deployment, you can create a new access code by adding an ### Added - The `PATCH /user` endpoint now includes an optional `access_code` parameter and returns a `403` response if given an invalid or inactive access code. -* Turn off hyp3 ACCESS spend by zeroing the max VCPUs in the associated deployment. -* Reduce product lifetime in hyp3 ACCESS deployment to 14 days. + +### Changed +- Turn off hyp3 ACCESS spend by zeroing the max VCPUs in the associated deployment. +- Reduce product lifetime in hyp3 ACCESS deployment to 14 days. ## [7.2.1]