From 590aa02931b784cd86e8af2bdf6cd2bb6cdaf3f7 Mon Sep 17 00:00:00 2001 From: Andrew <84722778+andrew-jameson@users.noreply.github.com> Date: Mon, 29 Jul 2024 14:14:38 -0400 Subject: [PATCH 1/2] Feat/1620 code signature (#3070) * Adding action during deploy * Bad branch name, quick merge. (#3069) * - Initial commit for duplicate record manager for TANF section 1 records * - more efficient error generation * - Added a way to track precedence in the error messages to avoid generating less meaningful messages - Updated error generation * - Fix lint - Add docstrings * - Added support to delete deplicate records * - add support for bulk/raw delete * - Update deletes on rollbacks to raw delete since it wont hurt anything * - Updated elastic and django to support raw/bulk deleting * - Updating rollback test since it is very relevant now that we are doing raw deletes * - raise log level * - Update test * - remove import * - Update error generation to use record and schema - rename var * - Update error precedence logic * - fix lint * - use correct method * - remove extra call to bulk create errors * - move duplicate manager into case consistency validator - Starting to remove SortedRecordSchemaPairs from case consistency validator - updating to support in memory record removal if they havent been serialized * - update add_record logic to include rpt_month_year * - Move SortedRecordSchema pairs to util - Update case consistency validator to not use OG SortedRecordSchemaPairs - Update dup logic to not consider records on the same line * - remove unused import * - Update class to store records based on hash pending the section type * - Update parse to leverage new data structure * - clearing after bulk create * - Rename to be more accurate - Add method to generate bulk create dictionary - Stub function for removing cases * - Updated class name - basing off of doc instead of schema * - Adding infrastructure to remove records from memory or the DB pending their state * - fix logic issue * - fix logic error * - added back list of cases * - fixed test_add_record * - Fix all case consistency tests * - Fix most lint errors * - remove print * - update rollback logic - fix most parse tests * - Update duplicate error messages - Added logging - Updated tests * - functionized dup manager a bit - updated validator to query errors from dup manager - added test for s1 duplicate detection * - Add test for S1 partial duplicate detection * - fix lint * - Add section 2 tests * - Fix reference error * - order by id * - parametrizing batch size - initial integration tests * - move creation logic in parse.py - add container to monitor what cases have serialized records - add test for in memory and DB based record removal with dupes detected * - added section 2 duplicate tests * - Test for partial duplicates - test doesnt behave as I would expect * - fix lint * - Fixed logic error where last case in file wouldnt be removed if it has cat4 errors * - Update SortedRecords to leverage an unsorted container to make bulk createing faster. Dont have to iterate/generate the required structure * - naming and doc string updates * - renaming functions - removing unnecessary members * - update doc strings * - Add test for family affiliation negating partial duplicity * - Updated to support duplicate checking on section 3/4 * - fix lint * - Update docstring * - Update to allow all record types to have duplicate detection * - add duplicate detection unit tests for all program types and record types * - fix lint * - Add cases for ssp * - Move parser fixtures to their own conftest.py * - Move conftest.py to the test folder * - removing whitespace * - Remove 'test' from fixture names * - Fix failing tests due to merge * - Update message for failing test * - Remove Vars from compose file - Remove unnecessary checks from cat4 validator - Updated error counts after removing check * - fixed test * - linting * - move partial hash checking logic to schema/fields * - Move fixtures to conftest.py * - Fix failing tests * - fixed tests * Revert "- move partial hash checking logic to schema/fields" This reverts commit ab379ce79ea1575802aa5b508662935e3a0f2f9d. * - Better way to do partial hash checking - Still in schema * - Moved hash generation into schema - updated tests and factories to now support agnostic hash generation * - Updated error message - fixed incorrect indentation on error counting * - move fixtures to conftest * add radio button for jurisdiction type * - fix lint * - fixed tests from merge conflict * add tests * - Update error creation logging to track all errors - Updated queries to most correct syntax for aggregates function - Updated tests * - lint * - move fixture * - Updated aggregates query to take into account cat4 errors * - lint * - import as Query * - moved fixture to correct location * - Updated cat4 duplicate errors to include fields that are duplicated * - fix lint * - update tests * - update from merge conflict * - Add additional exclusion criteria for necessary tests - add missing fixture decorators * - renamed parameters and classes to make more sense with their duties. - Added much more detailed descriptions where information surrounding duplicate detection can be confused. * - implement TODOs that came out of OH * - remove todo - add warning labels to shared object * - fix lint * - adding some generic parsing flow docs * update error string * fix tests * - Added useful debug logging to partial hash functions * - Updated duplicate detection to allow "partial duplicates" of s3 and s4 files - Added tests for partial dup s3/s4 * - updated file status to be consistent with expectations * - Updated stt filter to display stt name instead of code to make it easier to filter * - Remove custom filter in favor of dependency - Update STTFilter to leverage dependency filter * - update doc string * - fix lint * - Updated case_consistency validator to track which case has been validated to identify which case might need to be removed - decoupled dup manager and case consistency val error counts - Added more descriptive logging * - Update tests * - Add edge case cat4 test - unblock util function that was causing errors * - Updated manager to correctly delete serialized records - Updated DFS to correctly count total records and total records created - Updated cat4 test case to be more extensive * - Updated test * - fix lint * - Add elastic specific exception handling * - add extra case to file for duplicate detection * - Added creation of LogEntry objects in exception blocks * - Updated fiscal year filter to not list `All` twice - Updated STT filter lookup options * - fix lint * - Add line number to duplicate error ParserError objects * - Remove dup test class - Updated to use correct error generator * - fix lint * - Updated based on review feedback * - Update tests * - fix lint * don't show jurisdiction type for acf users * Create sprint-99-summary.md (#3038) * Create sprint-99-summary.md * Update docs/Sprint-Review/sprint-99-summary.md Co-authored-by: Lauren Frohlich <61251539+lfrohlich@users.noreply.github.com> --------- Co-authored-by: Lauren Frohlich <61251539+lfrohlich@users.noreply.github.com> * Create sprint-100-summary.md (#3039) * Create sprint-100-summary.md * Update docs/Sprint-Review/sprint-100-summary.md Co-authored-by: Lauren Frohlich <61251539+lfrohlich@users.noreply.github.com> * Update docs/Sprint-Review/sprint-100-summary.md Co-authored-by: Lauren Frohlich <61251539+lfrohlich@users.noreply.github.com> * Update docs/Sprint-Review/sprint-100-summary.md Co-authored-by: Lauren Frohlich <61251539+lfrohlich@users.noreply.github.com> --------- Co-authored-by: Lauren Frohlich <61251539+lfrohlich@users.noreply.github.com> Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com> * #3021 Adds FAQ update & release notes (#3040) * Adds FAQ update & release notes * Update product-updates/knowledge-center/index.html Co-authored-by: Alex P. <63075587+ADPennington@users.noreply.github.com> * Update index.html * Adds request access updates * Update product-updates/knowledge-center/index.html Co-authored-by: Alex P. <63075587+ADPennington@users.noreply.github.com> * Whitespace change to re-trigger full pipeline Whitespace change to re-trigger full pipeline --------- Co-authored-by: Alex P. <63075587+ADPennington@users.noreply.github.com> Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com> * 2693 cat2 messaging cleanup (#2963) * clean option string util * add friendly name and item num to cat 2 * fix linting * fix a couple tests * add parens to friendly names * lint * fix tests * fix more tests * fix merge duplication * fix tests * lint * fix tets * udpate required validator * Merge branch 'develop' into 2693-cat2-messaging-cleanup * fix tests * fix test * fix tests * fix lint --------- Co-authored-by: Alex P <63075587+ADPennington@users.noreply.github.com> Co-authored-by: Eric Lipe <125676261+elipe17@users.noreply.github.com> * 2687 Added email text for sendgrid (#3050) * 2687 Added email text for sendgrid * 2687 linting --------- Co-authored-by: Alex P <63075587+ADPennington@users.noreply.github.com> Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com> * - Added missing territories * adding placeholder for action to table * Frontpage link * Update TDP-environments-README.md --------- Co-authored-by: Eric Lipe Co-authored-by: Jan Timpe Co-authored-by: jtimpe <111305129+jtimpe@users.noreply.github.com> Co-authored-by: Eric Lipe <125676261+elipe17@users.noreply.github.com> Co-authored-by: Alex P <63075587+ADPennington@users.noreply.github.com> Co-authored-by: robgendron <163159602+robgendron@users.noreply.github.com> Co-authored-by: Lauren Frohlich <61251539+lfrohlich@users.noreply.github.com> Co-authored-by: Miles Reiter Co-authored-by: raftmsohani <97037188+raftmsohani@users.noreply.github.com> * swapping lines, readying for deploy * YAML spacing. * bypassing PR checks for faster turnaround * Pointing to blank revision, fixed to generic/latest * Dynamic filenames for testing differing environments. * moving badges out of table for sizing, hopefully * Newlines * appending git info to frontend base page * Feat/1620 merge pg upgrade (#3072) * - Upgraded local to postrgres 15.7 * - updating version on terraform * - disabling tests for now * - adding timeouts to TF for safety * - Deleting DB from TF state * - re-add DB * - remove deployment until DB is ready * - updated apt.yml to have pg15 client - updated deploy to only run apt commands * - re-enable deply dev * - UPdate client version - re-enable migrations * - really disable tests --------- Co-authored-by: Eric Lipe Co-authored-by: andrew-jameson * removing comment * undoing mass line comments * Update .circleci/build-and-test/workflows.yml * Apply suggestions from code review Co-authored-by: Alex P. <63075587+ADPennington@users.noreply.github.com> --------- Co-authored-by: andrew-jameson Co-authored-by: Eric Lipe Co-authored-by: Jan Timpe Co-authored-by: jtimpe <111305129+jtimpe@users.noreply.github.com> Co-authored-by: Eric Lipe <125676261+elipe17@users.noreply.github.com> Co-authored-by: Alex P <63075587+ADPennington@users.noreply.github.com> Co-authored-by: robgendron <163159602+robgendron@users.noreply.github.com> Co-authored-by: Lauren Frohlich <61251539+lfrohlich@users.noreply.github.com> Co-authored-by: Miles Reiter Co-authored-by: raftmsohani <97037188+raftmsohani@users.noreply.github.com> --- .github/workflows/deploy-on-label.yml | 13 ++++++++++++- README.md | 2 ++ .../TDP-environments-README.md | 17 ++++++++++++----- scripts/deploy-frontend.sh | 1 + 4 files changed, 27 insertions(+), 6 deletions(-) diff --git a/.github/workflows/deploy-on-label.yml b/.github/workflows/deploy-on-label.yml index 20fbfd8b3..f88e6d34e 100644 --- a/.github/workflows/deploy-on-label.yml +++ b/.github/workflows/deploy-on-label.yml @@ -58,9 +58,20 @@ jobs: STATE=${{ fromJson(steps.get-pr-checks.outputs.data).state }} echo "::set-output name=STATE::$STATE" echo "Current PR state: $STATE" + + - name: Live Environment Badge + uses: schneegans/dynamic-badges-action@v1.7.0 + with: + auth: ${{ secrets.LIVE_ENV_GIST }} + gistID: ded3a260ed8245a5b231ba726b3039df + filename: Live-Environments-${{steps.extract-deploy-env.outputs.DEPLOY_ENV}}.json + label: ${{steps.extract-deploy-env.outputs.DEPLOY_ENV}} + message: "${{ github.head_ref }} - ${{ github.sha }}" + color: blue + - name: Circle CI Deployment Trigger id: curl-circle-ci - if: steps.get-pr-state.outputs.STATE == 'success' + # if: steps.get-pr-state.outputs.STATE == 'success' uses: promiseofcake/circleci-trigger-action@v1 with: user-token: ${{ secrets.CIRCLE_CI_V2_TOKEN }} diff --git a/README.md b/README.md index 424accaa6..4e8241331 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,8 @@ Our vision is to build a new, secure, web-based data reporting system to improve |**Frontend Coverage**| [![Codecov-Frontend-Dev](https://codecov.io/gh/raft-tech/TANF-app/branch/develop/graph/badge.svg?flag=dev-frontend)](https://codecov.io/gh/raft-tech/TANF-app?flag=dev-frontend) | [![Codeco-Frontend-HHS](https://codecov.io/gh/HHS/TANF-app/branch/main/graph/badge.svg?flag=main-frontend)](https://codecov.io/gh/HHS/TANF-app?flag=main-frontend) | [![Codeco-Frontend-HHS](https://codecov.io/gh/HHS/TANF-app/branch/master/graph/badge.svg?flag=master-frontend)](https://codecov.io/gh/HHS/TANF-app?flag=master-frontend) |**Backend Coverage**| [![Codecov-Backend-Dev](https://codecov.io/gh/raft-tech/TANF-app/branch/develop/graph/badge.svg?flag=dev-backend)](https://codecov.io/gh/raft-tech/TANF-app/branch/develop?flag=dev-backend)| [![Codecov-Backend-HHS]( https://codecov.io/gh/HHS/TANF-app/branch/main/graph/badge.svg?flag=main-backend)](https://codecov.io/gh/HHS/TANF-app/branch/main?flag=main-backend) | [![Codecov-Backend-HHS]( https://codecov.io/gh/HHS/TANF-app/branch/master/graph/badge.svg?flag=master-backend)](https://codecov.io/gh/HHS/TANF-app/branch/master?flag=master-backend) +[Link to Current Development Deployments](https://github.com/raft-tech/TANF-app/blob/feat/1860/docs/Technical-Documentation/TDP-environments-README.md#development) + *Due to limitations imposed by Github and occasional slow server response times, some badges may require a page refresh to load.* *TDP is subject to the **[ACF Privacy Policy](https://www.acf.hhs.gov/privacy-policy)** and **[HHS Vulnerability Disclosure Policy](https://www.hhs.gov/vulnerability-disclosure-policy/index.html)***. diff --git a/docs/Technical-Documentation/TDP-environments-README.md b/docs/Technical-Documentation/TDP-environments-README.md index 162ac7275..214c5183a 100644 --- a/docs/Technical-Documentation/TDP-environments-README.md +++ b/docs/Technical-Documentation/TDP-environments-README.md @@ -2,11 +2,18 @@ ## Development -| Dev Site | Frontend URL | Backend URL | Purpose | -| -------- | -------- | -------- | -------- | -| A11y | https://tdp-frontend-a11y.app.cloud.gov | https://tdp-frontend-a11y.app.cloud.gov/admin/ | Space for accessibility testing | -| QASP | https://tdp-frontend-qasp.app.cloud.gov | https://tdp-frontend-qasp.app.cloud.gov/admin/ | Space for QASP review | -| raft | https://tdp-frontend-raft.app.cloud.gov | https://tdp-frontend-raft.app.cloud.gov/admin/ | Space for raft review | +| Dev Site | Frontend URL | Backend URL | Branch/Commit | Purpose | +| -------- | -------- | -------- | -------- | -------- | +| A11y | https://tdp-frontend-a11y.app.cloud.gov | https://tdp-frontend-a11y.app.cloud.gov/admin/ | | Space for accessibility testing | +| QASP | https://tdp-frontend-qasp.app.cloud.gov | https://tdp-frontend-qasp.app.cloud.gov/admin/ | | Space for QASP review | +| raft | https://tdp-frontend-raft.app.cloud.gov | https://tdp-frontend-raft.app.cloud.gov/admin/ | + | Space for raft review | + +![badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/andrew-jameson/ded3a260ed8245a5b231ba726b3039df/raw/Live-Environments-raft.json) + +![badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/andrew-jameson/ded3a260ed8245a5b231ba726b3039df/raw/Live-Environments-a11y.json) + +![badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/andrew-jameson/ded3a260ed8245a5b231ba726b3039df/raw/Live-Environments-qasp.json) ### Dependencies diff --git a/scripts/deploy-frontend.sh b/scripts/deploy-frontend.sh index 07d47e980..cbdd32245 100755 --- a/scripts/deploy-frontend.sh +++ b/scripts/deploy-frontend.sh @@ -57,6 +57,7 @@ update_frontend() unlink .env.production mkdir deployment + cp -r build deployment/public cp nginx/cloud.gov/buildpack.nginx.conf deployment/nginx.conf cp nginx/cloud.gov/locations.conf deployment/locations.conf From 128390a39ca35601982ea4a9c3a8032a1313d7fd Mon Sep 17 00:00:00 2001 From: Andrew <84722778+andrew-jameson@users.noreply.github.com> Date: Mon, 29 Jul 2024 14:26:57 -0400 Subject: [PATCH 2/2] hotfix, failed to revert pipline (#3116) --- .github/workflows/deploy-on-label.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/deploy-on-label.yml b/.github/workflows/deploy-on-label.yml index f88e6d34e..8d4720f07 100644 --- a/.github/workflows/deploy-on-label.yml +++ b/.github/workflows/deploy-on-label.yml @@ -71,7 +71,7 @@ jobs: - name: Circle CI Deployment Trigger id: curl-circle-ci - # if: steps.get-pr-state.outputs.STATE == 'success' + if: steps.get-pr-state.outputs.STATE == 'success' uses: promiseofcake/circleci-trigger-action@v1 with: user-token: ${{ secrets.CIRCLE_CI_V2_TOKEN }}