Release/3.1.6 sprint 86 #490
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* added md file for dropping DB * added section for db_backup & merged into README * Update CloudFoundry-DB-Connection.md --------- Co-authored-by: Andrew <[email protected]>
* - ADding t2 schema * - small fixes to naming - added t2.py - Updating var names to match field names * - Added new doc for T2 - Updated model fo rT2 - Added active parse function * - Added T3 schema defs * - Fixing most lint errors * - Updated T3 to multirow record * - Added unit test and data files - updated field names * - Updating var names to conform to DB max var name length * - Updating based on unit test failure * - adding datafile * - Adding unit tests for t2 and t3 * - Fixed field types - Updated test file * - Removing many migrations to consolodate into one migration * Revert "- Removing many migrations to consolodate into one migration" This reverts commit 1455ae4. * - Resolved test issues after merge * - Resolve lint errors * - Merged 1113.2 * - Removed unnecessary file * - Rename model fields - Combined migrations * - fixed spelling in test * - Fixed lint errors * rm commented validators * - Updated schema newlines to be consistent * - Updated field names - Updated tests - Updated migration * - consolodating migrations * - Updated readme and backup script * - Updated parse logic to batch create records * - Fixed lint errors * - Batching record serialization * - Updated parse logic * - Adding big s1 datafile * - fixing lint errors * - Removing test to see if its causing weird failure * - Updating based on comments - Removing big file since circi cant handle it * - fixing error in update method * - fixing error in update * - Removed extraneous seek * - updating ignore to ignore patch files * - Made evaluate_trailer pure/immutable * Revert "- Removing test to see if its causing weird failure" This reverts commit 64b78b7. * - Use custom transactions while parsing - Handle transaction rollback on parse failure/error - Update tests to use transaction logic - Created new decorator to rollback database when assertion error is hit - Making elastic search log level configurable - Added test for inter parse rollback * - updated store logic to one liner - updated decorator to catch all exceptions - updated tests * - removed function - renamed test * - refactored multirecord schema to be a schema manager instead - updated parse functions to use the new layout - updated bulk create to manage batch size * - Name update for clarity * - Fix lint errors * - Changing batch size to avoid crash in circi * - Updated based on review - Updated tests to reflect line number * - Updated based on review/OH - Added extra transaction safety * - Updated view to show latest datafiles - Added admin filter to show newest or all datafile records - Updated indices to allow easier elastic queries * - Updated based on review comments * - Updated based on feedback for preparse error handling - updated tests * - Updated search indices to have parent FK * - Fix lint errors * - Updated submission tests - Moved create_datafile to util * - fix lint errors * - removing frontend filtering * - addding datafile to admin model * Revert "- addding datafile to admin model" This reverts commit 35a6f24. * - Fixed issue where datafile FK wasnt populating - Regenerated migration * - Readding datafile back to admin view now that the error is resolved * - adding datafile back * Revert "- Readding datafile back to admin view now that the error is resolved" This reverts commit 2807425. * - Removed unnecessary fields - Updated dependencies - Updated filter * - Updated document to include required fields * - Moved datafile reference to avoid confusion * - Updating based on comments. - Added base class to keep things a little more DRY * - Refactored parsing to avoid custom transactions by leveraging the FK on the records. Rollbacks are a lot slower now, but hopefully that will happen much less versus successful parses. * - small optimization for the rollback * - Fix lint errors * - Removing commented code * - Installing build essentials * Revert "- Installing build essentials" This reverts commit 782a5f7. * - adding setup tools and wheel * Revert "- adding setup tools and wheel" This reverts commit f529728. * - Updating dependencies to get around pep issue * - Pin factoryboy - fix lint error * - Updating manifest * - Added EncryptedField class - Updated schema's - Updated datafiles since all are encrypted * - Fix lint errors * - Added decryption for ssp * - Making the encrypted check stronger * - Added section two models, schemas, and an initial test * - Allowing fields to be nullable * - add missing field * - Consolodating migrations * - Consolodate migration - Add tests * - Added encrypted field * - Refactored fields and rowschema into seperate files - Updated encrypted field to take decryption function * - fix lint errors * - Fixed file spacing * - fix imports - fix lint errors * - Fix import error * - Fix failing test * - Revert erroneous change * - Updating item numbers * - Fixed item numbers * - Updating to accomodate item number as string * - Removed erroneous update that was causing error - Fixed whitespace in datafiles * - White space error * - Resolved error * - Fixing test * - fix lint errors * - Added field validators for T4/T5 * - Added cat3 validators * - Resolve lint and unit test errors * - small optimization to validator * - Added tests for cat3 validators for T5 record * - Fix lint errors * - updated fields to correct types * - update race/ethnicity * - updated tests post merge - added check in make_validator to just return false if the value is None * - Updated based on review feedback * - Fix lint error * - Updating to the correct types * - Resolve conflicts * - Moving field updates to seperate migration to see if they take place * Revert "- Moving field updates to seperate migration to see if they take place" This reverts commit a3214f2. * - Revert migration 15 to original state - Generate new migration for field alterations * - Merged base branch and updated tests/factories * - Fixed test * - Fixed field names to correspond to model * - Remove duplicate function - Update ssn validator - Removed asserts with large number of parse errors * t7 model, index changes * add t7 to parser * update model mapping test * add s3-s10 * - Remove print statemtnt * - Resolved filter issue that made it seem like records werent being created - Added filter to remaining records * - fix lint errors * - correct form merge conflict * - Added specific validators to avoid duplication - Updated naming of validators * TANF Section 3 Parsing/Validation (#2649) * - Updated to support parsing section 3 data * - Added validators and tests for T6 record * - resolve lint errors * - Quick rename * - Update fields in test * - Fixed conflict * - Fix error from merge conflict * - Updated to create RPT_MONTH_YEAR from CALENDAR_QUARTER - Updated tests * - Added new validators for field change * - Remove debug code * - Genericize TransformField - Update all schemas to use TransformField - Move transforms to seperate file - Fix lint errors * - put kwargs in correct spot * - Very minor change to avoid setting field unnecessarily on all TransformFields * - Updated validator to call out name of field versus the value - Updated tests * - Fix lint * - Calling super to avoid duplicate code * - Added validators for transformed field * - Updating based on merge * - Fixed remaining merge conflicts * - Fix lint errors * - update error messages based on validator updates * - Corrected validator * - Resolved bug causing file to think it was encrytped after multiple submissions and changing the encryption header in between submissions * - UPdated migration - Updated test * - Added transform fields - UPdated migrations * - Stratified T7 to one record per month per section indicator and stratum. * - Updated tests * - Fix test * - Add T7 factory * - Fix lint error * - Remove test class until cat three validators exist * - fix lint errors * - Fixing colliding migration * - Generating schemas instead of defining * - Fix lint errors * - Updated based on review feedback * rm 17/18 * remake migration 18 * fix merge error * fix merge error * fix merge error * labels are stuck * rm * dont compute aggregates for section 3/4 * move save * Update tdrs-backend/tdpservice/parsers/test/factories.py Co-authored-by: Alex P. <[email protected]> * enhance t7 modeladmin * fix rpt_month_year off by 1 * lint * differentiate item 6a/6b/67 in t7 records --------- Co-authored-by: elipe17 <[email protected]> Co-authored-by: Alex P <[email protected]> Co-authored-by: Eric Lipe <[email protected]>
* saving state real quick * finishing merge with latest * Missed old test script * Added new test, more cleanup * Updating unit tests in DFS, preparing for 1610 * Merging in Jan's 1610 code for parserError useful-ness * Revert "Merging in Jan's 1610 code for parserError useful-ness" This reverts commit c5796da. * update to test to use dfs fixture * saving state before new 1610 merge * Resolving merge conflicts with 1610. * Linting changes and comparing to 1610 * Some unit test linting but inherited 1610 issues * Re-ordering job to run tests vs lint first. * Updates to linting and unit tests. * Fixing linting. * Update tdrs-backend/setup.cfg * updates per PR. * Excluding trailers for rejection * VSCode merge resolution is garbage. * Fixing precheck for not implemented types * Updating to error-handle not implemented schema types * - Updated view to show latest datafiles - Added admin filter to show newest or all datafile records - Updated indices to allow easier elastic queries * - Updated search indices to have parent FK * - Fix lint errors * - Updated submission tests - Moved create_datafile to util * - fix lint errors * - removing frontend filtering * - addding datafile to admin model * Revert "- addding datafile to admin model" This reverts commit 35a6f24. * - Fixed issue where datafile FK wasnt populating - Regenerated migration * - Readding datafile back to admin view now that the error is resolved * - adding datafile back * Revert "- Readding datafile back to admin view now that the error is resolved" This reverts commit 2807425. * - Removed unnecessary fields - Updated dependencies - Updated filter * - Updated document to include required fields * - Fixed failing test * add adminUrl to deployment cypress overrides * Adding "beta" banners to relevant error report sections (#2522) * Update views.py * Update views.py * Update SubmissionHistory.jsx * Update SubmissionHistory.test.js * Apply suggestions from code review Co-authored-by: Miles Reiter <[email protected]> * lint fixes --------- Co-authored-by: Miles Reiter <[email protected]> Co-authored-by: Alex P <[email protected]> Co-authored-by: andrew-jameson <[email protected]> * Create sprint-73-summary.md (#2565) * hotfix for large file sizes (#2542) * hotfix for large file sizes * apply timeouts/req limits to dev * filter identity pages from scan * IGNORE sql injection --------- Co-authored-by: Jan Timpe <[email protected]> Co-authored-by: mo sohani <[email protected]> Co-authored-by: Alex P <[email protected]> * updating validation error language * accidentally included coding challenge * rm comments * 2550 deactivation email link (#2557) * - updated nginx buildpack * - specifying different nginx version * - Updating changelog * - added script to update certain apps in cf - added workflow for each environment in circi * - fixed base config * - fixing jobs * - Updated based on feedback in OH * - Updating defaults * - Removing defaults * - Fixing indent * - Adding params to config * test * test * - updating work dir * - Adding checkout * - adding cf check * - logging into cf * - update cf check to install required binary * - removing unnecessary switch * - Forcing plugin installation * - test installing plugin from script also * - Adding url to email * - test code for sandbox * - using my email * Revert "Merge branch 'update-cf-os' into 2551-deactivation-email-link" This reverts commit e963b9d, reversing changes made to cc9cf81. * Revert "- using my email" This reverts commit cc9cf81. * Revert "- test code for sandbox" This reverts commit 0603774. --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]> * Update README.md (#2577) Add ATO Co-authored-by: Andrew <[email protected]> * Create 2023, Spring - Testing CSV & Excel-based error reports.md * Update README.md * Updating deliverable links (#2584) * User viewset not returning/duplicating users (#2573) * - Fixed issue not allowing pagination to work locally with nginx - Added ordering to user field to fix duplicates issue * - fix lint error * - Removing ID check since we cannot guarantee that the uuid that is generated per test run will be lexigraphically consistent --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]> * Update cf os (#2523) * - updated nginx buildpack * - specifying different nginx version * - Updating changelog * - added script to update certain apps in cf - added workflow for each environment in circi * - fixed base config * - fixing jobs * - Updated based on feedback in OH * - Updating defaults * - Removing defaults * - Fixing indent * - Adding params to config * test * test * - updating work dir * - Adding checkout * - adding cf check * - logging into cf * - update cf check to install required binary * - removing unnecessary switch * - Forcing plugin installation * - test installing plugin from script also * - Adding new dependencies * - adding package * - fixing broken install * - fixing libs * - using correct command * - gettign correct version of libc * - trying to upgrade libs * - testing * - Updated README and script * Revert "- Updated README and script" This reverts commit 92697b3. * - Removed unnecessary circi stuff - Removed script - Updated docs to callout updating secondary apps * - Correct spelling error --------- Co-authored-by: Andrew <[email protected]> * Item Number Mismatch (#2578) * - Updated schemas and models to reflect correct item numbers of fields * - Revert migration * - Updated header/trailer item numbers * - Fixed item numbers off by one errors --------- Co-authored-by: Andrew <[email protected]> * pipeline filtering (#2538) * pipeline changes that filter based on paths and branches. circle ci tracks specified branches in order to keep current functionality on HHS side. * updated syntax to be in line with build-all.yml * removed comma * WIP build flow docs * added Architecture Decision Record for the change to pipeline workflows * corrected file type of doc to .md --------- Co-authored-by: George Hudson <[email protected]> Co-authored-by: Andrew <[email protected]> * Hotfix Devops/2457 path filtering for documentation (#2597) * pipeline changes that filter based on paths and branches. circle ci tracks specified branches in order to keep current functionality on HHS side. * updated syntax to be in line with build-all.yml * removed comma * WIP build flow docs * added Architecture Decision Record for the change to pipeline workflows * corrected file type of doc to .md * build and test all on PRs even for documentation --------- Co-authored-by: George Hudson <[email protected]> * Create sprint-74-summary.md (#2596) Co-authored-by: Andrew <[email protected]> * added URL filters (#2580) * added URL filters * allow github to trigger owasp and label deploys (#2601) Co-authored-by: George Hudson <[email protected]> --------- Co-authored-by: Andrew <[email protected]> Co-authored-by: George Hudson <[email protected]> Co-authored-by: George Hudson <[email protected]> * Create sprint-75-summary.md (#2608) * Create sprint-76-summary.md (#2609) Co-authored-by: Andrew <[email protected]> * - Resolved failing tests * - Corrected merge thrash * - Using randbits to generate pk to get around confilcting sequence pks * Revert "- Using randbits to generate pk to get around confilcting sequence pks" This reverts commit ac9b065. * - Updating region in fixture instead of factory - letting django handle transaction for test * - Moved datafile reference to avoid confusion * pushing up incomplete codebase * Other unit tests now have passed w/ good error handling * Working tests, need to get setup for case aggregates populating via DB * - Updated queries - Added helper function - Need to merge in 2579 for queries to work * minor improvement to month2int * - Fixing most merge errors * - Fixing functions * - Updated queries based on generic relation * - Updated queries to count by case number instead of record number * - Added route - Updated task to create dfs * - updated tests to include dfs * Cleaning up most comments that are no longer necessary and fixed lint issues. * making minor updates, still broken tests. * updating pipfile.lock and rebuild image resolved test issues * Reorganizing tests, still failing in test_parse.py * deleted summary file, split into other test scripts. * Fixed missing self reference. * Linting fixes. * Found reference failure in deployed env. * Removing extra returns for missing record type. * lint fix * Addressed invocation of datafile for failing test * lint update for whitespace * Intermediary commit, broken test * add reverse relation * frontend summary status * new assignemnts in util * - updated rejected query to correctly count objs * - Fixing most tests * - Fixed user error. Swapped numbers by accident. * - make region None to avoid PK collision * - Fix lint errors * - Updating to avoid warning * add acceptance icon * add reverse relationship to summary model * vscode merge conflict resolution (#2623) * auto-create the external network * didn't stage commit properly * checking diffs, matching 1613.2 * doesn't work in pipeline. must be cached local * re-commenting in unit test * lint failures fixed --------- Co-authored-by: andrew-jameson <[email protected]> * add a pending status * add acceptance status tests * update warning icon * rm old assert * update rejected icon * url change per me, want pipeline to run e2e * Upgraded to querysets, fix PR comments, PE str * missing : not caught locally * Feat/1613 merge 2 (#2650) * Create sprint-78-summary.md (#2645) * Missing/unsaved parser_error for record_type * removing redundant tests * Hopefully resolved on unit tests and lint --------- Co-authored-by: Smithh-Co <[email protected]> Co-authored-by: andrew-jameson <[email protected]> * icontains * tests * Changing dict structure per 1612. * fixed tests and lint issues, parse is too complex * schema_manager replaces schema check * Saving state prior to merge-conflict. * Adopting latest manager, removing old error style. * Commented out t6 line during Office hours * minor reference update * Acclimating to schemaManager * lint-fix isinstance * syntax mistake with isinstance * Apply suggestions from code review * reverting search_index merge artifacts. * adjusting for removing unused "get-schema()" * whitespace lint * Feedback from Jan * Ensuring tests run/work. * Ensure we have leading zero in rptmonthyear. * Minor lint fix for exception logging * fix migration * add backend partially accepted status * add frontend partially accepted status * add partially accepted test * rm accidental commit * rm unused variable * resolving merge conflict problems * fixing tests from merge conflicts. * dumb lint fix * Update tdrs-backend/tdpservice/parsers/migrations/0007_alter_datafilesummary_datafile.py * fix merge errors * fix name attr not populated for preparsing errors * include preparsing errors in row_precheck_errors status * make header precheck errors force rejected status * lint msg --------- Co-authored-by: Andrew Jameson <[email protected]> Co-authored-by: Andrew <[email protected]> Co-authored-by: andrew-jameson <[email protected]> Co-authored-by: elipe17 <[email protected]> Co-authored-by: Miles Reiter <[email protected]> Co-authored-by: Alex P <[email protected]> Co-authored-by: Smithh-Co <[email protected]> Co-authored-by: mo sohani <[email protected]> Co-authored-by: Eric Lipe <[email protected]> Co-authored-by: Lauren Frohlich <[email protected]> Co-authored-by: Miles Reiter <[email protected]> Co-authored-by: George Hudson <[email protected]> Co-authored-by: George Hudson <[email protected]> Co-authored-by: raftmsohani <[email protected]>
* saving state real quick * finishing merge with latest * Missed old test script * Added new test, more cleanup * Updating unit tests in DFS, preparing for 1610 * Merging in Jan's 1610 code for parserError useful-ness * Revert "Merging in Jan's 1610 code for parserError useful-ness" This reverts commit c5796da. * update to test to use dfs fixture * saving state before new 1610 merge * Resolving merge conflicts with 1610. * Linting changes and comparing to 1610 * Some unit test linting but inherited 1610 issues * Re-ordering job to run tests vs lint first. * Updates to linting and unit tests. * Fixing linting. * Update tdrs-backend/setup.cfg * updates per PR. * Excluding trailers for rejection * VSCode merge resolution is garbage. * Fixing precheck for not implemented types * Updating to error-handle not implemented schema types * - Updated view to show latest datafiles - Added admin filter to show newest or all datafile records - Updated indices to allow easier elastic queries * - Updated search indices to have parent FK * - Fix lint errors * - Updated submission tests - Moved create_datafile to util * - fix lint errors * - removing frontend filtering * - addding datafile to admin model * Revert "- addding datafile to admin model" This reverts commit 35a6f24. * - Fixed issue where datafile FK wasnt populating - Regenerated migration * - Readding datafile back to admin view now that the error is resolved * - adding datafile back * Revert "- Readding datafile back to admin view now that the error is resolved" This reverts commit 2807425. * - Removed unnecessary fields - Updated dependencies - Updated filter * - Updated document to include required fields * - Fixed failing test * add adminUrl to deployment cypress overrides * Adding "beta" banners to relevant error report sections (#2522) * Update views.py * Update views.py * Update SubmissionHistory.jsx * Update SubmissionHistory.test.js * Apply suggestions from code review Co-authored-by: Miles Reiter <[email protected]> * lint fixes --------- Co-authored-by: Miles Reiter <[email protected]> Co-authored-by: Alex P <[email protected]> Co-authored-by: andrew-jameson <[email protected]> * Create sprint-73-summary.md (#2565) * hotfix for large file sizes (#2542) * hotfix for large file sizes * apply timeouts/req limits to dev * filter identity pages from scan * IGNORE sql injection --------- Co-authored-by: Jan Timpe <[email protected]> Co-authored-by: mo sohani <[email protected]> Co-authored-by: Alex P <[email protected]> * updating validation error language * accidentally included coding challenge * rm comments * 2550 deactivation email link (#2557) * - updated nginx buildpack * - specifying different nginx version * - Updating changelog * - added script to update certain apps in cf - added workflow for each environment in circi * - fixed base config * - fixing jobs * - Updated based on feedback in OH * - Updating defaults * - Removing defaults * - Fixing indent * - Adding params to config * test * test * - updating work dir * - Adding checkout * - adding cf check * - logging into cf * - update cf check to install required binary * - removing unnecessary switch * - Forcing plugin installation * - test installing plugin from script also * - Adding url to email * - test code for sandbox * - using my email * Revert "Merge branch 'update-cf-os' into 2551-deactivation-email-link" This reverts commit e963b9d, reversing changes made to cc9cf81. * Revert "- using my email" This reverts commit cc9cf81. * Revert "- test code for sandbox" This reverts commit 0603774. --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]> * Update README.md (#2577) Add ATO Co-authored-by: Andrew <[email protected]> * Create 2023, Spring - Testing CSV & Excel-based error reports.md * Update README.md * Updating deliverable links (#2584) * User viewset not returning/duplicating users (#2573) * - Fixed issue not allowing pagination to work locally with nginx - Added ordering to user field to fix duplicates issue * - fix lint error * - Removing ID check since we cannot guarantee that the uuid that is generated per test run will be lexigraphically consistent --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]> * Update cf os (#2523) * - updated nginx buildpack * - specifying different nginx version * - Updating changelog * - added script to update certain apps in cf - added workflow for each environment in circi * - fixed base config * - fixing jobs * - Updated based on feedback in OH * - Updating defaults * - Removing defaults * - Fixing indent * - Adding params to config * test * test * - updating work dir * - Adding checkout * - adding cf check * - logging into cf * - update cf check to install required binary * - removing unnecessary switch * - Forcing plugin installation * - test installing plugin from script also * - Adding new dependencies * - adding package * - fixing broken install * - fixing libs * - using correct command * - gettign correct version of libc * - trying to upgrade libs * - testing * - Updated README and script * Revert "- Updated README and script" This reverts commit 92697b3. * - Removed unnecessary circi stuff - Removed script - Updated docs to callout updating secondary apps * - Correct spelling error --------- Co-authored-by: Andrew <[email protected]> * Item Number Mismatch (#2578) * - Updated schemas and models to reflect correct item numbers of fields * - Revert migration * - Updated header/trailer item numbers * - Fixed item numbers off by one errors --------- Co-authored-by: Andrew <[email protected]> * pipeline filtering (#2538) * pipeline changes that filter based on paths and branches. circle ci tracks specified branches in order to keep current functionality on HHS side. * updated syntax to be in line with build-all.yml * removed comma * WIP build flow docs * added Architecture Decision Record for the change to pipeline workflows * corrected file type of doc to .md --------- Co-authored-by: George Hudson <[email protected]> Co-authored-by: Andrew <[email protected]> * Hotfix Devops/2457 path filtering for documentation (#2597) * pipeline changes that filter based on paths and branches. circle ci tracks specified branches in order to keep current functionality on HHS side. * updated syntax to be in line with build-all.yml * removed comma * WIP build flow docs * added Architecture Decision Record for the change to pipeline workflows * corrected file type of doc to .md * build and test all on PRs even for documentation --------- Co-authored-by: George Hudson <[email protected]> * Create sprint-74-summary.md (#2596) Co-authored-by: Andrew <[email protected]> * added URL filters (#2580) * added URL filters * allow github to trigger owasp and label deploys (#2601) Co-authored-by: George Hudson <[email protected]> --------- Co-authored-by: Andrew <[email protected]> Co-authored-by: George Hudson <[email protected]> Co-authored-by: George Hudson <[email protected]> * Create sprint-75-summary.md (#2608) * Create sprint-76-summary.md (#2609) Co-authored-by: Andrew <[email protected]> * - Resolved failing tests * - Corrected merge thrash * - Using randbits to generate pk to get around confilcting sequence pks * Revert "- Using randbits to generate pk to get around confilcting sequence pks" This reverts commit ac9b065. * - Updating region in fixture instead of factory - letting django handle transaction for test * - Moved datafile reference to avoid confusion * pushing up incomplete codebase * Other unit tests now have passed w/ good error handling * Working tests, need to get setup for case aggregates populating via DB * - Updated queries - Added helper function - Need to merge in 2579 for queries to work * minor improvement to month2int * - Fixing most merge errors * - Fixing functions * - Updated queries based on generic relation * - Updated queries to count by case number instead of record number * - Added route - Updated task to create dfs * - updated tests to include dfs * Cleaning up most comments that are no longer necessary and fixed lint issues. * making minor updates, still broken tests. * updating pipfile.lock and rebuild image resolved test issues * Reorganizing tests, still failing in test_parse.py * deleted summary file, split into other test scripts. * Fixed missing self reference. * Linting fixes. * Found reference failure in deployed env. * Removing extra returns for missing record type. * lint fix * Addressed invocation of datafile for failing test * lint update for whitespace * Intermediary commit, broken test * add reverse relation * frontend summary status * new assignemnts in util * - updated rejected query to correctly count objs * - Fixing most tests * - Fixed user error. Swapped numbers by accident. * - make region None to avoid PK collision * - Fix lint errors * - Updating to avoid warning * add acceptance icon * add reverse relationship to summary model * vscode merge conflict resolution (#2623) * auto-create the external network * didn't stage commit properly * checking diffs, matching 1613.2 * doesn't work in pipeline. must be cached local * re-commenting in unit test * lint failures fixed --------- Co-authored-by: andrew-jameson <[email protected]> * add a pending status * add acceptance status tests * update warning icon * rm old assert * update rejected icon * add case aggregates to submission history * url change per me, want pipeline to run e2e * update case aggregate columns * update col names * Upgraded to querysets, fix PR comments, PE str * missing : not caught locally * Feat/1613 merge 2 (#2650) * Create sprint-78-summary.md (#2645) * Missing/unsaved parser_error for record_type * removing redundant tests * Hopefully resolved on unit tests and lint --------- Co-authored-by: Smithh-Co <[email protected]> Co-authored-by: andrew-jameson <[email protected]> * icontains * tests * Changing dict structure per 1612. * fixed tests and lint issues, parse is too complex * schema_manager replaces schema check * Saving state prior to merge-conflict. * Adopting latest manager, removing old error style. * Commented out t6 line during Office hours * minor reference update * Acclimating to schemaManager * lint-fix isinstance * syntax mistake with isinstance * Apply suggestions from code review * reverting search_index merge artifacts. * adjusting for removing unused "get-schema()" * whitespace lint * Feedback from Jan * Ensuring tests run/work. * Ensure we have leading zero in rptmonthyear. * Minor lint fix for exception logging * fix migration * add backend partially accepted status * add frontend partially accepted status * add partially accepted test * rm accidental commit * rm unused variable * conditional submission history * fix test * resolving merge conflict problems * fixing tests from merge conflicts. * - fixed migration reference * dumb lint fix * - Fixed render error * Adding section * - Resolved type mismatches causing render failures * - Adding branch coverage * - Fix lint errors * - Fix lint errors * - Consolodated tetss * reducing line length for lint * - updated maxWidth to be percentage * Moving DFS migration into it's own file to avoid conflicts. * rename migrations * left align error report text * github got stuck :( * rm comment * rm margin, add text for no error report * fix tests * fix ssp case aggregates display --------- Co-authored-by: Andrew Jameson <[email protected]> Co-authored-by: Andrew <[email protected]> Co-authored-by: andrew-jameson <[email protected]> Co-authored-by: elipe17 <[email protected]> Co-authored-by: Miles Reiter <[email protected]> Co-authored-by: Alex P <[email protected]> Co-authored-by: Smithh-Co <[email protected]> Co-authored-by: mo sohani <[email protected]> Co-authored-by: Eric Lipe <[email protected]> Co-authored-by: Lauren Frohlich <[email protected]> Co-authored-by: Miles Reiter <[email protected]> Co-authored-by: George Hudson <[email protected]> Co-authored-by: George Hudson <[email protected]> Co-authored-by: raftmsohani <[email protected]>
* - small grammar fix * - Fix lint suggestions * - updated message * - Fixed lint errors * - Added correct extension to datafiles without one * - Adding cherry picks for file extension error handling * - Updated regex * - updating to keep file in dropbox in event of error to help user correct their mistake. * - Fix icon rendering incorrectly * update test file extensions * - making timeout longer * - Resolved issue causing test failure - resetting timeout * - passing param --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Miles Reiter <[email protected]> Co-authored-by: Jan Timpe <[email protected]>
Co-authored-by: Andrew <[email protected]>
* saving state real quick * finishing merge with latest * Missed old test script * Added new test, more cleanup * Updating unit tests in DFS, preparing for 1610 * Merging in Jan's 1610 code for parserError useful-ness * Revert "Merging in Jan's 1610 code for parserError useful-ness" This reverts commit c5796da. * update to test to use dfs fixture * saving state before new 1610 merge * Resolving merge conflicts with 1610. * Linting changes and comparing to 1610 * Some unit test linting but inherited 1610 issues * Re-ordering job to run tests vs lint first. * Updates to linting and unit tests. * Fixing linting. * Update tdrs-backend/setup.cfg * updates per PR. * Excluding trailers for rejection * VSCode merge resolution is garbage. * Fixing precheck for not implemented types * Updating to error-handle not implemented schema types * - Updated view to show latest datafiles - Added admin filter to show newest or all datafile records - Updated indices to allow easier elastic queries * - Updated search indices to have parent FK * - Fix lint errors * - Updated submission tests - Moved create_datafile to util * - fix lint errors * - removing frontend filtering * - addding datafile to admin model * Revert "- addding datafile to admin model" This reverts commit 35a6f24. * - Fixed issue where datafile FK wasnt populating - Regenerated migration * - Readding datafile back to admin view now that the error is resolved * - adding datafile back * Revert "- Readding datafile back to admin view now that the error is resolved" This reverts commit 2807425. * - Removed unnecessary fields - Updated dependencies - Updated filter * - Updated document to include required fields * - Fixed failing test * add adminUrl to deployment cypress overrides * Adding "beta" banners to relevant error report sections (#2522) * Update views.py * Update views.py * Update SubmissionHistory.jsx * Update SubmissionHistory.test.js * Apply suggestions from code review Co-authored-by: Miles Reiter <[email protected]> * lint fixes --------- Co-authored-by: Miles Reiter <[email protected]> Co-authored-by: Alex P <[email protected]> Co-authored-by: andrew-jameson <[email protected]> * Create sprint-73-summary.md (#2565) * hotfix for large file sizes (#2542) * hotfix for large file sizes * apply timeouts/req limits to dev * filter identity pages from scan * IGNORE sql injection --------- Co-authored-by: Jan Timpe <[email protected]> Co-authored-by: mo sohani <[email protected]> Co-authored-by: Alex P <[email protected]> * updating validation error language * accidentally included coding challenge * rm comments * 2550 deactivation email link (#2557) * - updated nginx buildpack * - specifying different nginx version * - Updating changelog * - added script to update certain apps in cf - added workflow for each environment in circi * - fixed base config * - fixing jobs * - Updated based on feedback in OH * - Updating defaults * - Removing defaults * - Fixing indent * - Adding params to config * test * test * - updating work dir * - Adding checkout * - adding cf check * - logging into cf * - update cf check to install required binary * - removing unnecessary switch * - Forcing plugin installation * - test installing plugin from script also * - Adding url to email * - test code for sandbox * - using my email * Revert "Merge branch 'update-cf-os' into 2551-deactivation-email-link" This reverts commit e963b9d, reversing changes made to cc9cf81. * Revert "- using my email" This reverts commit cc9cf81. * Revert "- test code for sandbox" This reverts commit 0603774. --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]> * Update README.md (#2577) Add ATO Co-authored-by: Andrew <[email protected]> * Create 2023, Spring - Testing CSV & Excel-based error reports.md * Update README.md * Updating deliverable links (#2584) * User viewset not returning/duplicating users (#2573) * - Fixed issue not allowing pagination to work locally with nginx - Added ordering to user field to fix duplicates issue * - fix lint error * - Removing ID check since we cannot guarantee that the uuid that is generated per test run will be lexigraphically consistent --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]> * Update cf os (#2523) * - updated nginx buildpack * - specifying different nginx version * - Updating changelog * - added script to update certain apps in cf - added workflow for each environment in circi * - fixed base config * - fixing jobs * - Updated based on feedback in OH * - Updating defaults * - Removing defaults * - Fixing indent * - Adding params to config * test * test * - updating work dir * - Adding checkout * - adding cf check * - logging into cf * - update cf check to install required binary * - removing unnecessary switch * - Forcing plugin installation * - test installing plugin from script also * - Adding new dependencies * - adding package * - fixing broken install * - fixing libs * - using correct command * - gettign correct version of libc * - trying to upgrade libs * - testing * - Updated README and script * Revert "- Updated README and script" This reverts commit 92697b3. * - Removed unnecessary circi stuff - Removed script - Updated docs to callout updating secondary apps * - Correct spelling error --------- Co-authored-by: Andrew <[email protected]> * Item Number Mismatch (#2578) * - Updated schemas and models to reflect correct item numbers of fields * - Revert migration * - Updated header/trailer item numbers * - Fixed item numbers off by one errors --------- Co-authored-by: Andrew <[email protected]> * pipeline filtering (#2538) * pipeline changes that filter based on paths and branches. circle ci tracks specified branches in order to keep current functionality on HHS side. * updated syntax to be in line with build-all.yml * removed comma * WIP build flow docs * added Architecture Decision Record for the change to pipeline workflows * corrected file type of doc to .md --------- Co-authored-by: George Hudson <[email protected]> Co-authored-by: Andrew <[email protected]> * Hotfix Devops/2457 path filtering for documentation (#2597) * pipeline changes that filter based on paths and branches. circle ci tracks specified branches in order to keep current functionality on HHS side. * updated syntax to be in line with build-all.yml * removed comma * WIP build flow docs * added Architecture Decision Record for the change to pipeline workflows * corrected file type of doc to .md * build and test all on PRs even for documentation --------- Co-authored-by: George Hudson <[email protected]> * Create sprint-74-summary.md (#2596) Co-authored-by: Andrew <[email protected]> * added URL filters (#2580) * added URL filters * allow github to trigger owasp and label deploys (#2601) Co-authored-by: George Hudson <[email protected]> --------- Co-authored-by: Andrew <[email protected]> Co-authored-by: George Hudson <[email protected]> Co-authored-by: George Hudson <[email protected]> * Create sprint-75-summary.md (#2608) * Create sprint-76-summary.md (#2609) Co-authored-by: Andrew <[email protected]> * - Resolved failing tests * - Corrected merge thrash * - Using randbits to generate pk to get around confilcting sequence pks * Revert "- Using randbits to generate pk to get around confilcting sequence pks" This reverts commit ac9b065. * - Updating region in fixture instead of factory - letting django handle transaction for test * - Moved datafile reference to avoid confusion * pushing up incomplete codebase * Other unit tests now have passed w/ good error handling * Working tests, need to get setup for case aggregates populating via DB * - Updated queries - Added helper function - Need to merge in 2579 for queries to work * minor improvement to month2int * - Fixing most merge errors * - Fixing functions * - Updated queries based on generic relation * - Updated queries to count by case number instead of record number * - Added route - Updated task to create dfs * - updated tests to include dfs * Cleaning up most comments that are no longer necessary and fixed lint issues. * making minor updates, still broken tests. * updating pipfile.lock and rebuild image resolved test issues * Reorganizing tests, still failing in test_parse.py * deleted summary file, split into other test scripts. * Fixed missing self reference. * Linting fixes. * Found reference failure in deployed env. * Removing extra returns for missing record type. * lint fix * Addressed invocation of datafile for failing test * lint update for whitespace * Intermediary commit, broken test * new assignemnts in util * - updated rejected query to correctly count objs * - Fixing most tests * - Fixed user error. Swapped numbers by accident. * - make region None to avoid PK collision * - Fix lint errors * - Updating to avoid warning * vscode merge conflict resolution (#2623) * auto-create the external network * didn't stage commit properly * checking diffs, matching 1613.2 * doesn't work in pipeline. must be cached local * re-commenting in unit test * lint failures fixed --------- Co-authored-by: andrew-jameson <[email protected]> * url change per me, want pipeline to run e2e * Upgraded to querysets, fix PR comments, PE str * missing : not caught locally * Feat/1613 merge 2 (#2650) * Create sprint-78-summary.md (#2645) * Missing/unsaved parser_error for record_type * removing redundant tests * Hopefully resolved on unit tests and lint --------- Co-authored-by: Smithh-Co <[email protected]> Co-authored-by: andrew-jameson <[email protected]> * icontains * tests * Changing dict structure per 1612. * fixed tests and lint issues, parse is too complex * schema_manager replaces schema check * Saving state prior to merge-conflict. * Adopting latest manager, removing old error style. * Commented out t6 line during Office hours * minor reference update * Acclimating to schemaManager * lint-fix isinstance * syntax mistake with isinstance * Apply suggestions from code review * reverting search_index merge artifacts. * adjusting for removing unused "get-schema()" * whitespace lint * Feedback from Jan * Ensuring tests run/work. * Ensure we have leading zero in rptmonthyear. * Minor lint fix for exception logging * resolving merge conflict problems * added the filters for datafile progtype * added url to admin filters * added datafile summary link to admin page * filtering parser errors * linting * Test data_file admin page * corrected datafile id for test file * linting * Added deplay to cypress test * changed datafile summary filter * Changed error report link * changed filter to reflect related name --------- Co-authored-by: Andrew Jameson <[email protected]> Co-authored-by: Andrew <[email protected]> Co-authored-by: andrew-jameson <[email protected]> Co-authored-by: elipe17 <[email protected]> Co-authored-by: Jan Timpe <[email protected]> Co-authored-by: Miles Reiter <[email protected]> Co-authored-by: Alex P <[email protected]> Co-authored-by: Smithh-Co <[email protected]> Co-authored-by: Eric Lipe <[email protected]> Co-authored-by: Lauren Frohlich <[email protected]> Co-authored-by: Miles Reiter <[email protected]> Co-authored-by: George Hudson <[email protected]> Co-authored-by: George Hudson <[email protected]>
renaming file Co-authored-by: Andrew <[email protected]>
* removed clamav from being deployed unless in production. Pointed AV SCAN enpoint to tanf-prod clam av scanner only * added network policy to connect to prod clamAV * use prod clam av in cloudgov.py too * fixing 'f-string is missing placeholders' now that we are hardcoding prod clamav * removed quotes from add-network-policy command when interpolation not needed * removed add-network-policy from deploy-backend.sh and added documentation for how to do manually. * add nginx router and manifest * cleaned up * Update README for CLAMAV * updated README file with deployment commands * added network policy for backend to clamav router * updated boundry diagram * updated boundry diagram * Removed AV_SCAN url from cloud.gov settings file * update setting environment variable * refactored circle ci for static clam av vars since only one server now. updated docs. * needs env_var_name, not string for login-cloud-dot-gov * revert back to env_var_name type for downstream login_cloud_dot_gov command * added prod prefix back in * readd AV_SCAN_URL * changed the inline comment in common.py settings * README file rewrite * added note * readded clam av nginx router url to manifest and setting network policies in the deploy-backend.sh script * updated for using backend to set ENV for ClamAV URL. * getting rid of spaces on empty last line of manifest * Remove AV_SCAN_URL from deploy-backend * updated README for clarification for clam-av-nginx-router naming * moving setting vars for clamav server to above cf push --------- Co-authored-by: George Hudson <[email protected]> Co-authored-by: Mo Sohani <[email protected]> Co-authored-by: raftmsohani <[email protected]> Co-authored-by: Andrew <[email protected]>
* Update cloudgov.py * hijack develop workflow to test deployments. * syntax typo. * I can't believe this typo. * linting whitespace * removed extra space * Adding in staging jwt_key due to recent deployment failure. * Removing self-reference branch filter for mergability --------- Co-authored-by: andrew-jameson <[email protected]> Co-authored-by: George Hudson <[email protected]>
Sprint summary Co-authored-by: Andrew <[email protected]>
* Create sprint-84-summary.md * Update sprint-84-summary.md --------- Co-authored-by: Andrew <[email protected]>
* Update cloudgov.py * hijack develop workflow to test deployments. * syntax typo. * I can't believe this typo. * linting whitespace * removed extra space * Adding in staging jwt_key due to recent deployment failure. * Removing self-reference branch filter for mergability * Updated elif logic based on CI failures --------- Co-authored-by: andrew-jameson <[email protected]> Co-authored-by: George Hudson <[email protected]>
* - small grammar fix * - Fix lint suggestions * - updated message * - Fixed lint errors * - Added correct extension to datafiles without one * - Adding cherry picks for file extension error handling * - Updated regex * - updating to keep file in dropbox in event of error to help user correct their mistake. * - Fix icon rendering incorrectly * update test file extensions * - making timeout longer * - Resolved issue causing test failure - resetting timeout * - passing param * - updated nginx conf --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Miles Reiter <[email protected]> Co-authored-by: Jan Timpe <[email protected]> Co-authored-by: Andrew <[email protected]>
* - Added support to fields that can allow blanks as valid fields - Added new tests - Updated make validator to handle type errors * - Updating blank fields to not be required * - Fix lint errors - Add datafile * - updating fields to be required until guidance comes out on what is not reqruired * - ADding None check * - Error checking * - Updated to run field validators even if field is empty but is allowed to be - removed unnecessary validator - updated schemas - * - Switching back to numIsBlank * -Adding back numIsBlank * - adding isNone since empty string fields are also None * - Added all missing fields to test file - Updated incorrect fields and catch validator error * - Updating test * - Fix lint errors * - Re-added missing fields - Updating tests * - Updated tests * - Fixing errors from merge * - Removed extra flag - Updated logic on when to run field validators - Updated schemas * - Fixed failing tests * - Fix lint * - Removing unnused validator --------- Co-authored-by: Andrew <[email protected]>
* - Updated request to execute 5 times before throwing error * - Adding error modal when stt list is empty * - Not defaulting to show error modal * - Updated to use redux state instead of component state for request counting * - Simplifying if block * - Updated to use our pre-built modal instead of re-inventing the wheel
* cat 2 validation * ssp s1 cat 3 validators * debug ssp section errors * change string validators to number validators * add df to case agg query * update ssp s1 test * update ssp tests * fix lint * update test * rm prints * update validator * Update tdrs-backend/tdpservice/parsers/schema_defs/ssp/m1.py Co-authored-by: Alex P. <[email protected]> * fix the validator i broke and update tests * udpate validator (again) * lint * Update tdrs-backend/tdpservice/parsers/schema_defs/ssp/m1.py Co-authored-by: Alex P. <[email protected]> * lint * rm unneeded cat3 validator * update docstring * rm unused validator * update m2 validator * fix education level validation for m3 * add required=False to allow blanks * fix other education level m3 validator * rm schema comments * fix tests --------- Co-authored-by: Alex P. <[email protected]> Co-authored-by: Andrew <[email protected]>
) Co-authored-by: George Hudson <[email protected]> Co-authored-by: Andrew <[email protected]>
* added documentation for our Nexus repository * updated documentation to include VM configuration, nexus container setup, and new proxy and group repo usage. * updated reamde with qol and grammatical fixes --------- Co-authored-by: George Hudson <[email protected]> Co-authored-by: Andrew <[email protected]>
Create sprint-85-summary.md
* ssp m6 boilerplate * update ssp m6 field names * update tests * rm logging * rm logging * fix tests --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]>
* - Added support for parsing and validating ssp section 4 * - fix lint errors * - Updated permissions test * - Fixed item number mismatch * - Remove generic relation - Add missing test - Re-made migration * - Fix lint errors * - Pre-calculating item numbers * - Formatting with Black * - Adding missing postparsing validator * - Deconflicting migrations * - Deconflicting migrations * - Fixing test --------- Co-authored-by: Jan Timpe <[email protected]> Co-authored-by: Alex P <[email protected]>
* - Added m4 and m5 - Added cat 2 validators * - Quick correction to preparse validator * - Updated item numbers - Added cat3 validators * - Adding datafile * - updated datafile to allow correct parsing of m4 records - added tests * - Fix lint errors * - Update permissions test * - Fixed failing unit test * - Adding order_by to keep consistent results * - Fixed obj ordering issue - Skipping super big file * - Added missing validator * - Updated import paths * - Updated DOB fields to make a little more sense * - Pre-emptively marking required false to prepare for merge * - Fix lint * - Formatting with Black * update m4/m5 validators * isInLimits -> matches (duh) * - Deconflicting migrations * - Fixing test * - Adding missing fixture * - Resolving migration name conflicts * - Fixing dependency * - Keeping consistent dependencies * - Linearizing deps --------- Co-authored-by: Jan Timpe <[email protected]> Co-authored-by: jtimpe <[email protected]>
Co-authored-by: Andrew <[email protected]>
ADPennington
approved these changes
Nov 24, 2023
There was a problem hiding this comment.
@lfrohlich @ttran-hub this release includes the following changes:
- features:
- TANF Section 1 space-filling allowed
- TANF Section 4 parsing/validation
- SSP Sections 1 thru 4 parsing/validation
- additional metadata on parsing/validation in DAC for sys admins
- parsing status added to submission history
- bug fixes:
- more meaningful error message when user tries to upload a file that does not have the proper file extension
- error message when combobox isnt populated
- documentation
- sprint summaries
- tech documentation for nexus and db issues
- devops: clamav reconfig to save RAM costs.
This will be merged after getting guidance on running the migrations.
|
fyi: no manual migrations needed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary of Changes
Please see https://github.com/raft-tech/TANF-app/releases/tag/v3.1.6-86