From 281cad7846b333e8e666119b1840ce5fa27d7a18 Mon Sep 17 00:00:00 2001 From: raftmsohani <97037188+raftmsohani@users.noreply.github.com> Date: Wed, 11 Oct 2023 16:09:19 -0400 Subject: [PATCH] 2411 metadata parsed datafiles (#2706) * saving state real quick * finishing merge with latest * Missed old test script * Added new test, more cleanup * Updating unit tests in DFS, preparing for 1610 * Merging in Jan's 1610 code for parserError useful-ness * Revert "Merging in Jan's 1610 code for parserError useful-ness" This reverts commit c5796da69d0e9a6d356057550378d536e2be5f8b. * update to test to use dfs fixture * saving state before new 1610 merge * Resolving merge conflicts with 1610. * Linting changes and comparing to 1610 * Some unit test linting but inherited 1610 issues * Re-ordering job to run tests vs lint first. * Updates to linting and unit tests. * Fixing linting. * Update tdrs-backend/setup.cfg * updates per PR. * Excluding trailers for rejection * VSCode merge resolution is garbage. * Fixing precheck for not implemented types * Updating to error-handle not implemented schema types * - Updated view to show latest datafiles - Added admin filter to show newest or all datafile records - Updated indices to allow easier elastic queries * - Updated search indices to have parent FK * - Fix lint errors * - Updated submission tests - Moved create_datafile to util * - fix lint errors * - removing frontend filtering * - addding datafile to admin model * Revert "- addding datafile to admin model" This reverts commit 35a6f24c36c3a4c00ddcfc40f20833530b0199f4. * - Fixed issue where datafile FK wasnt populating - Regenerated migration * - Readding datafile back to admin view now that the error is resolved * - adding datafile back * Revert "- Readding datafile back to admin view now that the error is resolved" This reverts commit 2807425059fd1b5b355edfb16d30d170cf869d7b. * - Removed unnecessary fields - Updated dependencies - Updated filter * - Updated document to include required fields * - Fixed failing test * add adminUrl to deployment cypress overrides * Adding "beta" banners to relevant error report sections (#2522) * Update views.py * Update views.py * Update SubmissionHistory.jsx * Update SubmissionHistory.test.js * Apply suggestions from code review Co-authored-by: Miles Reiter * lint fixes --------- Co-authored-by: Miles Reiter Co-authored-by: Alex P <63075587+ADPennington@users.noreply.github.com> Co-authored-by: andrew-jameson * Create sprint-73-summary.md (#2565) * hotfix for large file sizes (#2542) * hotfix for large file sizes * apply timeouts/req limits to dev * filter identity pages from scan * IGNORE sql injection --------- Co-authored-by: Jan Timpe Co-authored-by: mo sohani Co-authored-by: Alex P <63075587+ADPennington@users.noreply.github.com> * updating validation error language * accidentally included coding challenge * rm comments * 2550 deactivation email link (#2557) * - updated nginx buildpack * - specifying different nginx version * - Updating changelog * - added script to update certain apps in cf - added workflow for each environment in circi * - fixed base config * - fixing jobs * - Updated based on feedback in OH * - Updating defaults * - Removing defaults * - Fixing indent * - Adding params to config * test * test * - updating work dir * - Adding checkout * - adding cf check * - logging into cf * - update cf check to install required binary * - removing unnecessary switch * - Forcing plugin installation * - test installing plugin from script also * - Adding url to email * - test code for sandbox * - using my email * Revert "Merge branch 'update-cf-os' into 2551-deactivation-email-link" This reverts commit e963b9df48dd1f72ca0c5b192c979bac11851d11, reversing changes made to cc9cf81e9d76c42f51ffd5e102f6027d3eb5e645. * Revert "- using my email" This reverts commit cc9cf81e9d76c42f51ffd5e102f6027d3eb5e645. * Revert "- test code for sandbox" This reverts commit 06037747197d17ed8e63b086fcfcf048ecb50dc4. --------- Co-authored-by: Alex P <63075587+ADPennington@users.noreply.github.com> Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com> * Update README.md (#2577) Add ATO Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com> * Create 2023, Spring - Testing CSV & Excel-based error reports.md * Update README.md * Updating deliverable links (#2584) * User viewset not returning/duplicating users (#2573) * - Fixed issue not allowing pagination to work locally with nginx - Added ordering to user field to fix duplicates issue * - fix lint error * - Removing ID check since we cannot guarantee that the uuid that is generated per test run will be lexigraphically consistent --------- Co-authored-by: Alex P <63075587+ADPennington@users.noreply.github.com> Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com> * Update cf os (#2523) * - updated nginx buildpack * - specifying different nginx version * - Updating changelog * - added script to update certain apps in cf - added workflow for each environment in circi * - fixed base config * - fixing jobs * - Updated based on feedback in OH * - Updating defaults * - Removing defaults * - Fixing indent * - Adding params to config * test * test * - updating work dir * - Adding checkout * - adding cf check * - logging into cf * - update cf check to install required binary * - removing unnecessary switch * - Forcing plugin installation * - test installing plugin from script also * - Adding new dependencies * - adding package * - fixing broken install * - fixing libs * - using correct command * - gettign correct version of libc * - trying to upgrade libs * - testing * - Updated README and script * Revert "- Updated README and script" This reverts commit 92697b3e53d1fd87b8d3e7995abb9093aa26e307. * - Removed unnecessary circi stuff - Removed script - Updated docs to callout updating secondary apps * - Correct spelling error --------- Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com> * Item Number Mismatch (#2578) * - Updated schemas and models to reflect correct item numbers of fields * - Revert migration * - Updated header/trailer item numbers * - Fixed item numbers off by one errors --------- Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com> * pipeline filtering (#2538) * pipeline changes that filter based on paths and branches. circle ci tracks specified branches in order to keep current functionality on HHS side. * updated syntax to be in line with build-all.yml * removed comma * WIP build flow docs * added Architecture Decision Record for the change to pipeline workflows * corrected file type of doc to .md --------- Co-authored-by: George Hudson Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com> * Hotfix Devops/2457 path filtering for documentation (#2597) * pipeline changes that filter based on paths and branches. circle ci tracks specified branches in order to keep current functionality on HHS side. * updated syntax to be in line with build-all.yml * removed comma * WIP build flow docs * added Architecture Decision Record for the change to pipeline workflows * corrected file type of doc to .md * build and test all on PRs even for documentation --------- Co-authored-by: George Hudson * Create sprint-74-summary.md (#2596) Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com> * added URL filters (#2580) * added URL filters * allow github to trigger owasp and label deploys (#2601) Co-authored-by: George Hudson --------- Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com> Co-authored-by: George Hudson Co-authored-by: George Hudson * Create sprint-75-summary.md (#2608) * Create sprint-76-summary.md (#2609) Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com> * - Resolved failing tests * - Corrected merge thrash * - Using randbits to generate pk to get around confilcting sequence pks * Revert "- Using randbits to generate pk to get around confilcting sequence pks" This reverts commit ac9b0659a62f64c4114c41faf0baa659a92be07c. * - Updating region in fixture instead of factory - letting django handle transaction for test * - Moved datafile reference to avoid confusion * pushing up incomplete codebase * Other unit tests now have passed w/ good error handling * Working tests, need to get setup for case aggregates populating via DB * - Updated queries - Added helper function - Need to merge in 2579 for queries to work * minor improvement to month2int * - Fixing most merge errors * - Fixing functions * - Updated queries based on generic relation * - Updated queries to count by case number instead of record number * - Added route - Updated task to create dfs * - updated tests to include dfs * Cleaning up most comments that are no longer necessary and fixed lint issues. * making minor updates, still broken tests. * updating pipfile.lock and rebuild image resolved test issues * Reorganizing tests, still failing in test_parse.py * deleted summary file, split into other test scripts. * Fixed missing self reference. * Linting fixes. * Found reference failure in deployed env. * Removing extra returns for missing record type. * lint fix * Addressed invocation of datafile for failing test * lint update for whitespace * Intermediary commit, broken test * new assignemnts in util * - updated rejected query to correctly count objs * - Fixing most tests * - Fixed user error. Swapped numbers by accident. * - make region None to avoid PK collision * - Fix lint errors * - Updating to avoid warning * vscode merge conflict resolution (#2623) * auto-create the external network * didn't stage commit properly * checking diffs, matching 1613.2 * doesn't work in pipeline. must be cached local * re-commenting in unit test * lint failures fixed --------- Co-authored-by: andrew-jameson * url change per me, want pipeline to run e2e * Upgraded to querysets, fix PR comments, PE str * missing : not caught locally * Feat/1613 merge 2 (#2650) * Create sprint-78-summary.md (#2645) * Missing/unsaved parser_error for record_type * removing redundant tests * Hopefully resolved on unit tests and lint --------- Co-authored-by: Smithh-Co <121890311+Smithh-Co@users.noreply.github.com> Co-authored-by: andrew-jameson * icontains * tests * Changing dict structure per 1612. * fixed tests and lint issues, parse is too complex * schema_manager replaces schema check * Saving state prior to merge-conflict. * Adopting latest manager, removing old error style. * Commented out t6 line during Office hours * minor reference update * Acclimating to schemaManager * lint-fix isinstance * syntax mistake with isinstance * Apply suggestions from code review * reverting search_index merge artifacts. * adjusting for removing unused "get-schema()" * whitespace lint * Feedback from Jan * Ensuring tests run/work. * Ensure we have leading zero in rptmonthyear. * Minor lint fix for exception logging * resolving merge conflict problems * added the filters for datafile progtype * added url to admin filters * added datafile summary link to admin page * filtering parser errors * linting * Test data_file admin page * corrected datafile id for test file * linting * Added deplay to cypress test * changed datafile summary filter * Changed error report link * changed filter to reflect related name --------- Co-authored-by: Andrew Jameson Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com> Co-authored-by: andrew-jameson Co-authored-by: elipe17 Co-authored-by: Jan Timpe Co-authored-by: Miles Reiter Co-authored-by: Alex P <63075587+ADPennington@users.noreply.github.com> Co-authored-by: Smithh-Co <121890311+Smithh-Co@users.noreply.github.com> Co-authored-by: Eric Lipe <125676261+elipe17@users.noreply.github.com> Co-authored-by: Lauren Frohlich <61251539+lfrohlich@users.noreply.github.com> Co-authored-by: Miles Reiter Co-authored-by: George Hudson Co-authored-by: George Hudson --- tdrs-backend/tdpservice/data_files/admin.py | 56 +++++++++++++++++++ tdrs-backend/tdpservice/data_files/models.py | 14 +++++ .../tdpservice/data_files/test/test_admin.py | 22 ++++++++ .../cypress/e2e/accounts/accounts.js | 4 +- 4 files changed, 95 insertions(+), 1 deletion(-) create mode 100644 tdrs-backend/tdpservice/data_files/test/test_admin.py diff --git a/tdrs-backend/tdpservice/data_files/admin.py b/tdrs-backend/tdpservice/data_files/admin.py index 1ac5b2b2c..1a049dad3 100644 --- a/tdrs-backend/tdpservice/data_files/admin.py +++ b/tdrs-backend/tdpservice/data_files/admin.py @@ -3,12 +3,64 @@ from ..core.utils import ReadOnlyAdminMixin from .models import DataFile, LegacyFileTransfer +from tdpservice.parsers.models import DataFileSummary, ParserError +from django.conf import settings +from django.utils.html import format_html +DOMAIN = settings.FRONTEND_BASE_URL + +class DataFileSummaryPrgTypeFilter(admin.SimpleListFilter): + """Admin class filter for Program Type on datafile model.""" + + title = 'Program Type' + parameter_name = 'program_type' + + def lookups(self, request, model_admin): + """Return a list of tuples.""" + return [ + ('TAN', 'TAN'), + ('SSP', 'SSP'), + ] + + def queryset(self, request, queryset): + """Return a queryset.""" + if self.value(): + query_set_ids = [df.id for df in queryset if df.prog_type == self.value()] + return queryset.filter(id__in=query_set_ids) + else: + return queryset @admin.register(DataFile) class DataFileAdmin(ReadOnlyAdminMixin, admin.ModelAdmin): """Admin class for DataFile models.""" + def status(self, obj): + """Return the status of the data file summary.""" + return DataFileSummary.objects.get(datafile=obj).status + + def case_totals(self, obj): + """Return the case totals.""" + return DataFileSummary.objects.get(datafile=obj).case_aggregates + + def error_report_link(self, obj): + """Return the link to the error report.""" + pe_len = ParserError.objects.filter(file=obj).count() + + filtered_parserror_list_url = f'{DOMAIN}/admin/parsers/parsererror/?file=' + str(obj.id) + # have to find the error id from obj + return format_html("{field}", + field="Parser Errors: " + str(pe_len), + url=filtered_parserror_list_url) + + error_report_link.allow_tags = True + + def data_file_summary(self, obj): + """Return the data file summary.""" + df = DataFileSummary.objects.get(datafile=obj) + return format_html("{field}", + field=f'{df.id}' + ":" + df.get_status(), + url=f"{DOMAIN}/admin/parsers/datafilesummary/{df.id}/change/") + list_display = [ 'id', 'stt', @@ -16,6 +68,8 @@ class DataFileAdmin(ReadOnlyAdminMixin, admin.ModelAdmin): 'quarter', 'section', 'version', + 'data_file_summary', + 'error_report_link', ] list_filter = [ @@ -25,6 +79,8 @@ class DataFileAdmin(ReadOnlyAdminMixin, admin.ModelAdmin): 'user', 'year', 'version', + 'summary__status', + DataFileSummaryPrgTypeFilter ] @admin.register(LegacyFileTransfer) diff --git a/tdrs-backend/tdpservice/data_files/models.py b/tdrs-backend/tdpservice/data_files/models.py index b4248a9cd..abfcce8ab 100644 --- a/tdrs-backend/tdpservice/data_files/models.py +++ b/tdrs-backend/tdpservice/data_files/models.py @@ -152,6 +152,20 @@ class Meta: null=True ) + @property + def prog_type(self): + """Return the program type for a given section.""" + # e.g., 'SSP Closed Case Data' + if self.section.startswith('SSP'): + return 'SSP' + elif self.section.startswith('Tribal'): + return 'TAN' # problematic, do we need to infer tribal entirely from tribe/fips code? + else: + return 'TAN' + + # TODO: if given a datafile (section), we can reverse back to the program b/c the + # section string has "tribal/ssp" in it, then process of elimination we have tanf + @property def filename(self): """Return the correct filename for this data file.""" diff --git a/tdrs-backend/tdpservice/data_files/test/test_admin.py b/tdrs-backend/tdpservice/data_files/test/test_admin.py new file mode 100644 index 000000000..02701fe82 --- /dev/null +++ b/tdrs-backend/tdpservice/data_files/test/test_admin.py @@ -0,0 +1,22 @@ +"""Test DataFileAdmin methods.""" +import pytest +from django.contrib.admin.sites import AdminSite + +from tdpservice.data_files.admin import DataFileAdmin +from tdpservice.data_files.models import DataFile +from tdpservice.data_files.test.factories import DataFileFactory +from tdpservice.parsers.test.factories import DataFileSummaryFactory +from django.conf import settings + +@pytest.mark.django_db +def test_DataFileAdmin_status(): + """Test DataFileAdmin status method.""" + data_file = DataFileFactory() + data_file_summary = DataFileSummaryFactory(datafile=data_file) + data_file_admin = DataFileAdmin(DataFile, AdminSite()) + + assert data_file_admin.status(data_file) == data_file_summary.status + assert data_file_admin.case_totals(data_file) == data_file_summary.case_aggregates + DOMAIN = settings.FRONTEND_BASE_URL + assert data_file_admin.error_report_link(data_file) == \ + f"Parser Errors: 0" diff --git a/tdrs-frontend/cypress/e2e/accounts/accounts.js b/tdrs-frontend/cypress/e2e/accounts/accounts.js index 3c6158204..a1feaedf3 100644 --- a/tdrs-frontend/cypress/e2e/accounts/accounts.js +++ b/tdrs-frontend/cypress/e2e/accounts/accounts.js @@ -8,7 +8,9 @@ Then('{string} sees a Request Access form', (username) => { Then('{string} can see the hompage', (username) => { cy.visit('/home') - cy.contains('You have been approved for access to TDP.').should('exist') + cy.wait(2000).then(() => { + cy.contains('You have been approved for access to TDP.').should('exist') + }) }) When('{string} is in begin state', (username) => {