2411 metadata parsed datafiles (#2706)

* saving state real quick * finishing merge with latest * Missed old test script * Added new test, more cleanup * Updating unit tests in DFS, preparing for 1610 * Merging in Jan's 1610 code for parserError useful-ness * Revert "Merging in Jan's 1610 code for parserError useful-ness" This reverts commit c5796da. * update to test to use dfs fixture * saving state before new 1610 merge * Resolving merge conflicts with 1610. * Linting changes and comparing to 1610 * Some unit test linting but inherited 1610 issues * Re-ordering job to run tests vs lint first. * Updates to linting and unit tests. * Fixing linting. * Update tdrs-backend/setup.cfg * updates per PR. * Excluding trailers for rejection * VSCode merge resolution is garbage. * Fixing precheck for not implemented types * Updating to error-handle not implemented schema types * - Updated view to show latest datafiles - Added admin filter to show newest or all datafile records - Updated indices to allow easier elastic queries * - Updated search indices to have parent FK * - Fix lint errors * - Updated submission tests - Moved create_datafile to util * - fix lint errors * - removing frontend filtering * - addding datafile to admin model * Revert "- addding datafile to admin model" This reverts commit 35a6f24. * - Fixed issue where datafile FK wasnt populating - Regenerated migration * - Readding datafile back to admin view now that the error is resolved * - adding datafile back * Revert "- Readding datafile back to admin view now that the error is resolved" This reverts commit 2807425. * - Removed unnecessary fields - Updated dependencies - Updated filter * - Updated document to include required fields * - Fixed failing test * add adminUrl to deployment cypress overrides * Adding "beta" banners to relevant error report sections (#2522) * Update views.py * Update views.py * Update SubmissionHistory.jsx * Update SubmissionHistory.test.js * Apply suggestions from code review Co-authored-by: Miles Reiter <[email protected]> * lint fixes --------- Co-authored-by: Miles Reiter <[email protected]> Co-authored-by: Alex P <[email protected]> Co-authored-by: andrew-jameson <[email protected]> * Create sprint-73-summary.md (#2565) * hotfix for large file sizes (#2542) * hotfix for large file sizes * apply timeouts/req limits to dev * filter identity pages from scan * IGNORE sql injection --------- Co-authored-by: Jan Timpe <[email protected]> Co-authored-by: mo sohani <[email protected]> Co-authored-by: Alex P <[email protected]> * updating validation error language * accidentally included coding challenge * rm comments * 2550 deactivation email link (#2557) * - updated nginx buildpack * - specifying different nginx version * - Updating changelog * - added script to update certain apps in cf - added workflow for each environment in circi * - fixed base config * - fixing jobs * - Updated based on feedback in OH * - Updating defaults * - Removing defaults * - Fixing indent * - Adding params to config * test * test * - updating work dir * - Adding checkout * - adding cf check * - logging into cf * - update cf check to install required binary * - removing unnecessary switch * - Forcing plugin installation * - test installing plugin from script also * - Adding url to email * - test code for sandbox * - using my email * Revert "Merge branch 'update-cf-os' into 2551-deactivation-email-link" This reverts commit e963b9d, reversing changes made to cc9cf81. * Revert "- using my email" This reverts commit cc9cf81. * Revert "- test code for sandbox" This reverts commit 0603774. --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]> * Update README.md (#2577) Add ATO Co-authored-by: Andrew <[email protected]> * Create 2023, Spring - Testing CSV & Excel-based error reports.md * Update README.md * Updating deliverable links (#2584) * User viewset not returning/duplicating users (#2573) * - Fixed issue not allowing pagination to work locally with nginx - Added ordering to user field to fix duplicates issue * - fix lint error * - Removing ID check since we cannot guarantee that the uuid that is generated per test run will be lexigraphically consistent --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]> * Update cf os (#2523) * - updated nginx buildpack * - specifying different nginx version * - Updating changelog * - added script to update certain apps in cf - added workflow for each environment in circi * - fixed base config * - fixing jobs * - Updated based on feedback in OH * - Updating defaults * - Removing defaults * - Fixing indent * - Adding params to config * test * test * - updating work dir * - Adding checkout * - adding cf check * - logging into cf * - update cf check to install required binary * - removing unnecessary switch * - Forcing plugin installation * - test installing plugin from script also * - Adding new dependencies * - adding package * - fixing broken install * - fixing libs * - using correct command * - gettign correct version of libc * - trying to upgrade libs * - testing * - Updated README and script * Revert "- Updated README and script" This reverts commit 92697b3. * - Removed unnecessary circi stuff - Removed script - Updated docs to callout updating secondary apps * - Correct spelling error --------- Co-authored-by: Andrew <[email protected]> * Item Number Mismatch (#2578) * - Updated schemas and models to reflect correct item numbers of fields * - Revert migration * - Updated header/trailer item numbers * - Fixed item numbers off by one errors --------- Co-authored-by: Andrew <[email protected]> * pipeline filtering (#2538) * pipeline changes that filter based on paths and branches. circle ci tracks specified branches in order to keep current functionality on HHS side. * updated syntax to be in line with build-all.yml * removed comma * WIP build flow docs * added Architecture Decision Record for the change to pipeline workflows * corrected file type of doc to .md --------- Co-authored-by: George Hudson <[email protected]> Co-authored-by: Andrew <[email protected]> * Hotfix Devops/2457 path filtering for documentation (#2597) * pipeline changes that filter based on paths and branches. circle ci tracks specified branches in order to keep current functionality on HHS side. * updated syntax to be in line with build-all.yml * removed comma * WIP build flow docs * added Architecture Decision Record for the change to pipeline workflows * corrected file type of doc to .md * build and test all on PRs even for documentation --------- Co-authored-by: George Hudson <[email protected]> * Create sprint-74-summary.md (#2596) Co-authored-by: Andrew <[email protected]> * added URL filters (#2580) * added URL filters * allow github to trigger owasp and label deploys (#2601) Co-authored-by: George Hudson <[email protected]> --------- Co-authored-by: Andrew <[email protected]> Co-authored-by: George Hudson <[email protected]> Co-authored-by: George Hudson <[email protected]> * Create sprint-75-summary.md (#2608) * Create sprint-76-summary.md (#2609) Co-authored-by: Andrew <[email protected]> * - Resolved failing tests * - Corrected merge thrash * - Using randbits to generate pk to get around confilcting sequence pks * Revert "- Using randbits to generate pk to get around confilcting sequence pks" This reverts commit ac9b065. * - Updating region in fixture instead of factory - letting django handle transaction for test * - Moved datafile reference to avoid confusion * pushing up incomplete codebase * Other unit tests now have passed w/ good error handling * Working tests, need to get setup for case aggregates populating via DB * - Updated queries - Added helper function - Need to merge in 2579 for queries to work * minor improvement to month2int * - Fixing most merge errors * - Fixing functions * - Updated queries based on generic relation * - Updated queries to count by case number instead of record number * - Added route - Updated task to create dfs * - updated tests to include dfs * Cleaning up most comments that are no longer necessary and fixed lint issues. * making minor updates, still broken tests. * updating pipfile.lock and rebuild image resolved test issues * Reorganizing tests, still failing in test_parse.py * deleted summary file, split into other test scripts. * Fixed missing self reference. * Linting fixes. * Found reference failure in deployed env. * Removing extra returns for missing record type. * lint fix * Addressed invocation of datafile for failing test * lint update for whitespace * Intermediary commit, broken test * new assignemnts in util * - updated rejected query to correctly count objs * - Fixing most tests * - Fixed user error. Swapped numbers by accident. * - make region None to avoid PK collision * - Fix lint errors * - Updating to avoid warning * vscode merge conflict resolution (#2623) * auto-create the external network * didn't stage commit properly * checking diffs, matching 1613.2 * doesn't work in pipeline. must be cached local * re-commenting in unit test * lint failures fixed --------- Co-authored-by: andrew-jameson <[email protected]> * url change per me, want pipeline to run e2e * Upgraded to querysets, fix PR comments, PE str * missing : not caught locally * Feat/1613 merge 2 (#2650) * Create sprint-78-summary.md (#2645) * Missing/unsaved parser_error for record_type * removing redundant tests * Hopefully resolved on unit tests and lint --------- Co-authored-by: Smithh-Co <[email protected]> Co-authored-by: andrew-jameson <[email protected]> * icontains * tests * Changing dict structure per 1612. * fixed tests and lint issues, parse is too complex * schema_manager replaces schema check * Saving state prior to merge-conflict. * Adopting latest manager, removing old error style. * Commented out t6 line during Office hours * minor reference update * Acclimating to schemaManager * lint-fix isinstance * syntax mistake with isinstance * Apply suggestions from code review * reverting search_index merge artifacts. * adjusting for removing unused "get-schema()" * whitespace lint * Feedback from Jan * Ensuring tests run/work. * Ensure we have leading zero in rptmonthyear. * Minor lint fix for exception logging * resolving merge conflict problems * added the filters for datafile progtype * added url to admin filters * added datafile summary link to admin page * filtering parser errors * linting * Test data_file admin page * corrected datafile id for test file * linting * Added deplay to cypress test * changed datafile summary filter * Changed error report link * changed filter to reflect related name --------- Co-authored-by: Andrew Jameson <[email protected]> Co-authored-by: Andrew <[email protected]> Co-authored-by: andrew-jameson <[email protected]> Co-authored-by: elipe17 <[email protected]> Co-authored-by: Jan Timpe <[email protected]> Co-authored-by: Miles Reiter <[email protected]> Co-authored-by: Alex P <[email protected]> Co-authored-by: Smithh-Co <[email protected]> Co-authored-by: Eric Lipe <[email protected]> Co-authored-by: Lauren Frohlich <[email protected]> Co-authored-by: Miles Reiter <[email protected]> Co-authored-by: George Hudson <[email protected]> Co-authored-by: George Hudson <[email protected]>
raft-tech · Oct 11, 2023 · 281cad7 · 281cad7
1 parent e178c5c
commit 281cad7
Show file tree

Hide file tree

Showing 4 changed files with 95 additions and 1 deletion.
diff --git a/tdrs-backend/tdpservice/data_files/admin.py b/tdrs-backend/tdpservice/data_files/admin.py
@@ -3,19 +3,73 @@
 
 from ..core.utils import ReadOnlyAdminMixin
 from .models import DataFile, LegacyFileTransfer
+from tdpservice.parsers.models import DataFileSummary, ParserError
+from django.conf import settings
+from django.utils.html import format_html
 
+DOMAIN = settings.FRONTEND_BASE_URL
+
+class DataFileSummaryPrgTypeFilter(admin.SimpleListFilter):
+    """Admin class filter for Program Type on datafile model."""
+
+    title = 'Program Type'
+    parameter_name = 'program_type'
+
+    def lookups(self, request, model_admin):
+        """Return a list of tuples."""
+        return [
+            ('TAN', 'TAN'),
+            ('SSP', 'SSP'),
+        ]
+
+    def queryset(self, request, queryset):
+        """Return a queryset."""
+        if self.value():
+            query_set_ids = [df.id for df in queryset if df.prog_type == self.value()]
+            return queryset.filter(id__in=query_set_ids)
+        else:
+            return queryset
 
 @admin.register(DataFile)
 class DataFileAdmin(ReadOnlyAdminMixin, admin.ModelAdmin):
     """Admin class for DataFile models."""
 
+    def status(self, obj):
+        """Return the status of the data file summary."""
+        return DataFileSummary.objects.get(datafile=obj).status
+
+    def case_totals(self, obj):
+        """Return the case totals."""
+        return DataFileSummary.objects.get(datafile=obj).case_aggregates
+
+    def error_report_link(self, obj):
+        """Return the link to the error report."""
+        pe_len = ParserError.objects.filter(file=obj).count()
+
+        filtered_parserror_list_url = f'{DOMAIN}/admin/parsers/parsererror/?file=' + str(obj.id)
+        # have to find the error id from obj
+        return format_html("<a href='{url}'>{field}</a>",
+                           field="Parser Errors: " + str(pe_len),
+                           url=filtered_parserror_list_url)
+
+    error_report_link.allow_tags = True
+
+    def data_file_summary(self, obj):
+        """Return the data file summary."""
+        df = DataFileSummary.objects.get(datafile=obj)
+        return format_html("<a href='{url}'>{field}</a>",
+                           field=f'{df.id}' + ":" + df.get_status(),
+                           url=f"{DOMAIN}/admin/parsers/datafilesummary/{df.id}/change/")
+
     list_display = [
         'id',
         'stt',
         'year',
         'quarter',
         'section',
         'version',
+        'data_file_summary',
+        'error_report_link',
     ]
 
     list_filter = [
@@ -25,6 +79,8 @@ class DataFileAdmin(ReadOnlyAdminMixin, admin.ModelAdmin):
         'user',
         'year',
         'version',
+        'summary__status',
+        DataFileSummaryPrgTypeFilter
     ]
 
 @admin.register(LegacyFileTransfer)

diff --git a/tdrs-backend/tdpservice/data_files/models.py b/tdrs-backend/tdpservice/data_files/models.py
@@ -152,6 +152,20 @@ class Meta:
                                         null=True
                                         )
 
+    @property
+    def prog_type(self):
+        """Return the program type for a given section."""
+        # e.g., 'SSP Closed Case Data'
+        if self.section.startswith('SSP'):
+            return 'SSP'
+        elif self.section.startswith('Tribal'):
+            return 'TAN'  # problematic, do we need to infer tribal entirely from tribe/fips code?
+        else:
+            return 'TAN'
+
+        # TODO: if given a datafile (section), we can reverse back to the program b/c the
+        # section string has "tribal/ssp" in it, then process of elimination we have tanf
+
     @property
     def filename(self):
         """Return the correct filename for this data file."""

diff --git a/tdrs-backend/tdpservice/data_files/test/test_admin.py b/tdrs-backend/tdpservice/data_files/test/test_admin.py
@@ -0,0 +1,22 @@
+"""Test DataFileAdmin methods."""
+import pytest
+from django.contrib.admin.sites import AdminSite
+
+from tdpservice.data_files.admin import DataFileAdmin
+from tdpservice.data_files.models import DataFile
+from tdpservice.data_files.test.factories import DataFileFactory
+from tdpservice.parsers.test.factories import DataFileSummaryFactory
+from django.conf import settings
+
+@pytest.mark.django_db
+def test_DataFileAdmin_status():
+    """Test DataFileAdmin status method."""
+    data_file = DataFileFactory()
+    data_file_summary = DataFileSummaryFactory(datafile=data_file)
+    data_file_admin = DataFileAdmin(DataFile, AdminSite())
+
+    assert data_file_admin.status(data_file) == data_file_summary.status
+    assert data_file_admin.case_totals(data_file) == data_file_summary.case_aggregates
+    DOMAIN = settings.FRONTEND_BASE_URL
+    assert data_file_admin.error_report_link(data_file) == \
+        f"<a href='{DOMAIN}/admin/parsers/parsererror/?file={data_file.id}'>Parser Errors: 0</a>"
diff --git a/tdrs-frontend/cypress/e2e/accounts/accounts.js b/tdrs-frontend/cypress/e2e/accounts/accounts.js
@@ -8,7 +8,9 @@ Then('{string} sees a Request Access form', (username) => {
 
 Then('{string} can see the hompage', (username) => {
   cy.visit('/home')
-  cy.contains('You have been approved for access to TDP.').should('exist')
+  cy.wait(2000).then(() => {
+    cy.contains('You have been approved for access to TDP.').should('exist')
+  })
 })
 
 When('{string} is in begin state', (username) => {