Skip to content

Commit

Permalink
1612 case aggregates frontend (#2643)
Browse files Browse the repository at this point in the history 
* saving state real quick

* finishing merge with latest

* Missed old test script

* Added new test, more cleanup

* Updating unit tests in DFS, preparing for 1610

* Merging in Jan's 1610 code for parserError useful-ness

* Revert "Merging in Jan's 1610 code for parserError useful-ness"

This reverts commit c5796da.

* update to test to use dfs fixture

* saving state before new 1610 merge

* Resolving merge conflicts with 1610.

* Linting changes and comparing to 1610

* Some unit test linting but inherited 1610 issues

* Re-ordering job to run tests vs lint first.

* Updates to linting and unit tests.

* Fixing linting.

* Update tdrs-backend/setup.cfg

* updates per PR.

* Excluding trailers for rejection

* VSCode merge resolution is garbage.

* Fixing precheck for not implemented types

* Updating to error-handle not implemented schema types

* - Updated view to show latest datafiles
- Added admin filter to show newest or all datafile records
- Updated indices to allow easier elastic queries

* - Updated search indices to have parent FK

* - Fix lint errors

* - Updated submission tests
- Moved create_datafile to util

* - fix lint errors

* - removing frontend filtering

* - addding datafile to admin model

* Revert "- addding datafile to admin model"

This reverts commit 35a6f24.

* - Fixed issue where datafile FK wasnt populating
- Regenerated migration

* - Readding datafile back to admin view now that the error is resolved

* - adding datafile back

* Revert "- Readding datafile back to admin view now that the error is resolved"

This reverts commit 2807425.

* - Removed unnecessary fields
- Updated dependencies
- Updated filter

* - Updated document to include required fields

* - Fixed failing test

* add adminUrl to deployment cypress overrides

* Adding "beta" banners to relevant error report sections (#2522)

* Update views.py

* Update views.py

* Update SubmissionHistory.jsx

* Update SubmissionHistory.test.js

* Apply suggestions from code review

Co-authored-by: Miles Reiter <[email protected]>

* lint fixes

---------

Co-authored-by: Miles Reiter <[email protected]>
Co-authored-by: Alex P <[email protected]>
Co-authored-by: andrew-jameson <[email protected]>

* Create sprint-73-summary.md (#2565)

* hotfix for large file sizes (#2542)

* hotfix for large file sizes

* apply timeouts/req limits to dev

* filter identity pages from scan

* IGNORE sql injection

---------

Co-authored-by: Jan Timpe <[email protected]>
Co-authored-by: mo sohani <[email protected]>
Co-authored-by: Alex P <[email protected]>

* updating validation error language

* accidentally included coding challenge

* rm comments

* 2550 deactivation email link (#2557)

* - updated nginx buildpack

* - specifying different nginx version

* - Updating changelog

* - added script to update certain apps in cf
- added workflow for each environment in circi

* - fixed base config

* - fixing jobs

* - Updated based on feedback in OH

* - Updating defaults

* - Removing defaults

* - Fixing indent

* - Adding params to config

* test

* test

* - updating work dir

* - Adding checkout

* - adding cf check

* - logging into cf

* - update cf check to install required binary

* - removing unnecessary switch

* - Forcing plugin installation

* - test installing plugin from script also

* - Adding url to email

* - test code for sandbox

* - using my email

* Revert "Merge branch 'update-cf-os' into 2551-deactivation-email-link"

This reverts commit e963b9d, reversing
changes made to cc9cf81.

* Revert "- using my email"

This reverts commit cc9cf81.

* Revert "- test code for sandbox"

This reverts commit 0603774.

---------

Co-authored-by: Alex P <[email protected]>
Co-authored-by: Andrew <[email protected]>

* Update README.md (#2577)

Add ATO

Co-authored-by: Andrew <[email protected]>

* Create 2023, Spring - Testing CSV & Excel-based error reports.md

* Update README.md

* Updating deliverable links (#2584)

* User viewset not returning/duplicating users (#2573)

* - Fixed issue not allowing pagination to work locally with nginx
- Added ordering to user field to fix duplicates issue

* - fix lint error

* - Removing ID check since we cannot guarantee that the uuid that is generated per test run will be lexigraphically consistent

---------

Co-authored-by: Alex P <[email protected]>
Co-authored-by: Andrew <[email protected]>

* Update cf os (#2523)

* - updated nginx buildpack

* - specifying different nginx version

* - Updating changelog

* - added script to update certain apps in cf
- added workflow for each environment in circi

* - fixed base config

* - fixing jobs

* - Updated based on feedback in OH

* - Updating defaults

* - Removing defaults

* - Fixing indent

* - Adding params to config

* test

* test

* - updating work dir

* - Adding checkout

* - adding cf check

* - logging into cf

* - update cf check to install required binary

* - removing unnecessary switch

* - Forcing plugin installation

* - test installing plugin from script also

* - Adding new dependencies

* - adding package

* - fixing broken install

* - fixing libs

* - using correct command

* - gettign correct version of libc

* - trying to upgrade libs

* - testing

* - Updated README and script

* Revert "- Updated README and script"

This reverts commit 92697b3.

* - Removed unnecessary circi stuff
- Removed script
- Updated docs to callout updating secondary apps

* - Correct spelling error

---------

Co-authored-by: Andrew <[email protected]>

* Item Number Mismatch (#2578)

* - Updated schemas and models to reflect correct item numbers of fields

* - Revert migration

* - Updated header/trailer item numbers

* - Fixed item numbers off by one errors

---------

Co-authored-by: Andrew <[email protected]>

* pipeline filtering (#2538)

* pipeline changes that filter based on paths and branches. circle ci tracks specified branches in order to keep current functionality on HHS side.

* updated syntax to be in line with build-all.yml

* removed comma

* WIP build flow docs

* added Architecture Decision Record for the change to pipeline workflows

* corrected file type of doc to .md

---------

Co-authored-by: George Hudson <[email protected]>
Co-authored-by: Andrew <[email protected]>

* Hotfix Devops/2457 path filtering for documentation (#2597)

* pipeline changes that filter based on paths and branches. circle ci tracks specified branches in order to keep current functionality on HHS side.

* updated syntax to be in line with build-all.yml

* removed comma

* WIP build flow docs

* added Architecture Decision Record for the change to pipeline workflows

* corrected file type of doc to .md

* build and test all on PRs even for documentation

---------

Co-authored-by: George Hudson <[email protected]>

* Create sprint-74-summary.md (#2596)

Co-authored-by: Andrew <[email protected]>

* added URL filters (#2580)

* added URL filters

* allow github to trigger owasp and label deploys (#2601)

Co-authored-by: George Hudson <[email protected]>

---------

Co-authored-by: Andrew <[email protected]>
Co-authored-by: George Hudson <[email protected]>
Co-authored-by: George Hudson <[email protected]>

* Create sprint-75-summary.md (#2608)

* Create sprint-76-summary.md (#2609)

Co-authored-by: Andrew <[email protected]>

* - Resolved failing tests

* - Corrected merge thrash

* - Using randbits to generate pk to get around confilcting sequence pks

* Revert "- Using randbits to generate pk to get around confilcting sequence pks"

This reverts commit ac9b065.

* - Updating region in fixture instead of factory
- letting django handle transaction for test

* - Moved datafile reference to avoid confusion

* pushing up incomplete codebase

* Other unit tests now have passed w/ good error handling

* Working tests, need to get setup for case aggregates populating via DB

* - Updated queries
- Added helper function
- Need to merge in 2579 for queries to work

* minor improvement to month2int

* - Fixing most merge errors

* - Fixing functions

* - Updated queries based on generic relation

* - Updated queries to count by case number instead of record number

* - Added route
- Updated task to create dfs

* - updated tests to include dfs

* Cleaning up most comments that are no longer necessary and fixed lint issues.

* making minor updates, still broken tests.

* updating pipfile.lock and rebuild image resolved test issues

* Reorganizing tests, still failing in test_parse.py

* deleted summary file, split into other test scripts.

* Fixed missing self reference.

* Linting fixes.

* Found reference failure in deployed env.

* Removing extra returns for missing record type.

* lint fix

* Addressed invocation of datafile for failing test

* lint update for whitespace

* Intermediary commit, broken test

* add reverse relation

* frontend summary status

* new assignemnts in util

* - updated rejected query to correctly count objs

* - Fixing most tests

* - Fixed user error. Swapped numbers by accident.

* - make region None to avoid PK collision

* - Fix lint errors

* - Updating to avoid warning

* add acceptance icon

* add reverse relationship to summary model

* vscode merge conflict resolution (#2623)

* auto-create the external network

* didn't stage commit properly

* checking diffs, matching 1613.2

* doesn't work in pipeline. must be cached local

* re-commenting in unit test

* lint failures fixed

---------

Co-authored-by: andrew-jameson <[email protected]>

* add a pending status

* add acceptance status tests

* update warning icon

* rm old assert

* update rejected icon

* add case aggregates to submission history

* url change per me, want pipeline to run e2e

* update case aggregate columns

* update col names

* Upgraded to querysets, fix PR comments, PE str

* missing : not caught locally

* Feat/1613 merge 2 (#2650)

* Create sprint-78-summary.md (#2645)

* Missing/unsaved parser_error for record_type

* removing redundant tests

* Hopefully resolved on unit tests and lint

---------

Co-authored-by: Smithh-Co <[email protected]>
Co-authored-by: andrew-jameson <[email protected]>

* icontains

* tests

* Changing dict structure per 1612.

* fixed tests and lint issues, parse is too complex

* schema_manager replaces schema check

* Saving state prior to merge-conflict.

* Adopting latest manager, removing old error style.

* Commented out t6 line during Office hours

* minor reference update

* Acclimating to schemaManager

* lint-fix isinstance

* syntax mistake with isinstance

* Apply suggestions from code review

* reverting search_index merge artifacts.

* adjusting for removing unused "get-schema()"

* whitespace lint

* Feedback from Jan

* Ensuring tests run/work.

* Ensure we have leading zero in rptmonthyear.

* Minor lint fix for exception logging

* fix migration

* add backend partially accepted status

* add frontend partially accepted status

* add partially accepted test

* rm accidental commit

* rm unused variable

* conditional submission history

* fix test

* resolving merge conflict problems

* fixing tests from merge conflicts.

* - fixed migration reference

* dumb lint fix

* - Fixed render error

* Adding section

* - Resolved type mismatches causing render failures

* - Adding branch coverage

* - Fix lint errors

* - Fix lint errors

* - Consolodated tetss

* reducing line length for lint

* - updated maxWidth to be percentage

* Moving DFS migration into it's own file to avoid conflicts.

* rename migrations

* left align error report text

* github got stuck :(

* rm comment

* rm margin, add text for no error report

* fix tests

* fix ssp case aggregates display

---------

Co-authored-by: Andrew Jameson <[email protected]>
Co-authored-by: Andrew <[email protected]>
Co-authored-by: andrew-jameson <[email protected]>
Co-authored-by: elipe17 <[email protected]>
Co-authored-by: Miles Reiter <[email protected]>
Co-authored-by: Alex P <[email protected]>
Co-authored-by: Smithh-Co <[email protected]>
Co-authored-by: mo sohani <[email protected]>
Co-authored-by: Eric Lipe <[email protected]>
Co-authored-by: Lauren Frohlich <[email protected]>
Co-authored-by: Miles Reiter <[email protected]>
Co-authored-by: George Hudson <[email protected]>
Co-authored-by: George Hudson <[email protected]>
Co-authored-by: raftmsohani <[email protected]>
  • Loading branch information
@jtimpe @andrew-jameson
@elipe17 @reitermb @ADPennington @Smithh-Co @raftmsohani @lfrohlich @George-Hudson
15 people authored Oct 5, 2023
1 parent 8de88cd commit 954edc7
Show file tree
Hide file tree
Showing 5 changed files with 194 additions and 45 deletions.
1 change: 0 additions & 1 deletion tdrs-backend/tdpservice/parsers/test/test_parse.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,6 @@ def test_parse_small_correct_file(test_datafile, dfs):
]}

assert dfs.get_status() == DataFileSummary.Status.ACCEPTED

assert TANF_T1.objects.count() == 1

# spot check
Expand Down
1 change: 0 additions & 1 deletion tdrs-backend/tdpservice/parsers/util.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -276,7 +276,6 @@ def transform_to_months(quarter):
case _:
raise ValueError("Invalid quarter value.")


def month_to_int(month):
"""Return the integer value of a month."""
return datetime.strptime(month, '%b').strftime('%m')
Expand Down
1 change: 1 addition & 0 deletions tdrs-frontend/src/assets/Reports.scss
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
border: none;
text-decoration: underline;
color: #264A64;
text-align: left;
}

.section-download:hover {
Expand Down
199 changes: 166 additions & 33 deletions tdrs-frontend/src/components/SubmissionHistory/SubmissionHistory.jsx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,63 @@ const SubmissionSummaryStatusIcon = ({ status }) => {
)
}

const CaseAggregatesHeader = ({ section }) =>
section === 1 || section === 2 ? (
<>
<th scope="col" rowSpan={2}>
Month
</th>
<th scope="col" rowSpan={2}>
Cases Without Errors
</th>
<th scope="col" rowSpan={2}>
Cases With Errors
</th>
<th scope="col" rowSpan={2}>
Records Unable To Process
</th>
</>
) : (
<>
<th scope="col" rowSpan={2}>
Month
</th>
<th scope="col" rowSpan={2}>
Total
</th>
<th scope="col" rowSpan={2}>
Cases With Errors
</th>
</>
)

const CaseAggregatesRow = ({ data, section }) =>
section === 1 || section === 2 ? (
data ? (
<>
<th scope="row">{data.month}</th>
<td>{data.accepted_without_errors}</td>
<td>{data.accepted_with_errors}</td>
</>
) : (
<>
<th scope="row">-</th>
<td>N/A</td>
<td>N/A</td>
</>
)
) : data ? (
<>
<th scope="row">{data.month}</th>
<td>{data.total}</td>
</>
) : (
<>
<th scope="row">-</th>
<td>N/A</td>
</>
)

const SubmissionHistoryRow = ({ file }) => {
const dispatch = useDispatch()

Expand All @@ -72,35 +129,96 @@ const SubmissionHistoryRow = ({ file }) => {
}
}

const section_index = (element) => file.section.includes(element)

const section = fileUploadSections.findIndex(section_index) + 1

return (
<tr>
<td>{formatDate(file.createdAt)}</td>
<td>{file.submittedBy}</td>
<td>
<button className="section-download" onClick={downloadFile}>
{file.fileName}
</button>
</td>
<td>
{file.summary ? (
<>
<SubmissionSummaryStatusIcon status={file.summary.status} />
{file.summary && file.summary.status
? file.summary.status
: 'Pending'}
</>
) : (
'N/A'
)}
</td>
<td>
{file.hasError > 0 ? (
<button className="section-download" onClick={returned_errors}>
{file.year}-{file.quarter}-{file.section}.xlsx
<>
<tr>
<th scope="rowgroup" rowSpan={3}>
{formatDate(file.createdAt)}
</th>
<th scope="rowgroup" rowSpan={3}>
{file.submittedBy}
</th>
<th scope="rowgroup" rowSpan={3}>
<button className="section-download" onClick={downloadFile}>
{file.fileName}
</button>
) : null}
</td>
</tr>
</th>
<CaseAggregatesRow
data={
file.summary &&
file.summary.case_aggregates &&
file.summary.case_aggregates.months
? file.summary.case_aggregates.months[0]
: null
}
section={section}
/>

<th scope="rowgroup" rowSpan={3}>
{file.summary &&
file.summary.case_aggregates &&
file.summary.case_aggregates.months
? file.summary.case_aggregates.rejected
: 'N/A'}
</th>

<th scope="rowgroup" rowSpan={3}>
<span>
<SubmissionSummaryStatusIcon
status={file.summary ? file.summary.status : 'Pending'}
/>
</span>
{file.summary && file.summary.status
? file.summary.status
: 'Pending'}
</th>

<th scope="rowgroup" rowSpan={3}>
{file.summary &&
file.summary.status &&
file.summary.status !== 'Pending' ? (
file.hasError > 0 ? (
<button className="section-download" onClick={returned_errors}>
{file.year}-{file.quarter}-{file.section}.xlsx
</button>
) : (
'No Errors'
)
) : (
'Pending'
)}
</th>
</tr>

<tr>
<CaseAggregatesRow
data={
file.summary &&
file.summary.case_aggregates &&
file.summary.case_aggregates.months
? file.summary.case_aggregates.months[1]
: null
}
section={section}
/>
</tr>
<tr>
<CaseAggregatesRow
data={
file.summary &&
file.summary.case_aggregates &&
file.summary.case_aggregates.months
? file.summary.case_aggregates.months[2]
: null
}
section={section}
/>
</tr>
</>
)
}

Expand All @@ -118,18 +236,33 @@ const SectionSubmissionHistory = ({ section, label, files }) => {
const pageEnd = Math.min(files.length, pageStart + pageSize)

return (
<div className="submission-history-section">
<div
className="submission-history-section usa-table-container--scrollable"
style={{ maxWidth: '100%' }}
tabIndex={0}
>
<table className="usa-table usa-table--striped">
<caption>{`Section ${section} - ${label}`}</caption>
{files && files.length > 0 ? (
<>
<thead>
<tr>
<th>Submitted On</th>
<th>Submitted By</th>
<th>File Name</th>
<th>Acceptance Status</th>
<th>Error Reports (In development)</th>
<th scope="col" rowSpan={2}>
Submitted On
</th>
<th scope="col" rowSpan={2}>
Submitted By
</th>
<th scope="col" rowSpan={2}>
File Name
</th>
<CaseAggregatesHeader section={section} />
<th scope="col" rowSpan={2}>
Status
</th>
<th scope="col" rowSpan={2}>
Error Reports (In development)
</th>
</tr>
</thead>
<tbody>
Expand Down
37 changes: 27 additions & 10 deletions tdrs-frontend/src/components/SubmissionHistory/SubmissionHistory.test.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -339,13 +339,25 @@ describe('SubmissionHistory', () => {
})

it.each([
'Pending',
'Accepted',
'Accepted with Errors',
'Partially Accepted with Errors',
'Rejected',
null,
])('Shows the submission acceptance status', (status) => {
['Pending', 'Active Case Data'],
['Pending', 'Closed Case Data'],
['Pending', 'Aggregate Data'],
['Accepted', 'Active Case Data'],
['Accepted', 'Closed Case Data'],
['Accepted', 'Aggregate Data'],
['Accepted with Errors', 'Active Case Data'],
['Accepted with Errors', 'Closed Case Data'],
['Accepted with Errors', 'Aggregate Data'],
['Partially Accepted with Errors', 'Active Case Data'],
['Partially Accepted with Errors', 'Closed Case Data'],
['Partially Accepted with Errors', 'Aggregate Data'],
['Rejected', 'Active Case Data'],
['Rejected', 'Closed Case Data'],
['Rejected', 'Aggregate Data'],
[null, 'Active Case Data'],
[null, 'Closed Case Data'],
[null, 'Aggregate Data'],
])('Shows the submission acceptance status section 3', (status, section) => {
const state = {
reports: {
files: [
Expand All @@ -354,7 +366,7 @@ describe('SubmissionHistory', () => {
fileName: 'test1.txt',
fileType: 'TANF',
quarter: 'Q1',
section: 'Active Case Data',
section: section,
uuid: '123-4-4-321',
year: '2023',
s3_version_id: '321-0-0-123',
Expand Down Expand Up @@ -392,8 +404,13 @@ describe('SubmissionHistory', () => {

setup(store)

expect(screen.queryByText('Acceptance Status')).toBeInTheDocument()
expect(screen.queryByText('Status')).toBeInTheDocument()
expect(screen.queryByText('test1.txt')).toBeInTheDocument()
expect(screen.queryByText(status || 'Pending')).toBeInTheDocument()

if (status && status !== 'Pending') {
expect(screen.queryByText(status)).toBeInTheDocument()
} else {
expect(screen.queryAllByText('Pending')).toHaveLength(2)
}
})
})

0 comments on commit 954edc7

Please sign in to comment.