Feat/2646 total errors aggregates #2800

jtimpe · 2024-01-05T15:51:50Z

Summary of Changes

Pull request closes #2646

Implements a new total_errors_by_month aggregation function, implemented for section 3+4 submissions
Updates the frontend SubmissionHistory component structure to be more readable and limit conditional rendering. some small code duplications and inefficiencies can be ironed out if necessary

How to Test

List the steps to test the PR
These steps are generic, please adjust as necessary.

cd tdrs-backend && docker-compose up
cd tdrs-frontend && docker-compose up --build

Open http://localhost:3000/ and sign in.
Submit data files for all sections to test the frontend restructuring
Submit data files for sections 3+4 and validate the "Total Errors" column in the submission history

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

A DataFileSummary created for a submission has case_aggregates populated with the correct data for the section
lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

Are all areas of code introduced in this PR meaningfully tested?
- If this PR introduces backend code changes, are they meaningfully tested?
- If this PR introduces frontend code changes, are they meaningfully tested?
Are code coverage minimums met?
- Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
- Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

Are backend code style checks passing on CircleCI?
Are frontend code style checks passing on CircleCI?
Are code maintainability principles being followed?

Deliverable 4: Accessible

Does this PR complete the epic?
Are links included to any other gov-approved PRs associated with epic?
Does PR include documentation for Raft's a11y review?
Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

Does this PR provide background for why coding decisions were made?
If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
If this PR introduces dependencies, are their licenses documented?
Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

Does the OWASP Scan pass on CircleCI?
Do manual code review and manual testing detect any new security issues?
If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

the purpose of the research
methods used to conduct the research
who participated in the research
what was tested and how
impact of research on TDP
(if applicable) final design mockups produced for TDP development

codecov · 2024-01-19T16:15:40Z

Codecov Report

Attention: Patch coverage is 77.92208% with 17 lines in your changes are missing coverage. Please review.

Project coverage is 93.56%. Comparing base (0619329) to head (b0e063a).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2800      +/-   ##
===========================================
- Coverage    93.62%   93.56%   -0.06%     
===========================================
  Files          262      265       +3     
  Lines         6055     6080      +25     
  Branches       508      510       +2     
===========================================
+ Hits          5669     5689      +20     
- Misses         291      298       +7     
+ Partials        95       93       -2

Flag	Coverage Δ
dev-backend	`93.75% <75.00%> (-0.07%)`	⬇️
dev-frontend	`92.62% <78.68%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...components/SubmissionHistory/SubmissionHistory.jsx	`100.00% <100.00%> (+16.21%)`	⬆️
tdrs-backend/tdpservice/scheduling/parser_task.py	`50.00% <50.00%> (-2.64%)`	⬇️
tdrs-backend/tdpservice/parsers/aggregates.py	`91.48% <78.57%> (-5.49%)`	⬇️
...mponents/SubmissionHistory/CaseAggregatesTable.jsx	`81.25% <81.25%> (ø)`
...ponents/SubmissionHistory/TotalAggregatesTable.jsx	`66.66% <66.66%> (ø)`
...ntend/src/components/SubmissionHistory/helpers.jsx	`82.75% <82.75%> (ø)`

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0619329...b0e063a. Read the comment docs.

tdrs-backend/tdpservice/parsers/test/test_parse.py

raftmsohani · 2024-01-22T12:46:00Z

tdrs-backend/tdpservice/scheduling/parser_task.py

@@ -27,6 +27,8 @@ def parse(data_file_id):

    if "Case Data" in data_file.section:


not related to this PR but we should have had "Case Data" as a CONSTANT, and I would put it in .DataFile.py

tdrs-backend/tdpservice/parsers/util.py

elipe17

Looks good!

raftmsohani · 2024-02-01T17:39:26Z

tdrs-backend/tdpservice/parsers/aggregates.py

+        month_int = month_to_int(month)
+        rpt_month_year = int(f"{calendar_year}{month_int}")
+
+        error_count = ParserError.objects.filter(


we should be able to query all ParserErrors before running the for loop, and have the counts in a table.

Another note: as we are parsing more and more files, this query would take forever to run, the problem is df is indexed easily but the rpt_month_year will be slow.

i pulled the main ParserError.objects.all() out of the loop - are you suggesting grouping by rpt_month_year or something to that effect as well?

raftmsohani

more efficient query!

ADPennington · 2024-02-13T18:02:16Z

@jtimpe did this fall off my radar?

jtimpe · 2024-02-13T18:06:18Z

@jtimpe did this fall off my radar?

nope! i am still addressing PR feedback and getting approvals - once i've done so i'll follow back up with you

raftmsohani

LGTM!

andrew-jameson

Good to go!

ADPennington · 2024-02-26T17:14:32Z

@jtimpe there is a merge conflict here. can you ping me when its ready again?

ADPennington

lgtm @jtimpe 🚀

Section 3 and 4 total errors for tanf, tribal, and ssp get populated with 0 or non-zero values when status != Rejected
there are a couple of edge cases that I presume will be addressed in later tickets (let me know if this is not the case):
- Accepted with errors status and only error detected is in trailer record: Total Errors == 0 ⚠️ (maybe Cat1 Cleanup will resolve this )
- any Partially Accepted with Errors status: Total Errors == 0 – this status only comes up when at least one data record (not header/trailer) is not correct length to parse. stratum and aggregate data files only have one data record. so this status does not make sense. ⚠️ will be addressed in #2754

test notes here

jtimpe and others added 10 commits January 5, 2024 10:46

total_errors_by_month aggregator

58ddd8d

impl total_errors aggregator

b86e66f

submission history component structure

15be219

total errors frontend impl

525c18c

Merge branch 'develop' into feat/2646-total-errors-aggregates

52f1025

add section 3/4 aggregates to tests

da3f2b0

Merge branch 'develop' into feat/2646-total-errors-aggregates

f3a20ed

lint

6b6410c

fix null coalescing

6d4f282

fix+add aggregates tests

dd039b1

jtimpe self-assigned this Jan 19, 2024

jtimpe added the raft review This issue is ready for raft review label Jan 19, 2024

jtimpe requested review from andrew-jameson, raftmsohani and elipe17 January 19, 2024 16:32

elipe17 reviewed Jan 19, 2024

View reviewed changes

tdrs-backend/tdpservice/parsers/test/test_parse.py Show resolved Hide resolved

raftmsohani reviewed Jan 22, 2024

View reviewed changes

tdrs-backend/tdpservice/parsers/test/test_parse.py Outdated Show resolved Hide resolved

raftmsohani reviewed Jan 22, 2024

View reviewed changes

jtimpe commented Jan 22, 2024

View reviewed changes

tdrs-backend/tdpservice/parsers/util.py Outdated Show resolved Hide resolved

andrew-jameson reviewed Jan 22, 2024

View reviewed changes

tdrs-backend/tdpservice/parsers/util.py Outdated Show resolved Hide resolved

andrew-jameson reviewed Jan 22, 2024

View reviewed changes

tdrs-backend/tdpservice/parsers/util.py Outdated Show resolved Hide resolved

jtimpe added 8 commits January 23, 2024 13:18

add tribal tests, positive case test

cd5f733

remove unused code from total_errors aggregate func

b1889aa

remove unnecessary saves

850bbd0

Merge branch 'develop' into feat/2646-total-errors-aggregates

d671e2a

move total_errors to aggregates.py

1c2ce4f

extra blank line

7c3180c

undo formatter changes

54e98e5

extra space

1124d5a

jtimpe requested a review from raftmsohani January 23, 2024 19:11

elipe17 approved these changes Jan 31, 2024

View reviewed changes

raftmsohani reviewed Feb 1, 2024

View reviewed changes

raftmsohani requested changes Feb 1, 2024

View reviewed changes

jtimpe and others added 2 commits February 13, 2024 13:06

Merge branch 'develop' into feat/2646-total-errors-aggregates

579b526

make the query more efficient

12c3d6b

jtimpe requested review from raftmsohani and andrew-jameson and removed request for andrew-jameson February 13, 2024 18:23

raftmsohani approved these changes Feb 14, 2024

View reviewed changes

andrew-jameson approved these changes Feb 20, 2024

View reviewed changes

andrew-jameson added QASP Review and removed raft review This issue is ready for raft review labels Feb 20, 2024

ADPennington added 2 commits February 21, 2024 11:24

Merge branch 'develop' into feat/2646-total-errors-aggregates

51a5a09

Merge branch 'develop' into feat/2646-total-errors-aggregates

13a7c18

ADPennington added the Blocked Label for Pull Requests that are currently blocked by a dependency label Feb 26, 2024

jtimpe added 3 commits February 26, 2024 12:51

Merge branch 'develop' into feat/2646-total-errors-aggregates

94f2477

rm unused code

f836320

fix error report url

b0e063a

ADPennington added Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI and removed Blocked Label for Pull Requests that are currently blocked by a dependency labels Feb 26, 2024

ADPennington approved these changes Feb 27, 2024

View reviewed changes

ADPennington added Ready to Merge and removed QASP Review Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI labels Feb 27, 2024

andrew-jameson merged commit c8d0934 into develop Feb 28, 2024
25 checks passed

andrew-jameson deleted the feat/2646-total-errors-aggregates branch February 28, 2024 17:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/2646 total errors aggregates #2800

Feat/2646 total errors aggregates #2800

jtimpe commented Jan 5, 2024 •

edited by ADPennington

Loading

codecov bot commented Jan 19, 2024 •

edited

Loading

raftmsohani Jan 22, 2024

elipe17 left a comment

raftmsohani Feb 1, 2024

jtimpe Feb 13, 2024

raftmsohani left a comment

ADPennington commented Feb 13, 2024

jtimpe commented Feb 13, 2024

raftmsohani left a comment

andrew-jameson left a comment

ADPennington commented Feb 26, 2024

ADPennington left a comment

		@@ -27,6 +27,8 @@ def parse(data_file_id):

		if "Case Data" in data_file.section:

Feat/2646 total errors aggregates #2800

Feat/2646 total errors aggregates #2800

Conversation

jtimpe commented Jan 5, 2024 • edited by ADPennington Loading

Summary of Changes

How to Test

Deliverables

codecov bot commented Jan 19, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

elipe17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raftmsohani left a comment

Choose a reason for hiding this comment

ADPennington commented Feb 13, 2024

jtimpe commented Feb 13, 2024

raftmsohani left a comment

Choose a reason for hiding this comment

andrew-jameson left a comment

Choose a reason for hiding this comment

ADPennington commented Feb 26, 2024

ADPennington left a comment

Choose a reason for hiding this comment

jtimpe commented Jan 5, 2024 •

edited by ADPennington

Loading

codecov bot commented Jan 19, 2024 •

edited

Loading