2795 s1 duplicates

[elipe17]

Summary of Changes

• Added new class that manages duplicate detection while case cat4 is occurring
• Updated cat4 validation to leverage duplicate detector
• Updated parse logic to not create records in a case if it has any cat4 errors
  Pull request closes As tech lead, I need TDP to detect duplicate records within a file and not store them in the db.  #2795

CPU and Memory Analysis

• Used Docker to track the web container stats during parsing of the largest file we have in the repo (~50MB) on the develop branch and this branch.
• The overall results can be found in super_big_file_results.txt. The data listed in that text file can be found in the accompanying files: develop_super_big_file.txt and dup_super_big_file.txt.
• The data was generated with: track_docker_stats.sh.txt (GitHub won't let you upload a pure shell file).
  super_big_file_results.txt
  dup_super_big_file.txt
  develop_super_big_file.txt
  track_docker_stats.sh.txt

How to Test

List the steps to test the PR
These steps are generic, please adjust as necessary.

cd tdrs-frontend && docker-compose up
cd tdrs-backend && docker-compose up

1. Open http://localhost:3000/ and sign in.
2. Create a file for TANF section 1 - 4 that has one or more duplicate or partial duplicate records.
3. Submit file
4. Verify no records tied to that case are created and the error report reflects cat4 duplicate errors

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• In section 1 files, exact and partial duplicate records are detected, not stored in db, and error message included in the feedback report.
• In section 2 files, exact and partial duplicate records are detected, not stored in db, and error message included in the feedback report.
• section 3 files, partial duplicate T6 records are detected, not stored in db, and error message included in the feedback report.
• section 4 files, partial duplicate T7 records are detected, not stored in db, and error message included in the feedback report.
• duplicate detection logic is applied consistently to TANF, SSP, and Tribal TANF files.
• cat 4 records not stored in db
• Testing Checklist has been run and all tests pass
• README is updated, if necessary

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • If this PR introduces backend code changes, are they meaningfully tested?
  • If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• Are backend code style checks passing on CircleCI?
• Are frontend code style checks passing on CircleCI?
• Are code maintainability principles being followed?

Deliverable 4: Accessible

• Does this PR complete the epic?
• Are links included to any other gov-approved PRs associated with epic?
• Does PR include documentation for Raft's a11y review?
• Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• Does this PR provide background for why coding decisions were made?
• If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces dependencies, are their licenses documented?
• Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• Does the OWASP Scan pass on CircleCI?
• Do manual code review and manual testing detect any new security issues?
• If new issues detected, is investigation and/or remediation plan documented?

[ADPennington]

duplicative error messages produced by the following file

SSP_Sec1.txt

Every M1 record should have at least one corresponding M2 or M3 record with the same RPT_MONTH_YEAR and CASE_NUMBER.
Every M1 record should have at least one corresponding M2 or M3 record with the same RPT_MONTH_YEAR and CASE_NUMBER.

[ADPennington]

@elipe17 this is in great shape. couple of suggestions on cat 4 error messages that either need clarity or are redundant and could be dropped.

Testing Summary:

• expected logic is in-place for Section 1 🚀
• expected logic is in-place for Section 2 🚀
• expected logic is in-place for Section 3 🚀
• expected logic is in-place for Section 4 🚀
• cat 4 ⚠️
• consistency across tribal, tanf, and ssp 🚀

[ADPennington]

as per our async discussion, i think its worth revisiting the discussion about how many times the cat 4-related error messages appear in the error report, especially in scenarios where these are related to duplicates. this looks great! thanks @elipe17