2411 metadata parsed datafiles

[raftmsohani]

Summary of Changes

Provide a brief summary of changes
Pull request closes #2411 _

How to Test

To test, use the following commands to start the app:

cd tdrs-frontend && docker-compose -f docker-compose.yml -f docker-compose.local.yml up -d
cd tdrs-backend && docker-compose -f docker-compose.yml -f docker-compose.local.yml up -d 

1. Open http://localhost:3000/ and sign in.
2. Load a couple of test files to add datafile errors
3. Make sure you have an admin user account and login to the admin page
4. Proceed to data file page from the left panel

You will see the loaded datafiles and the added two columns as shown below:

Deliverables

Added new 'program type' filter to datafile list page, as well as two new columns:

• DATA FILE SUMMARY
• ERROR REPORT LINK

Deliverable 1: Accepted Features

Checklist of ACs:

• Parsed datafiles are filterable on: STT, Year, Quarter, Program Type
• Admin view of parsed datafiles contain: file status, case totals, err report link, header enc flag, link to STT object, link to originating DataFile
• Clickable obj link is not an incrementing integer

• lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • If this PR introduces backend code changes, are they meaningfully tested?
  • If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• Are backend code style checks passing on CircleCI?
• Are frontend code style checks passing on CircleCI?
• Are code maintainability principles being followed?

Deliverable 4: Accessible

• Does this PR complete the epic?
• Are links included to any other gov-approved PRs associated with epic?
• Does PR include documentation for Raft's a11y review?
• Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• Does this PR provide background for why coding decisions were made?
• If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces dependencies, are their licenses documented?
• Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• Does the OWASP Scan pass on CircleCI?
• Do manual code review and manual testing detect any new security issues?
• If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• the purpose of the research
• methods used to conduct the research
• who participated in the research
• what was tested and how
• impact of research on TDP
• (if applicable) final design mockups produced for TDP development

[andrew-jameson]

Working great for me locally! Let's get those e2e pipeline errors fixed and move this forward, thanks!

[raftmsohani]

Working great for me locally! Let's get those e2e pipeline errors fixed and move this forward, thanks!

@andrew-jameson : thanks for your review Andrew, the tests are now passing

[jtimpe]

bit of a nitpick on the custom filter classes but everything is working as expected.

[jtimpe]

we have a couple more statuses than just Accepted and Rejected, could they be added? also rather than create a custom filter class, you could add 'datafilesummary__status' to the DataFileAdmin's list_filter - i tested that out quickly and it correctly ports all the DataFileSummary.Status options without needing to hard code anything

[jtimpe]

not really specific to this PR, but similar logic is repeated everywhere and i keep thinking we should have a first-class member for the program type. similar to my other comment this would also allow you to use a native django list filter for the program type without creating a custom class

[raftmsohani]

Having prog_type as property means calculation is done in backend server vs DB. While normally querying the DB is faster, there are cases where pushing the query into DB is not efficient since during writing to prog_type need a db_lock. An example is calculated values where property can change base on two, three different fields.
Having said that, in this case since prog_type only needs to be updated during the initial write, I agree with your suggestion

[ADPennington]

looks great @raftmsohani 🚀 especially like how i can obtain the status details of the file from the data_files api endpoint and also how the parsing errors for each file can be easily accessed by clicking the link in DAC. id like to discuss with @ttran-hub if it would be helpful to add more cols to parser errors table. if so, this will be captured in an enhancement ticket.