Re-parse Command

[jtimpe]

Summary of Changes

• Introduced new command that allows an admin user to re-parse a subset or all of the datafiles in the database.
  • Allows admin to specify whether to re-parse in place or to create new indices
  • Creates DB backup
• Created Django command that parametrizes the DB backup routine
  Pull request closes Implement (small) data lifecycle (backup/archive ES) #3004

How to Test

cd tdrs-frontend && docker-compose up --build
cd tdrs-backend && docker-compose up --build

1. Open http://localhost:3000/ and sign in.
2. Submit some datafiles for different program types, fiscal years, and fiscal quarters.
3. Open http://localhost:9200/_cat/indices/?pretty&v&s=index and verify the elastic indices match the DAC
4. Execute the clean_and_reparse Django command a myriad of times with different values for the parameters.

• Between each run of the command verify with the Elastic url above and the DAC that you see what you would expect.

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• Postgres record counts never change/Postgres always remains source of truth
• Elastic indices match with parameters passed to re-parse command
• App can archive/delete live data set
• Testing Checklist has been run and all tests pass
• README is updated, if necessary
• lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • If this PR introduces backend code changes, are they meaningfully tested?
  • If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• Are backend code style checks passing on CircleCI?
• Are frontend code style checks passing on CircleCI?
• Are code maintainability principles being followed?

Deliverable 4: Accessible

• Does this PR complete the epic?
• Are links included to any other gov-approved PRs associated with epic?
• Does PR include documentation for Raft's a11y review?
• Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• Does this PR provide background for why coding decisions were made?
• If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces dependencies, are their licenses documented?
• Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• Does the OWASP Scan pass on CircleCI?
• Do manual code review and manual testing detect any new security issues?
• If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• the purpose of the research
• methods used to conduct the research
• who participated in the research
• what was tested and how
• impact of research on TDP
• (if applicable) final design mockups produced for TDP development

[codecov]

Codecov Report

Attention: Patch coverage is 67.85714% with 9 lines in your changes missing coverage. Please review.

Project coverage is 93.01%. Comparing base (fd5042e) to head (ef3c03d).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3041      +/-   ##
===========================================
- Coverage    93.04%   93.01%   -0.03%     
===========================================
  Files          277      277              
  Lines         7420     7421       +1     
  Branches       657      660       +3     
===========================================
- Hits          6904     6903       -1     
- Misses         414      415       +1     
- Partials       102      103       +1     

Flag	Coverage Δ
dev-backend	93.08% <67.85%> (-0.04%)	⬇️
dev-frontend	92.60% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
tdrs-backend/tdpservice/parsers/util.py	93.57% <100.00%> (+0.04%)	⬆️
tdrs-backend/tdpservice/settings/common.py	99.31% <100.00%> (+<0.01%)	⬆️
tdrs-backend/tdpservice/scheduling/parser_task.py	50.00% <25.00%> (-2.00%)	⬇️
tdrs-backend/tdpservice/parsers/parse.py	85.61% <70.00%> (-0.50%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd5042e...ef3c03d. Read the comment docs.

[andrew-jameson]

Step 4 in the 'How to Test' section needs a little more verbosity and usage examples for QASP unless a markdown document is planned.

[raftmsohani]

LGTM! Tried a couple of edge cases and it worked. Great Work! 🥇

[ADPennington]

Reminder to comment out once qasp-approved.

[ADPennington]

this looks great @elipe17 🚀 thank you!

Notes for awareness and ticket priority-setting @robgendron @lfrohlich @ttran-hub:

• this feature currently operates from CLI. This PR includes documentation for how to run these commands from GFE. Sys admins will pair on this task, for now, but Re-Parse Django Action  #3106 will track feasibility work for being able to trigger re-parsing from the TDP interfaces.

• current options facilitate re-parsing primarily by fiscal period, but Additional datafile filters #3108 will track more options to further customize which files to re-parse (e.g. all tribes, all active files for Washington state, latest version of a file, etc. )

• when the re-parsing command is run, the original submission details (e.g. data submitter, data file) is retained, but the submission date is overwritten. [Re-parse command] Retain original submission date when command runs #3107 will address this.

• the dev team recommends that we hold off on executing this command until Re-parse Meta Model #3064 and Spike - Guarantee Sequential Execution of Re-parse Command #3065 are implemented, which will help ensure that sys admins has more visibility into the status of the re-parsing process to help ensure that the db is in a stable state.