3106 reparse django action

[raftmsohani]

Summary of Changes

Pull request closes #3106

• Added an action button to data_file admin page for reparsing fiels
• Added a script which creates a modal to confirm action
• Changed reparsing command to perform reparsing on selected number of files

How to Test

1. Open http://localhost:3000/ and sign in.
2. submit a couple of files
3. go to /admin/data_files page
4. choose a couple of files, and use action button to start the reparse
5. you should be redirected to another page to confirm reparsing
6. push confirm to proceed, then look at the logs

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• OFA System Administrators can initiate a reparsing run from DAC
• OFA System Administrators can filter/select datafiles for this action
• Re-Parsing runs follow safety protocols implemented in #3065 whether from cli cmd or DAC
• Ensure Testing Checklist passes successfully
• Restrict the action such that DIGIT can not run re-parse command

• lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • If this PR introduces backend code changes, are they meaningfully tested?
  • If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• Are backend code style checks passing on CircleCI?
• Are frontend code style checks passing on CircleCI?
• Are code maintainability principles being followed?

Deliverable 4: Accessible

• Does this PR complete the epic?
• Are links included to any other gov-approved PRs associated with epic?
• Does PR include documentation for Raft's a11y review?
• Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• Does this PR provide background for why coding decisions were made?
• If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces dependencies, are their licenses documented?
• Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• Does the OWASP Scan pass on CircleCI?
• Do manual code review and manual testing detect any new security issues?
• If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• the purpose of the research
• methods used to conduct the research
• who participated in the research
• what was tested and how
• impact of research on TDP
• (if applicable) final design mockups produced for TDP development

[raftmsohani]

@ADPennington did the change on log message. Please re-review

[ADPennington]

@ADPennington did the change on log message. Please re-review

@raftmsohani when you have some time, Can you get the merge conflicts resolved?

[ADPennington]

@jtimpe I am confirming that this log change works as expected when triggering reparsing from DAC and terminal on a specific files 👍🏾

[ADPennington]

T = reparsing from terminal
D = reparsing from DAC

[ADPennington]

i tested this new option on a subset of files with this command:

python manage.py clean_and_reparse -f "1215,1030,2246,1807,1737,1826,1044"

and this immediately triggered the reparsing process and did not prompt me to confirm that I wanted to reparse X number of files.

vcap@:~$ python manage.py clean_and_reparse -f "1215,1030,2246,1807,1737,1826,1044"
************** reparse all False
************** selected files [1215, 1030, 2246, 1807, 1737, 1826, 1044]


INSIDE FILE COUNTS MATCH:
7, 7, 0




INSIDE FILE COUNTS MATCH:
7, 7, 0


2024-10-17 22:01:51,087 INFO clean_and_reparse.py::_backup:L49 :  Beginning reparse DB Backup.
Beginning reparse DB Backup.
2024-10-17 22:01:51,092 INFO db_backup.py::get_system_values:L54 :  Using postgres client at: /home/vcap/deps/0/apt/usr/lib/postgresql/15/bin/
Using postgres client at: /home/vcap/deps/0/apt/usr/lib/postgresql/15/bin/
2024-10-17 22:01:51,094 INFO db_backup.py::backup_database:L89 :  Executing backup command: /home/vcap/deps/0/apt/usr/lib/postgresql/15/bin/pg_dump -Fc 
...
...

[ADPennington]

i expected something like this:

vcap@:~$ python manage.py clean_and_reparse -y 2025
************** reparse all False
************** selected files None

You have selected to reparse datafiles for FY 2025 and Q1-4. The reparsed files will NOT be stored in new indices and the old indices
These options will delete and reparse (2) datafiles.
Continue [y/n]?

[ADPennington]

per Jan:

-f is only used by the action (which confirms via javascript), which is why the additional conditional checks weren't added

evidence here

[ADPennington]

[ADPennington]

Excellent work @raftmsohani 🚀 testing notes below ⬇️ cc: @ttran-hub

Tests

• via DAC:

  • most recent version of all files submitted for FY21, tribes only.
  • one file
  • selected list of files
  • while one reparsing process was in-progress, triggering another reparse action

• via management command (terminal):

  • all FY2025 files
  • selected list of files (by file id)

Notes

• All of the tests were successful 🚀

• Tests reaffirmed that its possible to trigger a new parsing action while one is in-progress, but it seems better to wait until the initial process is complete (the metamodel id for the new action will not appear until the initial process completes).

• the LogEntries currently includes a list of files by ID to be reparsed, but could use some improvement to its more clear which files have been queued for reparsing. Relatedly, it would be helpful to add a logentry and/or email notification to admins when the reparsing process finished. In the meantime, we should monitor the metamodel metrics.
  

• The following fields in the reparse metamodel are N/A, especially when reparsing from DAC, and may need to be refactored or removed in Re-parse command refactor #3205
  

• Reparsing via terminal seems most useful if we want to reparse all versions of files by fiscal period. It is technically possible to select specific files to be reparsed, but confirmation prompts aren't currently included before the process begins, so not recommended at this time (reference) ⚠️