As sys admin, I want to be able to reparse datafile sets

[andrew-jameson]

Description:
Implement ability to re-parse/refresh already-submitted data files and have a lifecycle for this generated data. This is a solution for #2870 and the re-indexing issues.

Acceptance Criteria:

• Manual step documented for triggering task to re-parse all old data files
• System Administrator can delete/re-parse a set of data files
• Admin should be able to update by quarter for all data types.
• Testing Checklist has been run and all tests pass
• README is updated, if necessary

Tasks:

• Build filter for data files submitted before certain date aka prior submission period
• Build out manual task trigger for orchestrating lifecycle, deployments, and re-parsing
• Run Testing Checklist and confirm all tests pass

Notes:

• @ADPennington has mentioned that prod data files from fy22 would be ideal candidates because those are not only un-parsed (submitted before parsing was in prod), but also may not adhere to current validation logic standards/guidance.

Supporting Documentation:

• Please see Spike: As tech lead, I need elastic re-indexing to be automated  #2870
• https://elasticsearch-dsl.readthedocs.io/en/latest/search_dsl.html#delete-by-query

Open Questions:

• Do we want filtering/granularity for a given quarter, program type (tanf vs ssp), full year, etc.? since last reporting period? Alex gave guidance to focus on latest quarter in fy24.
• Do we need to implement drops/archives for all postgres data as well?

[andrew-jameson]

• Build DAC filter on datafiles by quarter
• Add DAC action for 're-parse'
• Implement base unit tests
• Based on list of datafile id's, iterate .delay() calls per id generating new celery tasks -- or do we want by quarter? how does admin input that? shell_plus w/ django cmd?
• Implement a orchestration func that:
  • a) cleans up postgres
  • b) cleans up ES
  • c) resets DFS
  • d) re-calls parse_datafile() w/ param(s)
• Have some check that all datafile ids were handled. how might we handle conflicts if data in db already exists?

[robgendron]

11 points remaining.

[robgendron]

This will be completed in #3004. Andrew's May 15 notes has become basis for #3004. Deemed closed.

[ADPennington]

@lfrohlich we will close this ticket and capture more details in #3004 cc: @robgendron @andrew-jameson