3075-latency-parsing-errors

[raftmsohani]

Summary of Changes

Provide a brief summary of changes
Pull request closes #3075 _

How to Test

List the steps to test the PR
These steps are generic, please adjust as necessary.

cd tdrs-frontend && docker-compose -f docker-compose.yml -f docker-compose.local.yml up -d
cd tdrs-backend && docker-compose -f docker-compose.yml -f docker-compose.local.yml up -d 

1. Open http://localhost:3000/ and sign in.
2. Proceed with functional tests as described herein.
3. Test steps should be captured in the demo GIF(s) and/or screenshots below.

Demo GIF(s) and screenshots for testing procedure

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• [insert ACs here]
• lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • If this PR introduces backend code changes, are they meaningfully tested?
  • If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• Are backend code style checks passing on CircleCI?
• Are frontend code style checks passing on CircleCI?
• Are code maintainability principles being followed?

Deliverable 4: Accessible

• Does this PR complete the epic?
• Are links included to any other gov-approved PRs associated with epic?
• Does PR include documentation for Raft's a11y review?
• Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• Does this PR provide background for why coding decisions were made?
• If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces dependencies, are their licenses documented?
• Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• Does the OWASP Scan pass on CircleCI?
• Do manual code review and manual testing detect any new security issues?
• If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• the purpose of the research
• methods used to conduct the research
• who participated in the research
• what was tested and how
• impact of research on TDP
• (if applicable) final design mockups produced for TDP development

[raftmsohani]

As part of this spike, I investigated the timing of admin page tables, which uses boilerplate django settings to query models and show them in tables.
With models that include more columns, even if we uselist_display to limit columns in the table, django still queries all columns and filters listing columns after the query is done.
As an example, for Parser Error model, we are setting:

list_display = [
        'row_number',
        'field_name',
        'error_type',
        'error_message',
    ]

which is intended to limit the display columns to four listed columns, but the ParserError model has 15 columns. Additionally, one of the fields is JSON field, which generally takes more time to query.

By changing the query set, I could limit the fields in the query to the ones listed for display. This improved the query time by 10-15%.

With data in DB to start growing, we will need to start looking into improving efficiency of querysets using methods descrbed in here

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.66%. Comparing base (265d196) to head (71ce3c7).
Report is 24 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3236   +/-   ##
========================================
  Coverage    90.65%   90.66%           
========================================
  Files          299      299           
  Lines         8490     8493    +3     
  Branches       794      794           
========================================
+ Hits          7697     7700    +3     
  Misses         676      676           
  Partials       117      117           

Flag	Coverage Δ
dev-backend	90.39% <100.00%> (+<0.01%)	⬆️
dev-frontend	92.66% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
tdrs-backend/tdpservice/parsers/admin.py	100.00% <100.00%> (ø)
tdrs-backend/tdpservice/settings/common.py	99.34% <100.00%> (+<0.01%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 265d196...71ce3c7. Read the comment docs.

[raftmsohani]

TODO: remove before merge, only for testing

[raftmsohani]

TODO: this is for my personal environment. change it for your environment if you are planning to look at Sentry yourself.