Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2991 remove SFTP and ACF titan related codes #3003

Merged
merged 5 commits into from
Jun 7, 2024
Merged

2991 remove SFTP and ACF titan related codes #3003

merged 5 commits into from
Jun 7, 2024

Conversation

raftmsohani
Copy link

@raftmsohani raftmsohani commented May 22, 2024 •
edited by ADPennington
Loading

Summary of Changes

Pull request closes #2991 _

PR removes all logic for transferring uploaded files to ACF TITAN via SFTP

How to Test

List the steps to test the PR
These steps are generic, please adjust as necessary.

cd tdrs-frontend && docker-compose -f docker-compose.yml -f docker-compose.local.yml up -d
cd tdrs-backend && docker-compose -f docker-compose.yml -f docker-compose.local.yml up -d
  1. Open http://localhost:3000/ and sign in.
  2. Proceed with functional tests as described herein.
  3. Test steps should be captured in the demo GIF(s) and/or screenshots below.

Demo GIF(s) and screenshots for testing procedure

Screenshot 2024-05-31 at 11 41 37 AM

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

  • files submitted through TDP are not transferred to ACFTitan server.

sftpp2
sftpp1

  • file transfer env vars are no longer present in the codebase, circleci, or backend environments
    sftpp3

  • backend logs do not show evidence of file transfer attempts.

 2024-06-06T12:42:32.28-0400 [APP/PROC/WEB/0] OUT 10.255.141.85 - - [06/Jun/2024:16:42:32 +0000] "GET /v1/stts/alpha HTTP/1.0" 200 36455 "https://tdp-frontend-qasp.app.cloud.gov/data-files" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
   2024-06-06T12:42:34.58-0400 [APP/PROC/WEB/0] ERR [2024-06-06 16:42:34 +0000] [569] [DEBUG] GET /v1/data_files/
   2024-06-06T12:42:34.63-0400 [APP/PROC/WEB/0] OUT 10.255.141.85 - - [06/Jun/2024:16:42:34 +0000] "GET /v1/data_files/?year=2024&quarter=Q1&stt=1&file_type=tanf HTTP/1.0" 200 740 "https://tdp-frontend-qasp.app.cloud.gov/data-files" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
   2024-06-06T12:42:37.47-0400 [APP/PROC/WEB/0] ERR [2024-06-06 16:42:37 +0000] [566] [DEBUG] Closing connection.
   2024-06-06T12:42:38.26-0400 [APP/PROC/WEB/0] ERR [2024-06-06 16:42:38 +0000] [568] [DEBUG] POST /v1/data_files/
   2024-06-06T12:42:38.30-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,309 DEBUG views.py::create:L57 :  DataFileViewSet: <rest_framework.request.Request: POST '/v1/data_files/'>
   2024-06-06T12:42:38.30-0400 [APP/PROC/WEB/0] ERR DataFileViewSet: <rest_framework.request.Request: POST '/v1/data_files/'>
   2024-06-06T12:42:38.31-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,310 DEBUG validators.py::validate_file_infection:L54 :  CLAMAV_NEEDED noted as True, proceeding with scan.
   2024-06-06T12:42:38.31-0400 [APP/PROC/WEB/0] ERR CLAMAV_NEEDED noted as True, proceeding with scan.
   2024-06-06T12:42:38.31-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,310 DEBUG clients.py::__init__:L37 :  Set clamav endpoint_url as 'http://tdp-clamav-nginx-dev.apps.internal:9000/scan'
   2024-06-06T12:42:38.31-0400 [APP/PROC/WEB/0] ERR Set clamav endpoint_url as 'http://tdp-clamav-nginx-dev.apps.internal:9000/scan'
   2024-06-06T12:42:38.31-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,311 DEBUG clients.py::scan_file:L64 :  Initiating virus scan for file: test_staging.txt
   2024-06-06T12:42:38.31-0400 [APP/PROC/WEB/0] ERR Initiating virus scan for file: test_staging.txt
   2024-06-06T12:42:38.32-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,323 DEBUG clients.py::scan_file:L90 :  File scan marked as CLEAN for file: test_staging.txt
   2024-06-06T12:42:38.32-0400 [APP/PROC/WEB/0] ERR File scan marked as CLEAN for file: test_staging.txt
   2024-06-06T12:42:38.61-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,613 DEBUG views.py::create:L64 :  DataFileViewSet: status: 201
   2024-06-06T12:42:38.61-0400 [APP/PROC/WEB/0] ERR DataFileViewSet: status: 201
   2024-06-06T12:42:38.61-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,617 INFO views.py::create:L69 :  Preparing parse task: User META -> user: [email protected], stt: Alabama (01). Datafile META -> datafile: 642, section: Active Case Data, quarter Q1, year 2024.
   2024-06-06T12:42:38.61-0400 [APP/PROC/WEB/0] ERR Preparing parse task: User META -> user: [email protected], stt: Alabama (01). Datafile META -> datafile: 642, section: Active Case Data, quarter Q1, year 2024.
   2024-06-06T12:42:38.65-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,655 INFO views.py::create:L74 :  Submitted parse task to queue for datafile 642.
   2024-06-06T12:42:38.65-0400 [APP/PROC/WEB/0] ERR Submitted parse task to queue for datafile 642.
   2024-06-06T12:42:38.68-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,689 INFO parser_task.py::parse:L25 :  DataFile parsing started for file ADS.E2J.NDM1.TS01
   2024-06-06T12:42:38.68-0400 [APP/PROC/WEB/0] ERR [2024-06-06 16:42:38,689: INFO/ForkPoolWorker-1] DataFile parsing started for file ADS.E2J.NDM1.TS01
   2024-06-06T12:42:38.82-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,822 DEBUG views.py::create:L83 :  DataFileViewSet: return val: <Response status_code=201, "text/html; charset=utf-8">
   2024-06-06T12:42:38.82-0400 [APP/PROC/WEB/0] ERR DataFileViewSet: return val: <Response status_code=201, "text/html; charset=utf-8">
   2024-06-06T12:42:38.82-0400 [APP/PROC/WEB/0] OUT 10.255.141.85 - - [06/Jun/2024:16:42:38 +0000] "POST /v1/data_files/ HTTP/1.0" 201 430 "https://tdp-frontend-qasp.app.cloud.gov/data-files" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
   2024-06-06T12:42:38.83-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,834 INFO row_schema.py::parse_and_validate:L60 :  2 preparser error(s) encountered.
   2024-06-06T12:42:38.83-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,834 INFO parse.py::parse_datafile:L34 :  Preparser Error: 2 header errors encountered.
   2024-06-06T12:42:38.83-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,834 DEBUG parse.py::bulk_create_errors:L150 :  Bulk creating ParserErrors.
   2024-06-06T12:42:38.83-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,839 INFO parse.py::bulk_create_errors:L153 :  Created 2/1 ParserErrors.
   2024-06-06T12:42:38.85-0400 [APP/PROC/WEB/0] ERR 2024-06-06 16:42:38,850 INFO parser_task.py::parse:L38 :  Parsing finished for file -> {id: 642, filename: test_staging.txt, STT: Alabama (01), S3 location: data_files/2024/Q1/1/Active Case Data/test_staging.txt} with status Rejected and 1 errors.
   2024-06-06T12:42:38.85-0400 [APP/PROC/WEB/0] ERR [2024-06-06 16:42:38,850: INFO/ForkPoolWorker-1] Parsing finished for file -> {id: 642, filename: test_staging.txt, STT: Alabama (01), S3 location: data_files/2024/Q1/1/Active Case Data/test_staging.txt} with status Rejected and 1 errors.
   2024-06-06T12:42:38.92-0400 [APP/PROC/WEB/0] ERR [2024-06-06 16:42:38 +0000] [567] [DEBUG] POST /v1/logs/
   2024-06-06T12:42:38.96-0400 [APP/PROC/WEB/0] OUT 10.255.141.85 - - [06/Jun/2024:16:42:38 +0000] "POST /v1/logs/ HTTP/1.0" 200 9 "https://tdp-frontend-qasp.app.clo
  • all other existing TDP features are functioning as expected.
  • Testing Checklist has been run and all tests pass
  • README is updated, if necessary
  • lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

  • Are all areas of code introduced in this PR meaningfully tested?
    • If this PR introduces backend code changes, are they meaningfully tested?
    • If this PR introduces frontend code changes, are they meaningfully tested?
  • Are code coverage minimums met?
    • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
    • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

  • Are backend code style checks passing on CircleCI?
  • Are frontend code style checks passing on CircleCI?
  • Are code maintainability principles being followed?

Deliverable 4: Accessible

  • Does this PR complete the epic?
  • Are links included to any other gov-approved PRs associated with epic?
  • Does PR include documentation for Raft's a11y review?
  • Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

  • Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

  • Does this PR provide background for why coding decisions were made?
  • If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
  • If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
  • If this PR introduces dependencies, are their licenses documented?
  • Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

  • Does the OWASP Scan pass on CircleCI?
  • Do manual code review and manual testing detect any new security issues?
  • If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

  • the purpose of the research
  • methods used to conduct the research
  • who participated in the research
  • what was tested and how
  • impact of research on TDP
  • (if applicable) final design mockups produced for TDP development

@raftmsohani raftmsohani self-assigned this May 22, 2024
@codecov Codecov
Copy link

codecov bot commented May 22, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.10%. Comparing base (00ac3d1) to head (4e92ee0).
Report is 1 commits behind head on develop.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #3003      +/-   ##
===========================================
+ Coverage    93.05%   93.10%   +0.05%     
===========================================
  Files          276      275       -1     
  Lines         7108     7036      -72     
  Branches       599      596       -3     
===========================================
- Hits          6614     6551      -63     
+ Misses         399      391       -8     
+ Partials        95       94       -1
Flag Coverage Δ
dev-backend 93.18% <100.00%> (+0.06%) ⬆️
dev-frontend 92.62% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
tdrs-backend/tdpservice/data_files/views.py 88.99% <100.00%> (-0.20%) ⬇️
tdrs-backend/tdpservice/settings/common.py 99.29% <ø> (-0.03%) ⬇️

... and 1 file with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cc37558...4e92ee0. Read the comment docs.

@raftmsohani raftmsohani added Deploy with CircleCI-raft Deploy to https://tdp-frontend-raft.app.cloud.gov through CircleCI raft review This issue is ready for raft review and removed WIP labels May 31, 2024
elipe17
elipe17 approved these changes Jun 3, 2024
View reviewed changes
Copy link

@elipe17 elipe17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No transfer occurred when submitting datafiles as expected. Approving with the added comment that we still need to handle the env vars during this PRs merge.

jtimpe
jtimpe approved these changes Jun 5, 2024
View reviewed changes
Copy link

@jtimpe jtimpe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

my favorite kind of pr. good work!

@raftmsohani raftmsohani requested review from ADPennington and removed request for andrew-jameson June 6, 2024 14:34
@ADPennington ADPennington added Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI and removed Deploy with CircleCI-raft Deploy to https://tdp-frontend-raft.app.cloud.gov through CircleCI labels Jun 6, 2024
ADPennington
ADPennington approved these changes Jun 6, 2024
View reviewed changes
Copy link
Collaborator

@ADPennington ADPennington left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm @raftmsohani 🥇

  • please delete the env vars from raft's circleci. I have deleted the env vars from qasp env and will also plan to delete the env vars from hhs circleci and staging and prod apps in the next release.

@ADPennington ADPennington added Ready to Merge and removed QASP Review Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI labels Jun 6, 2024
@raftmsohani raftmsohani merged commit 9f2b277 into develop Jun 7, 2024
27 checks passed
@raftmsohani raftmsohani deleted the 2991-depreciate-sftp-file-transfer branch June 7, 2024 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@elipe17 elipe17 elipe17 approved these changes

@jtimpe jtimpe jtimpe approved these changes

@ADPennington ADPennington ADPennington approved these changes

Assignees

@raftmsohani raftmsohani

Labels
backend raft review This issue is ready for raft review Ready to Merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

As tech lead, I need the sftp file transfer feature to be deprecated
4 participants
@raftmsohani @ADPennington @jtimpe @elipe17