Singular ClamAV instance #2660

George-Hudson · 2023-08-14T19:54:14Z

…CAN enpoint to tanf-prod clam av scanner only

Summary of Changes

Provide a brief summary of changes
Pull request closes #2429 _

How to Test

Deploy the branch to one of the dev/staging instances and then follow the steps below:

Open the app and sign in.
Proceed with registration (if necessary)
Once you have access to submit file, do so with one of the correct test files.

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

[insert ACs here]
lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

Are all areas of code introduced in this PR meaningfully tested?
- If this PR introduces backend code changes, are they meaningfully tested?
- If this PR introduces frontend code changes, are they meaningfully tested?
Are code coverage minimums met?
- Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
- Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

Are backend code style checks passing on CircleCI?
Are frontend code style checks passing on CircleCI?
Are code maintainability principles being followed?

Deliverable 4: Accessible

Does this PR complete the epic?
Are links included to any other gov-approved PRs associated with epic?
Does PR include documentation for Raft's a11y review?
Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

Does this PR provide background for why coding decisions were made?
If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
If this PR introduces dependencies, are their licenses documented?
Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

Does the OWASP Scan pass on CircleCI?
Do manual code review and manual testing detect any new security issues?
If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

the purpose of the research
methods used to conduct the research
who participated in the research
what was tested and how
impact of research on TDP
(if applicable) final design mockups produced for TDP development

…CAN enpoint to tanf-prod clam av scanner only

codecov · 2023-08-14T20:04:45Z

Codecov Report

Merging #2660 (9e2c437) into develop (8ec31f9) will decrease coverage by 0.25%.
Report is 41 commits behind head on develop.
The diff coverage is 92.48%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2660      +/-   ##
===========================================
- Coverage    92.99%   92.75%   -0.25%     
===========================================
  Files          219      235      +16     
  Lines         4482     5201     +719     
  Branches       385      452      +67     
===========================================
+ Hits          4168     4824     +656     
- Misses         242      290      +48     
- Partials        72       87      +15

Flag	Coverage Δ
dev-backend	`92.57% <92.48%> (-0.27%)`	⬇️
dev-frontend	`93.59% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
tdrs-backend/tdpservice/core/logger.py	`100.00% <100.00%> (ø)`
...rs-backend/tdpservice/data_files/test/factories.py	`100.00% <100.00%> (ø)`
tdrs-backend/tdpservice/data_files/views.py	`90.65% <100.00%> (+0.08%)`	⬆️
tdrs-backend/tdpservice/email/email.py	`76.78% <100.00%> (-1.55%)`	⬇️
...vice/email/helpers/account_deactivation_warning.py	`100.00% <100.00%> (ø)`
...backend/tdpservice/email/helpers/account_status.py	`84.84% <100.00%> (ø)`
tdrs-backend/tdpservice/email/helpers/data_file.py	`100.00% <100.00%> (ø)`
tdrs-backend/tdpservice/parsers/admin.py	`100.00% <100.00%> (ø)`
...rs/migrations/0002_alter_parsererror_error_type.py	`100.00% <ø> (ø)`
...vice/parsers/migrations/0006_auto_20230810_1500.py	`100.00% <100.00%> (ø)`
... and 42 more

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8289c01...9e2c437. Read the comment docs.

andrew-jameson

I believe we need to ensure network-policy are added that enables non-prod instances to talk to the prod clamav. Should only be a couple lines in deploy-backend.sh.

George-Hudson · 2023-08-22T15:40:44Z

@andrew-jameson
Not sure this is the way. We are currently not using the add-network-policy to establish these container to container connections, instead are designating an http endpoint to hit instead. Shouldn't we instead allow traffic coming from our different spaces into the prod ClamAV server?

ADPennington · 2023-08-22T16:50:53Z

Question: does this require us to share prod space keys with dev? @George-Hudson

George-Hudson · 2023-08-23T15:30:18Z

Question: does this require us to share prod space keys with dev? @George-Hudson

@ADPennington @andrew-jameson that is unclear. The documentation does not mention it and it doesn't make sense to me to do it this way. I would think it's more of a dns routing issue and less of an internal networking issue. We currently don't establish a network policy in order to connect to in-space clamAV servers, so I am not sure if we need it to reach a different-cf-space clamAV server.

…prod clamav

raftmsohani · 2023-08-28T14:01:56Z

The devspace apps need to be able to connect to prod clamav, therefore we need to add network policy for all instances and open up their connection protocol and port using the command:

cf add-network-policy SOURCE_APP --destination-app DESTINATION_APP [-s DESTINATION_SPACE_NAME [-o DESTINATION_ORG_NAME]] [--protocol (tcp | udp) --port RANGE]

e.g.: cf add-network-policy tdp-backend-sandbox clamav-rest -s tanf-prod --protocol tcp --port 9000

raftmsohani · 2023-09-22T13:45:06Z

tdrs-backend/clamav-router/README.md

@@ -0,0 +1,49 @@
+# CLAMAV 


Need to consolidate this with the README file in /Technical-Documentation

tdrs-backend/manifest.yml

raftmsohani · 2023-09-22T14:17:41Z

scripts/deploy-backend.sh

@@ -107,6 +106,17 @@ update_backend()
    # Add network policy to allow frontend to access backend
    cf add-network-policy "$CGAPPNAME_FRONTEND" "$CGAPPNAME_BACKEND" --protocol tcp --port 8080

+    if ["$CF_SPACE" = "tanf-prod" ]; then
+      # Add network policy to allow backend to access tanf-prod services
+      cf add-network-policy "$CGAPPNAME_BACKEND" clamav-rest --protocol tcp --port 9000


Question: Do we need to set the AV_SCAN_URL in prod?

we definitely need the env var in prod

raftmsohani · 2023-09-22T14:21:25Z

scripts/deploy-backend.sh

@@ -40,7 +40,6 @@ set_cf_envs()
  "AMS_CLIENT_ID"
  "AMS_CLIENT_SECRET"
  "AMS_CONFIGURATION_ENDPOINT"
-  "AV_SCAN_URL"


The env var cannot come from outside envs since it is dynamic

…F-app into 2429/single-clamav

raftmsohani · 2023-09-25T15:33:45Z

tdrs-backend/clamav-router/nginx.conf

+    }
+    server {
+        listen 9000;
+        location /scan {


This opens a route to clamav prod, we might want to add a note somewhere for revising this later on. Although all traffic is from cloud.gov space, but they are from a space out of prod

…to 2429/single-clamav

raftmsohani · 2023-09-28T19:22:12Z

Testing this on cloud.gov, it looks like cloud.gov doesn't allow opening any port in NGINX other than default 8080. Still investigating to see if there is any other way to do this

.circleci/deployment/commands.yml

tdrs-backend/tdpservice/settings/common.py

…cies in the deploy-backend.sh script

George-Hudson · 2023-09-29T16:12:57Z

GitHub isn't showing the changes to scripts/deploy-backend.sh for this PR

https://github.com/raft-tech/TANF-app/blob/9e2c43748bac90c7043693905c0b3d2f5f68009e/scripts/deploy-backend.sh#L110C5-L122C2

going to close this and remake

removed clamav from being deployed unless in production. Pointed AV S…

99531e7

…CAN enpoint to tanf-prod clam av scanner only

George-Hudson added the Deploy with CircleCI-sandbox label Aug 14, 2023

andrew-jameson requested changes Aug 17, 2023

View reviewed changes

added network policy to connect to prod clamAV

43527f5

George-Hudson added Deploy with CircleCI-sandbox and removed Deploy with CircleCI-sandbox labels Aug 23, 2023

use prod clam av in cloudgov.py too

58cf0d1

George-Hudson added Deploy with CircleCI-sandbox and removed Deploy with CircleCI-sandbox labels Aug 25, 2023

George Hudson added 2 commits August 25, 2023 11:17

Merge branch 'develop' into 2429/single-clamav

4a55e18

fixing 'f-string is missing placeholders' now that we are hardcoding …

8c98ce2

…prod clamav

George-Hudson added Deploy with CircleCI-sandbox and removed Deploy with CircleCI-sandbox labels Aug 25, 2023

added prod prefix back in

006d60d

raftmsohani reviewed Sep 22, 2023

View reviewed changes

tdrs-backend/manifest.yml Show resolved Hide resolved

raftmsohani reviewed Sep 22, 2023

View reviewed changes

George Hudson added 2 commits September 22, 2023 09:59

Merge branch 'develop' into 2429/single-clamav

acf6814

readd AV_SCAN_URL

9b72df8

George-Hudson requested a review from raftmsohani September 22, 2023 18:00

raftmsohani added 3 commits September 24, 2023 15:18

changed the inline comment in common.py settings

ee6c265

Merge branch '2429/single-clamav' of https://github.com/raft-tech/TAN…

d695c41

…F-app into 2429/single-clamav

README file rewrite

3c6467e

raftmsohani reviewed Sep 25, 2023

View reviewed changes

George-Hudson added raft review This issue is ready for raft review and removed Deploy with CircleCI-sandbox labels Sep 25, 2023

George Hudson and others added 3 commits September 25, 2023 12:05

added note

19ee517

Merge branch '2429/single-clamav' of github.com:raft-tech/TANF-app in…

79c1f85

…to 2429/single-clamav

Merge branch 'develop' into 2429/single-clamav

1f39173

raftmsohani added Deploy with CircleCI-raft Deploy to https://tdp-frontend-raft.app.cloud.gov through CircleCI and removed Deploy with CircleCI-raft Deploy to https://tdp-frontend-raft.app.cloud.gov through CircleCI labels Sep 28, 2023

Merge branch 'develop' into 2429/single-clamav

714bbc8

raftmsohani added Deploy with CircleCI-raft Deploy to https://tdp-frontend-raft.app.cloud.gov through CircleCI and removed Deploy with CircleCI-raft Deploy to https://tdp-frontend-raft.app.cloud.gov through CircleCI labels Sep 28, 2023

raftmsohani reviewed Sep 28, 2023

View reviewed changes

.circleci/deployment/commands.yml Outdated Show resolved Hide resolved

raftmsohani reviewed Sep 28, 2023

View reviewed changes

tdrs-backend/tdpservice/settings/common.py Show resolved Hide resolved

George Hudson added 2 commits September 29, 2023 08:54

readded clam av nginx router url to manifest and setting network poli…

16ab89f

…cies in the deploy-backend.sh script

updated for using backend to set ENV for ClamAV URL.

9e2c437

George-Hudson closed this Sep 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Singular ClamAV instance #2660

Singular ClamAV instance #2660

George-Hudson commented Aug 14, 2023 •

edited by raftmsohani

Loading

codecov bot commented Aug 14, 2023 •

edited

Loading

andrew-jameson left a comment

George-Hudson commented Aug 22, 2023 •

edited

Loading

ADPennington commented Aug 22, 2023

George-Hudson commented Aug 23, 2023 •

edited

Loading

raftmsohani commented Aug 28, 2023 •

edited

Loading

raftmsohani Sep 22, 2023

raftmsohani Sep 22, 2023

raftmsohani Sep 29, 2023

raftmsohani Sep 22, 2023

raftmsohani Sep 25, 2023

raftmsohani commented Sep 28, 2023

George-Hudson commented Sep 29, 2023

Singular ClamAV instance #2660

Singular ClamAV instance #2660

Conversation

George-Hudson commented Aug 14, 2023 • edited by raftmsohani Loading

Summary of Changes

How to Test

Deliverables

codecov bot commented Aug 14, 2023 • edited Loading

Codecov Report

andrew-jameson left a comment

Choose a reason for hiding this comment

George-Hudson commented Aug 22, 2023 • edited Loading

ADPennington commented Aug 22, 2023

George-Hudson commented Aug 23, 2023 • edited Loading

raftmsohani commented Aug 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raftmsohani commented Sep 28, 2023

George-Hudson commented Sep 29, 2023

George-Hudson commented Aug 14, 2023 •

edited by raftmsohani

Loading

codecov bot commented Aug 14, 2023 •

edited

Loading

George-Hudson commented Aug 22, 2023 •

edited

Loading

George-Hudson commented Aug 23, 2023 •

edited

Loading

raftmsohani commented Aug 28, 2023 •

edited

Loading