Merge branch 'develop' into 1621-descriptive-error-message-page-authe…

…ntication-source-unavailable
raft-tech · Jul 22, 2024 · 9bf0ac7 · 9bf0ac7
2 parents 90b64a6 + fd5042e
commit 9bf0ac7
Show file tree

Hide file tree

Showing 18 changed files with 1,664 additions and 274 deletions.
diff --git a/docs/Sprint-Review/sprint-101-summary.md b/docs/Sprint-Review/sprint-101-summary.md
@@ -0,0 +1,87 @@
+# sprint-101-summary
+
+6/5/2024 - 6/18/2024
+
+**Dev:**
+
+_**Prioritized DAC and Notifications Work**_&#x20;
+
+* As sys admin, I want to be able to reparse datafile sets #2978
+* As a software engineer, I want to be able to test django-admin-508 #3008
+* As tech lead, I need the STT filter for search\_indexes to be updated #2950
+* As a data analyst I want to be notified of approaching data deadlines #2473
+* add `SENDGRID_API_KEY` to deploy.backend.sh #2677
+* Implement (small) data lifecycle (backup/archive ES) #3004
+* As a developer I want to test django-508 repo #2980\
+
+
+**DevOps:**
+
+_**Successful deployments across environments and pipeline stability investments**_
+
+* Application health monitoring #831
+
+**Design:**
+
+_**Close out error guide work, coordinate with dev on a plan for Cat 3 problems introduced by Cat 2 work, support spec-writing for upcoming work, and continued error audit dev ticket refinement.**_
+
+* Error Report Guide #2847 is going through final edits&#x20;
+  * Walk-on Dear Colleague letter link update to this PR (or spin up a separate ticket if deployment of the letter to OFA's website doesn't align to this)
+* Deliver spec for #3014 (Blanked-out values in Submission History)
+* \#3021 Updated KC Release Notes & Update Indicator FAQ - stretch goal for this sprint
+* Write follow-on / spec tickets from #2909 findings - stretch/ongoing lift
+* Category 3 error messages clean-up #2792 - stretch/ongoing lift
+
+## Tickets
+
+### Completed/Merged
+
+* [#2980 As a developer I want to test django-508 repo](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2980)
+* [#2892 Correct misleading error message for unaligned reporting year/q against header year/q](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2892)
+* [#2909 \[Research Spike\] OOtB OFA Kibana Experience & DIGIT Data Access](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2909)
+* [#2991 As tech lead, I need the sftp file transfer feature to be deprecated](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2991)
+* [#2847 \[Design Deliverable\] Error Report Knowledge Center Explainer](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2847)
+* [#3024 2897 follow-on for a11y-related enhancement ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3024)
+* [#2897 As a data analyst I want finalized language and guidance resources in Submission History & Error Reports ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2897)
+
+### Submitted (QASP Review, OCIO Review)
+
+* [#2133 \[Dev\] Enhancement for Request Access form (Tribe discoverability) ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2133)
+* [#3023 as STT approved user, I need my IP address whitelisted so i can access TDP](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3023)
+* [#3000 \[Design Deliverable\] TDP Poster for summer 2024 conferences](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3000)
+* [#2795 As tech lead, I need TDP to detect duplicate records within a file and not store them in the db. ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2795)
+* [#2693 \[Error Audit\] Category 2 error messages clean-up ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2693)
+* [#2801 Friendly name cleanup ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2801)
+* [#2883 Pre-Made Reporting Dashboards on Kibana](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2883)
+* [#3021 \[Design Deliverable\] Updated KC Release Notes & Update Indicator FAQ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3021)
+* [#2954 Extend SESSION\_COOKIE\_AGE](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2954)
+
+### Ready to Merge
+
+*
+
+### Closed (Not Merged)
+
+* [#2491 Create root-level docker-compose configuration file(s)](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2491)
+* [#1690 As a system admin, I need a way to be redirected to frontend from DAC](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1690)
+* [#2351 As a user I want to be notified when the files are being scanned or uploaded when I push upload button](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2351)
+* [#2591 Allow `manage.py` commands to be run by circleci](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2591)
+
+### Moved to Next Sprint&#x20;
+
+**In Progress**&#x20;
+
+* [#3004 Implement (small) data lifecycle (backup/archive ES)](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3004)
+* [#831 \[Spike\] As a Tech Lead, I want to get alerts when there is a backend or frontend error that affects an STT user ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/831)
+* [#2978 As sys admin, I want to be able to reparse datafile sets](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2978)
+
+#### Blocked
+
+*
+
+**Raft Review**
+
+* [#2950 As tech lead, I need the STT filter for search\_indexes to be updated ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2950)
+* [#3008 As a software engineer, I want to be able to test django-admin-508](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3008)
+* [#3016 Spike - Cat2 Validator Improvement](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3016)
+* [#2473  As a data analyst I want to be notified of approaching data deadlines](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2473)
diff --git a/tdrs-backend/tdpservice/data_files/test/test_api.py b/tdrs-backend/tdpservice/data_files/test/test_api.py
@@ -100,10 +100,10 @@ def assert_error_report_tanf_file_content_matches_with_friendly_names(response):
 
         assert ws.cell(row=1, column=1).value == "Please refer to the most recent versions of the coding " \
             + "instructions (linked below) when looking up items and allowable values during the data revision process"
-        assert ws.cell(row=8, column=COL_ERROR_MESSAGE).value == ("if Cash Amount :873 validator1 passed then Cash and "
-                                                                  "Cash Equivalents: Number of Months T1 Item -1 (Cash "
-                                                                  "and Cash Equivalents: Number of Months): 0 is not "
-                                                                  "larger than 0.")
+        assert ws.cell(row=8, column=COL_ERROR_MESSAGE).value == (
+            "if Cash Amount :873 validator1 passed then Item 21B "
+            "(Cash and Cash Equivalents: Number of Months) 0 is not larger than 0."
+        )
 
     @staticmethod
     def assert_error_report_ssp_file_content_matches_with_friendly_names(response):
@@ -134,9 +134,10 @@ def assert_error_report_file_content_matches_without_friendly_names(response):
 
         assert ws.cell(row=1, column=1).value == "Please refer to the most recent versions of the coding " \
             + "instructions (linked below) when looking up items and allowable values during the data revision process"
-        assert ws.cell(row=8, column=COL_ERROR_MESSAGE).value == ("if CASH_AMOUNT :873 validator1 passed then "
-                                                                  "NBR_MONTHS T1 Item -1 (NBR_MONTHS): 0 is not "
-                                                                  "larger than 0.")
+        assert ws.cell(row=8, column=COL_ERROR_MESSAGE).value == (
+            "if CASH_AMOUNT :873 validator1 passed then Item 21B "
+            "(Cash and Cash Equivalents: Number of Months) 0 is not larger than 0."
+        )
 
     @staticmethod
     def assert_data_file_exists(data_file_data, version, user):

diff --git a/tdrs-backend/tdpservice/email/email_enums.py b/tdrs-backend/tdpservice/email/email_enums.py
@@ -13,3 +13,4 @@ class EmailType(Enum):
     REQUEST_DENIED = 'request-denied.html'
     DEACTIVATION_WARNING = 'account-deactivation-warning.html'
     ACCOUNT_DEACTIVATED = 'account-deactivated.html'
+    UPCOMING_SUBMISSION_DEADLINE = 'upcoming-submission-deadline.html'
diff --git a/tdrs-backend/tdpservice/email/tasks.py b/tdrs-backend/tdpservice/email/tasks.py
@@ -11,6 +11,11 @@
 import logging
 from tdpservice.email.helpers.account_access_requests import send_num_access_requests_email
 from tdpservice.email.helpers.account_deactivation_warning import send_deactivation_warning_email
+from tdpservice.stts.models import STT
+from tdpservice.data_files.models import DataFile
+from tdpservice.email.email import automated_email, log
+from tdpservice.email.email_enums import EmailType
+from tdpservice.parsers.util import calendar_to_fiscal
 
 
 logger = logging.getLogger(__name__)
@@ -75,3 +80,84 @@ def email_admin_num_access_requests():
                                    subject,
                                    email_context,
                                    )
+
+
+@shared_task
+def send_data_submission_reminder(due_date, reporting_period, fiscal_quarter):
+    """Send all Data Analysts a reminder to submit if they have not already."""
+    now = datetime.now()
+    fiscal_year = calendar_to_fiscal(now.year, fiscal_quarter)
+
+    all_locations = STT.objects.all()
+
+    reminder_locations = []
+    year_quarter_files = DataFile.objects.all().filter(
+        year=fiscal_year,
+        quarter=fiscal_quarter,
+    )
+
+    for loc in all_locations:
+        submitted_sections = year_quarter_files.filter(stt=loc).values_list('section', flat=True).distinct()
+        required_sections = loc.filenames.keys()
+
+        submitted_all_sections = True
+        for s in required_sections:
+            if s not in submitted_sections:
+                submitted_all_sections = False
+
+        if not submitted_all_sections:
+            reminder_locations.append(loc)
+
+    template_path = EmailType.UPCOMING_SUBMISSION_DEADLINE.value
+    text_message = f'Your datafiles are due by {due_date}.'
+
+    all_data_analysts = User.objects.all().filter(
+        account_approval_status=AccountApprovalStatusChoices.APPROVED,
+        groups=Group.objects.get(name='Data Analyst')
+    )
+
+    for loc in reminder_locations:
+        tanf_ssp_label = 'TANF and SSP' if loc.ssp else 'TANF'
+        subject = f'Action Requested: Please submit your {tanf_ssp_label} data files'
+
+        recipients = all_data_analysts.filter(stt=loc)
+
+        for rec in recipients:
+            context = {
+                'first_name': rec.first_name,
+                'fiscal_year': fiscal_year,
+                'fiscal_quarter': fiscal_quarter,
+                'submission_deadline': due_date,
+                'url': settings.FRONTEND_BASE_URL,
+                'subject': subject
+            }
+
+            logger_context = {
+                'user_id': rec.id,
+                'object_id': rec.id,
+                'object_repr': rec.username,
+            }
+
+            automated_email(
+                email_path=template_path,
+                recipient_email=rec.username,
+                subject=subject,
+                email_context=context,
+                text_message=text_message,
+                logger_context=logger_context
+            )
+
+        if len(recipients) == 0:
+            system_user, created = User.objects.get_or_create(username='system')
+            if created:
+                log('Created reserved system user.')
+
+            logger_context = {
+                'user_id': system_user.pk,
+                'object_id': loc.id,
+                'object_repr': loc.name,
+            }
+            log(
+                f"{loc.name} has no recipients for data submission deadline reminder.",
+                logger_context=logger_context if not settings.DEBUG else None
+            )
diff --git a/tdrs-backend/tdpservice/email/templates/upcoming-submission-deadline.html b/tdrs-backend/tdpservice/email/templates/upcoming-submission-deadline.html
@@ -0,0 +1,27 @@
+{% extends 'base.html' %}
+{% block content %}
+
+<!-- Body copy -->
+<p>Hello {{ first_name }},</p>
+<p>This is a friendly reminder that your data files are due in 5 days. </p>
+<p>Please sign in to the <a href="{{ url }}" target="_blank">TANF Data Portal</a> 
+to upload and submit data files for <b>Fiscal Year {{ fiscal_year }} - {{ fiscal_quarter}}</b> by <b>{{ submission_deadline }}.</b></p>
+
+<!-- This link uses descriptive text to inform the user what will happen with the link is tapped. -->
+<!-- It also uses inline styles since some email clients won't render embedded styles from the head. -->
+<!-- <a href="" style="color: #336a90; text-decoration: underline;">Reset your password now</a> -->
+<p><a class="button" href="{{ url }}" style="background-color:#336a90;border-radius:4px;color:#ffffff;display:inline-block;font-family:sans-serif;font-size:18px;font-weight:bold;line-height:60px;text-align:center;text-decoration:none;width: auto; padding-left: 24px; padding-right: 24px;-webkit-text-size-adjust:none;">Submit your data files</a></p>
+
+<b>Need help?</b><br>
+We're here for you! Check out the <a href="http://tdp-project-updates.app.cloud.gov/knowledge-center/" target="_blank">TDP Knowledge Center</a> for specific gudiance on  <a href="https://tdp-project-updates.app.cloud.gov/knowledge-center/uploading-data.html" target="_blank">Submitting Data Files</a> and <a href="https://tdp-project-updates.app.cloud.gov/knowledge-center/faq.html">Frequently Asked Questions</a>.
+<p> 
+TDP is now the only method for data submissions; you should not be submitting data through SFTP or Cyberfusion. Please reach out to the TDP support team at <a href="mailto:[email protected]" target="_blank">[email protected]</a> if you have any questions or need assistance.
+<p>Thank you,</p>
+<p>The TDP Team</p>
+
+
+<!-- This link uses descriptive text to inform the user what will happen with the link is tapped. -->
+<!-- It also uses inline styles since some email clients won't render embedded styles from the head. -->
+<!-- <a href="" style="color: #336a90; text-decoration: underline;">Reset your password now</a> -->
+
+{% endblock %}
diff --git a/tdrs-backend/tdpservice/email/test/test_upcoming_deadline_email.py b/tdrs-backend/tdpservice/email/test/test_upcoming_deadline_email.py
@@ -0,0 +1,134 @@
+"""Test function for sending upcoming data deadline reminders."""
+import pytest
+from datetime import datetime
+from django.core import mail
+from tdpservice.email.tasks import send_data_submission_reminder
+from django.contrib.auth.models import Group
+from tdpservice.stts.models import STT
+from tdpservice.users.models import User
+from tdpservice.data_files.models import DataFile
+
+
+@pytest.mark.parametrize('due_date, reporting_period, fiscal_quarter', [
+    ('February 14', 'Oct - Dec', 'Q1'),
+    ('May 15th', 'Jan - Mar', 'Q2'),
+    ('August 14th', 'Apr - Jun', 'Q3'),
+    ('November 14th', 'Jul - Sep', 'Q4'),
+])
+@pytest.mark.django_db
+def test_upcoming_deadline_sends_no_sections_submitted(
+        due_date, reporting_period, fiscal_quarter):
+    """Test that the send_deactivation_warning_email function runs when no sections have been submitted."""
+    stt = STT.objects.create(
+        name='Arkansas',
+        filenames={
+            "Active Case Data": "test-filename.txt",
+            "Closed Case Data": "test-filename-closed.txt",
+        }
+    )
+
+    data_analyst = User.objects.create(
+        username='[email protected]',
+        stt=stt,
+        account_approval_status='Approved'
+    )
+    data_analyst.groups.add(Group.objects.get(name='Data Analyst'))
+    data_analyst.save()
+
+    send_data_submission_reminder(due_date, reporting_period, fiscal_quarter)
+
+    assert len(mail.outbox) == 1
+    assert mail.outbox[0].subject == "Action Requested: Please submit your TANF data files"
+
+
+@pytest.mark.parametrize('due_date, reporting_period, fiscal_quarter', [
+    ('February 14', 'Oct - Dec', 'Q1'),
+    ('May 15th', 'Jan - Mar', 'Q2'),
+    ('August 14th', 'Apr - Jun', 'Q3'),
+    ('November 14th', 'Jul - Sep', 'Q4'),
+])
+@pytest.mark.django_db
+def test_upcoming_deadline_sends_some_sections_submitted(
+        due_date, reporting_period, fiscal_quarter):
+    """Test that the send_deactivation_warning_email function runs when some sections have been submitted."""
+    stt = STT.objects.create(
+        name='Arkansas',
+        filenames={
+            "Active Case Data": "test-filename.txt",
+            "Closed Case Data": "test-filename-closed.txt",
+        }
+    )
+
+    data_analyst = User.objects.create(
+        username='[email protected]',
+        stt=stt,
+        account_approval_status='Approved'
+    )
+    data_analyst.groups.add(Group.objects.get(name='Data Analyst'))
+    data_analyst.save()
+
+    now = datetime.now()
+    fiscal_year = now.year - 1 if fiscal_quarter == 'Q1' else now.year
+
+    _ = DataFile.create_new_version({
+        "section": 'Active Case Data',
+        "quarter": fiscal_quarter,
+        "year": fiscal_year,
+        "stt": stt,
+        "user": data_analyst,
+    })
+
+    send_data_submission_reminder(due_date, reporting_period, fiscal_quarter)
+
+    assert len(mail.outbox) == 1
+    assert mail.outbox[0].subject == "Action Requested: Please submit your TANF data files"
+
+
+@pytest.mark.parametrize('due_date, reporting_period, fiscal_quarter', [
+    ('February 14', 'Oct - Dec', 'Q1'),
+    ('May 15th', 'Jan - Mar', 'Q2'),
+    ('August 14th', 'Apr - Jun', 'Q3'),
+    ('November 14th', 'Jul - Sep', 'Q4'),
+])
+@pytest.mark.django_db
+def test_upcoming_deadline_no_send_when_all_sections_complete(
+        due_date, reporting_period, fiscal_quarter):
+    """Test that the send_deactivation_warning_email function does not run when all sections have been submitted."""
+    stt = STT.objects.create(
+        name='Arkansas',
+        filenames={
+            "Active Case Data": "test-filename.txt",
+            "Closed Case Data": "test-filename-closed.txt",
+        }
+    )
+
+    data_analyst = User.objects.create(
+        username='[email protected]',
+        stt=stt,
+        account_approval_status='Approved'
+    )
+    data_analyst.groups.add(Group.objects.get(name='Data Analyst'))
+    data_analyst.save()
+
+    now = datetime.now()
+    fiscal_year = now.year - 1 if fiscal_quarter == 'Q1' else now.year
+
+    _ = DataFile.create_new_version({
+        "section": 'Active Case Data',
+        "quarter": fiscal_quarter,
+        "year": fiscal_year,
+        "stt": stt,
+        "user": data_analyst,
+    })
+
+    _ = DataFile.create_new_version({
+        "section": 'Closed Case Data',
+        "quarter": fiscal_quarter,
+        "year": fiscal_year,
+        "stt": stt,
+        "user": data_analyst,
+    })
+
+    send_data_submission_reminder(due_date, reporting_period, fiscal_quarter)
+
+    assert len(mail.outbox) == 0