Merge pull request #481 from raft-tech/release/v3.1.5-Sprint-82

Release/v3.1.5 sprint 82

.circleci/build-and-test/jobs.yml

@@ -5,14 +5,14 @@
       - checkout
       - docker-compose-check
       - docker-compose-up-backend
-      - run:
-          name: Execute Python Linting Test
-          command: cd tdrs-backend; docker-compose run --rm web bash -c "flake8 ."
       - run:
           name: Run Unit Tests And Create Code Coverage Report
           command: |
             cd tdrs-backend;
             docker-compose run --rm web bash -c "./wait_for_services.sh && pytest --cov-report=xml"
+      - run:
+          name: Execute Python Linting Test
+          command: cd tdrs-backend; docker-compose run --rm web bash -c "flake8 ."
       - upload-codecov:
           component: backend
           coverage-report: ./tdrs-backend/coverage.xml

.circleci/config.yml

@@ -82,5 +82,5 @@ workflows:
                 - develop
                 - main
                 - master
-                - /^release.*/      
-      
+                - /^release.*/
+

docs/Sprint-Review/sprint-80-summary.md

@@ -0,0 +1,51 @@
+# Sprint 80 Summary
+
+08/16/23 - 08/29/23
+
+Velocity: Dev (20)
+
+## Sprint Goal
+* Continue parsing engine development for TANF Sections (01-04), complete decoupling backend application spike and continue integration test epic (2282). 
+* UX to continue regional staff research, service design blueprint (.1 and .2) and bridge onboarding to >85% of total users
+* DevOps to investigate nightlyscan issues and resolve utlity images for CircleCI and container registry.
+
+
+## Tickets
+### Completed/Merged
+* [#2369 As tech lead, we need the parsing engine to run quailty checks across TANF section 1](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2369)
+* [#1110 TANF (03) Parsing and Validation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1110)
+* [#2282 As tech lead, I want a file upload integration test](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2282)
+* [#1784 - Email Relay](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1784)
+
+### Ready to Merge
+* N/A
+
+### Submitted (QASP Review, OCIO Review)
+* [#1109 TANF (02) Parsing and Validation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1109)
+
+### Closed (not merged)
+* N/A
+
+## Moved to Next Sprint (Blocked, Raft Review, In Progress)
+### In Progress
+* [#2116 Container Registry Creation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2116)
+* [#2429 Singular ClamAV scanner](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2429)
+* [#1111 TANF (04) Parsing and Validation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1111)
+
+
+### Blocked
+* N/A
+
+
+### Raft Review
+* [#1610 As a user, I need information about the acceptance of my data and a link for the error report](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1610)
+* [#1612 Detailed case level metadata](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1612)
+* [#1613 As a developer, I need parsed file meta data (TANF Section 1)](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/board)
+* [#2626 (Spike) improve parsing logging](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2626)
+
+### Demo
+* Internal:
+    * 2369 / 1110 - TANF Sections (01 and 03) Parsing and Validation
+* External:
+    * 2369 / 1110 - TANF Sections (01 and 03) Parsing and Validation
+

scripts/zap-scanner.sh

@@ -139,7 +139,6 @@ ZAP_CLI_OPTIONS="\
   -config globalexcludeurl.url_list.url\(21\).regex='^https:\/\/.*\.identitysandbox.gov\/.*$' \
   -config globalexcludeurl.url_list.url\(21\).description='Site - IdentitySandbox.gov' \
   -config globalexcludeurl.url_list.url\(21\).enabled=true \
-
   -config spider.postform=true"
 
 # How long ZAP will crawl the app with the spider process

tdrs-backend/docker-compose.local.yml

@@ -80,7 +80,7 @@ services:
     build: .
     command: >
       bash -c "./wait_for_services.sh &&
-               ./gunicorn_start.sh && 
+               ./gunicorn_start.sh &&
                celery -A tdpservice.settings worker -l info"
     ports:
       - "5555:5555"
@@ -106,5 +106,5 @@ volumes:
 
 networks:
   default:
-    external:
-      name: external-net
+    name: external-net
+    external: true

tdrs-backend/docker-compose.yml

@@ -100,6 +100,7 @@ services:
       ./gunicorn_start.sh && celery -A tdpservice.settings worker -l info"
     ports:
       - "5555:5555"
+    tty: true
     depends_on:
       - clamav-rest
       - localstack
@@ -123,5 +124,5 @@ volumes:
 
 networks:
   default:
-    external:
       name: external-net
+      external: true

tdrs-backend/tdpservice/core/logger.py

@@ -0,0 +1,38 @@
+"""Contains core logging functionality for TDP."""
+
+import logging
+
+class ColorFormatter(logging.Formatter):
+    """Simple formatter class to add color to log messages based on log level."""
+
+    BLACK = '\033[0;30m'
+    RED = '\033[0;31m'
+    GREEN = '\033[0;32m'
+    BROWN = '\033[0;33m'
+    BLUE = '\033[0;34m'
+    PURPLE = '\033[0;35m'
+    CYAN = '\033[0;36m'
+    GREY = '\033[0;37m'
+
+    DARK_GREY = '\033[1;30m'
+    LIGHT_RED = '\033[1;31m'
+    LIGHT_GREEN = '\033[1;32m'
+    YELLOW = '\033[1;33m'
+    LIGHT_BLUE = '\033[1;34m'
+    LIGHT_PURPLE = '\033[1;35m'
+    LIGHT_CYAN = '\033[1;36m'
+    WHITE = '\033[1;37m'
+
+    RESET = "\033[0m"
+
+    def __init__(self, *args, **kwargs):
+        self._colors = {logging.DEBUG: self.CYAN,
+                        logging.INFO: self.GREEN,
+                        logging.WARNING: self.YELLOW,
+                        logging.ERROR: self.LIGHT_RED,
+                        logging.CRITICAL: self.RED}
+        super(ColorFormatter, self).__init__(*args, **kwargs)
+
+    def format(self, record):
+        """Format the record to be colored based on the log level."""
+        return self._colors.get(record.levelno, self.WHITE) + super().format(record) + self.RESET

tdrs-backend/tdpservice/data_files/models.py

@@ -221,6 +221,15 @@ def find_latest_version(self, year, quarter, section, stt):
             version=version, year=year, quarter=quarter, section=section, stt=stt,
         ).first()
 
+    def __repr__(self):
+        """Return a string representation of the model."""
+        return f"{{id: {self.id}, filename: {self.original_filename}, STT: {self.stt}, S3 location: " + \
+               f"{self.s3_location}}}"
+
+    def __str__(self):
+        """Return a string representation of the model."""
+        return f"filename: {self.original_filename}"
+
 class LegacyFileTransferManager(models.Manager):
     """Extends object manager functionality for LegacyFileTransfer model."""
 

tdrs-backend/tdpservice/data_files/test/factories.py

@@ -18,7 +18,7 @@ class Meta:
     extension = "txt"
     section = "Active Case Data"
     quarter = "Q1"
-    year = "2020"
+    year = 2020
     version = 1
     user = factory.SubFactory(UserFactory)
     stt = factory.SubFactory(STTFactory)

tdrs-backend/tdpservice/data_files/views.py

@@ -66,6 +66,10 @@ def create(self, request, *args, **kwargs):
             data_file_id = response.data.get('id')
             data_file = DataFile.objects.get(id=data_file_id)
 
+            logger.info(f"Preparing parse task: User META -> user: {request.user}, stt: {data_file.stt}. " +
+                        f"Datafile META -> datafile: {data_file_id}, section: {data_file.section}, " +
+                        f"quarter {data_file.quarter}, year {data_file.year}.")
+
             parser_task.parse.delay(data_file_id)
             logger.info("Submitted parse task to queue for datafile %s.", data_file_id)
 

tdrs-backend/tdpservice/parsers/admin.py

@@ -15,4 +15,11 @@ class ParserErrorAdmin(admin.ModelAdmin):
     ]
 
 
+class DataFileSummaryAdmin(admin.ModelAdmin):
+    """ModelAdmin class for DataFileSummary objects generated in parsing."""
+
+    list_display = ['status', 'case_aggregates', 'datafile']
+
+
 admin.site.register(models.ParserError, ParserErrorAdmin)
+admin.site.register(models.DataFileSummary, DataFileSummaryAdmin)

tdrs-backend/tdpservice/parsers/fields.py

@@ -1,5 +1,9 @@
 """Datafile field representations."""
 
+import logging
+
+logger = logging.getLogger(__name__)
+
 def value_is_empty(value, length):
     """Handle 'empty' values as field inputs."""
     empty_values = [
@@ -36,6 +40,7 @@ def parse_value(self, line):
         value = line[self.startIndex:self.endIndex]
 
         if value_is_empty(value, self.endIndex-self.startIndex):
+            logger.debug(f"Field: '{self.name}' at position: [{self.startIndex}, {self.endIndex}) is empty.")
             return None
 
         match self.type:
@@ -44,9 +49,13 @@ def parse_value(self, line):
                     value = int(value)
                     return value
                 except ValueError:
+                    logger.error(f"Error parsing field value: {value} to integer.")
                     return None
             case 'string':
                 return value
+            case _:
+                logger.warn(f"Unknown field type: {self.type}.")
+                return None
 
 class TransformField(Field):
     """Represents a field that requires some transformation before serializing."""

tdrs-backend/tdpservice/parsers/migrations/0002_alter_parsererror_error_type.py

@@ -14,5 +14,5 @@ class Migration(migrations.Migration):
             model_name='parsererror',
             name='error_type',
             field=models.TextField(choices=[('1', 'File pre-check'), ('2', 'Record value invalid'), ('3', 'Record value consistency'), ('4', 'Case consistency'), ('5', 'Section consistency'), ('6', 'Historical consistency')], max_length=128),
-        ),
+        )
     ]

tdrs-backend/tdpservice/parsers/migrations/0007_datafilesummary.py

@@ -0,0 +1,24 @@
+# Generated by Django 3.2.15 on 2023-09-20 15:35
+
+from django.db import migrations, models
+import django.db.models.deletion
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('data_files', '0012_datafile_s3_versioning_id'),
+        ('parsers', '0006_auto_20230810_1500'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='DataFileSummary',
+            fields=[
+                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('status', models.CharField(choices=[('Pending', 'Pending'), ('Accepted', 'Accepted'), ('Accepted with Errors', 'Accepted With Errors'), ('Rejected', 'Rejected')], default='Pending', max_length=50)),
+                ('case_aggregates', models.JSONField(null=True)),
+                ('datafile', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='data_files.datafile')),
+            ],
+        ),
+    ]

tdrs-backend/tdpservice/parsers/models.py

@@ -5,7 +5,7 @@
 from django.utils.translation import gettext_lazy as _
 from django.contrib.contenttypes.fields import GenericForeignKey
 from django.contrib.contenttypes.models import ContentType
-
+from tdpservice.data_files.models import DataFile
 
 class ParserErrorCategoryChoices(models.TextChoices):
     """Enum of ParserError error_type."""
@@ -57,12 +57,54 @@ def rpt_month_name(self):
 
     def __repr__(self):
         """Return a string representation of the model."""
-        return f"ParserError {self.id} for file {self.file} and object key {self.object_id}"
+        return f"{{id: {self.id}, file: {self.file.id}, row: {self.row_number}, column: {self.column_number}, " + \
+               f"error message: {self.error_message}}}"
 
     def __str__(self):
         """Return a string representation of the model."""
-        return f"ParserError {self.id}"
+        return f"ParserError {self.__dict__}"
 
     def _get_error_message(self):
         """Return the error message."""
         return self.error_message
+
+class DataFileSummary(models.Model):
+    """Aggregates information about a parsed file."""
+
+    class Status(models.TextChoices):
+        """Enum for status of parsed file."""
+
+        PENDING = "Pending"  # file has been uploaded, but not validated
+        ACCEPTED = "Accepted"
+        ACCEPTED_WITH_ERRORS = "Accepted with Errors"
+        REJECTED = "Rejected"
+
+    status = models.CharField(
+        max_length=50,
+        choices=Status.choices,
+        default=Status.PENDING,
+    )
+
+    datafile = models.ForeignKey(DataFile, on_delete=models.CASCADE)
+
+    case_aggregates = models.JSONField(null=True, blank=False)
+
+    def get_status(self):
+        """Set and return the status field based on errors and models associated with datafile."""
+        errors = ParserError.objects.filter(file=self.datafile)
+        [print(error) for error in errors]
+
+        # excluding row-level pre-checks and trailer pre-checks.
+        precheck_errors = errors.filter(error_type=ParserErrorCategoryChoices.PRE_CHECK)\
+                                .exclude(field_name="Record_Type")\
+                                .exclude(error_message__icontains="trailer")\
+                                .exclude(error_message__icontains="Unknown Record_Type was found.")
+
+        if errors is None:
+            return DataFileSummary.Status.PENDING
+        elif errors.count() == 0:
+            return DataFileSummary.Status.ACCEPTED
+        elif precheck_errors.count() > 0:
+            return DataFileSummary.Status.REJECTED
+        else:
+            return DataFileSummary.Status.ACCEPTED_WITH_ERRORS