@@ -581,11 +668,10 @@
Examples of Common Errors
-
Errors related to header or trailer records:
+
File Pre-Check: Errors related to header or trailer records:
Header-related errors are often the result of submitting files for a fiscal period that is not consistent with the time period in the header record (e.g. trying to submit 2022 data for a 2024 submission). Other header or trailer errors may be related to how the file was generated (e.g. the file produced is missing a header or trailer record). Some examples of how these types of error may appear in your error report are included below:
-
Submitted reporting year: 2024, quarter: 1 doesn't match file reporting year 2022, quarter: 4.
@@ -593,29 +679,31 @@
Errors related to header or trailer records:
Your file does not begin with a HEADER record.
-
-
Please refer to the Transmission File Header Record definitions to compare your file's header or trailer to the expected layout.
-
Errors related to record length:
+
File Pre-Check: Errors related to record length:
Record length-related errors will be raised if the specified record is not aligned with the record layout requirements. For example, this kind of error may appear as follows in the report:
-
-
-
-
T6 record length is 409 characters but must be 379.
+
+
Please refer to the Transmission File Layout documents to compare your records against their expected layouts.
-
-
Please refer to the Transmission File Layout documents to compare your records against their expected layouts.
+
+
+
+
Errors with inconsistent values across related records may require review of the coding instructions to determine the proper correction. In the example below, the error is communicating that a T1 (family) record was found in the file that did not have a corresponding T2 (adult) or T3 (child) record, which effectively means that person-level records associated with this family are missing from the file.
+
+
Every T1 record should have at least one corresponding T2 or T3 record with the same Report Month & Year and Case Number.
+
+
-
Errors related to invalid values for a specific item/data element:
+
Record Value Invalid: Errors related to invalid values for a specific item/data element:
Invalid value errors can come up when a specific item/data element has an unexpected value (e.g. a letter or a symbol was reported for the zip code field, such as: "462$1"):
@@ -630,32 +718,15 @@
Errors related to invalid values for a specific item/data element:
TANF (ACF-199) and SSP-MOE (ACF-209) Coding Instructions
-
-
Errors related to inconsistent values for related items/data elements in the same record:
+
Record Value Inconsistency: Errors related to inconsistent values for related items/data elements in the same record:
Some errors may require review of the coding instructions for multiple items (and their respective values) to determine the proper correction. In the example below, the error is communicating that the value reported for Item 49 is in a conflict with the value for Item 30 in the same record. This message suggests a problem with either the value of Item 49 or the value of Item 30. Refer to the coding instructions and your own data to determine which value needs to be corrected.
-
-
-
-
-
If Item 30 (Family Affiliation) is 1 then Item 49 (Work Participation Status) must be in set of values [01, 02, 05, 07, 09, 15, 17, 18, 19, 99].
-
-
-
-
-
-
Errors with inconsistent values across related records may require review of the coding instructions to determine the proper correction. In the example below, the error is communicating that a T1 (family) record was found in the file that did not have a corresponding T2 (adult) or T3 (child) record, which effectively means that person-level records associated with this family are missing from the file.
-
-
-
-
-
Every T1 record should have at least one corresponding T2 or T3 record with the same Report Month & Year and Case Number.
-
+
If Item 30 (Family Affiliation) is 1 then Item 49 (Work Participation Status) must be in set of values [01, 02, 05, 07, 09, 15, 17, 18, 19, 99].
+
+
-
-
@@ -208,6 +229,7 @@ services:
volumes:
localstack_data:
postgres_data:
+ grafana_pg_data:
elastic_data:
prometheus_data:
grafana_data:
diff --git a/tdrs-backend/gunicorn_start.sh b/tdrs-backend/gunicorn_start.sh
index 40a77af88..2e6c0eeb3 100755
--- a/tdrs-backend/gunicorn_start.sh
+++ b/tdrs-backend/gunicorn_start.sh
@@ -36,4 +36,11 @@ fi
gunicorn_cmd="gunicorn $gunicorn_params"
+if [[ $1 == "cloud" ]]; then
+ echo "Starting Promtail"
+ wget https://github.com/grafana/loki/releases/download/v3.1.1/promtail-linux-amd64.zip
+ unzip -a promtail-linux-amd64.zip && rm -rf promtail-linux-amd64.zip
+ ./promtail-linux-amd64 -config.file=./plg/promtail/config.yml &
+fi
+
exec $gunicorn_cmd
diff --git a/tdrs-backend/manifest.buildpack.yml b/tdrs-backend/manifest.buildpack.yml
index fc9d1460c..99e502987 100755
--- a/tdrs-backend/manifest.buildpack.yml
+++ b/tdrs-backend/manifest.buildpack.yml
@@ -3,10 +3,11 @@ applications:
- name: tdp-backend
memory: 2G
instances: 1
- disk_quota: 2G
+ disk_quota: 4G
+ command: "./gunicorn_start.sh cloud"
env:
REDIS_URI: redis://localhost:6379
buildpacks:
- https://github.com/cloudfoundry/apt-buildpack
- https://github.com/cloudfoundry/python-buildpack.git#v1.8.3
- command: "./gunicorn_start.sh"
+ - https://github.com/cloudfoundry/binary-buildpack
diff --git a/tdrs-backend/plg/deploy.sh b/tdrs-backend/plg/deploy.sh
new file mode 100755
index 000000000..11adaebdd
--- /dev/null
+++ b/tdrs-backend/plg/deploy.sh
@@ -0,0 +1,124 @@
+#!/bin/bash
+set -e
+
+help() {
+ echo "Deploy the PLG stack or a Postgres exporter to the Cloud Foundry space you're currently authenticated in."
+ echo "Syntax: deploy.sh [-h|a|p|u|d]"
+ echo "Options:"
+ echo "h Print this help message."
+ echo "a Deploy the entire PLG stack."
+ echo "p Deploy a postgres exporter. Requires -u and -d"
+ echo "u Requires -p. The database URI the exporter should connect with."
+ echo "d The Cloud Foundry service name of the RDS instance. Should be included with all deployments."
+ echo
+}
+
+deploy_pg_exporter() {
+ pushd postgres-exporter
+ MANIFEST=manifest.$1.yml
+ cp manifest.yml $MANIFEST
+
+ APP_NAME="pg-exporter-$1"
+
+ yq eval -i ".applications[0].name = \"$APP_NAME\"" $MANIFEST
+ yq eval -i ".applications[0].env.DATA_SOURCE_NAME = \"$2\"" $MANIFEST
+ yq eval -i ".applications[0].services[0] = \"$3\"" $MANIFEST
+
+ cf push --no-route -f $MANIFEST -t 180 --strategy rolling
+ cf map-route $APP_NAME apps.internal --hostname $APP_NAME
+
+ # Add policy to allow prometheus to talk to pg-exporter
+ # TODO: this logic needs to be updated to allow routing accross spaces based on where we want PLG to live.
+ cf add-network-policy prometheus $APP_NAME -s "tanf-dev" --protocol tcp --port 9187
+ rm $MANIFEST
+ popd
+}
+
+deploy_grafana() {
+ pushd grafana
+ APP_NAME="grafana"
+ DATASOURCES="datasources.yml"
+ cp datasources.template.yml $DATASOURCES
+ MANIFEST=manifest.tmp.yml
+ cp manifest.yml $MANIFEST
+
+ yq eval -i ".datasources[0].url = \"http://prometheus.apps.internal:8080\"" $DATASOURCES
+ yq eval -i ".datasources[1].url = \"http://loki.apps.internal:8080\"" $DATASOURCES
+ yq eval -i ".applications[0].services[0] = \"$1\"" $MANIFEST
+
+ cf push --no-route -f $MANIFEST -t 180 --strategy rolling
+ # cf map-route $APP_NAME apps.internal --hostname $APP_NAME
+ # Give Grafana a public route for now. Might be able to swap to internal route later.
+ cf map-route "$APP_NAME" app.cloud.gov --hostname "${APP_NAME}"
+
+ # Add policy to allow grafana to talk to prometheus and loki
+ cf add-network-policy $APP_NAME prometheus --protocol tcp --port 8080
+ cf add-network-policy $APP_NAME loki --protocol tcp --port 8080
+ rm $DATASOURCES
+ rm $MANIFEST
+ popd
+}
+
+deploy_prometheus() {
+ pushd prometheus
+ cf push --no-route -f manifest.yml -t 180 --strategy rolling
+ cf map-route prometheus apps.internal --hostname prometheus
+ popd
+}
+
+deploy_loki() {
+ pushd loki
+ cf push --no-route -f manifest.yml -t 180 --strategy rolling
+ cf map-route loki apps.internal --hostname loki
+ popd
+}
+
+err_help_exit() {
+ echo $1
+ echo
+ help
+ popd
+ exit
+}
+
+while getopts ":hap:u:d:" option; do
+ case $option in
+ h) # display Help
+ help
+ exit;;
+ a) # Deploy PLG stack
+ DEPLOY="plg";;
+ p) # Deploy a Postgres exporter to $ENV
+ ENV=$OPTARG
+ DEPLOY="pg-exporter";;
+ u) # Bind a Postgres exporter to $DB_URI
+ DB_URI=$OPTARG;;
+ d) # Bind a Postgres exporter or Grafana to $DB_SERVICE_NAME
+ DB_SERVICE_NAME=$OPTARG;;
+ \?) # Invalid option
+ echo "Error: Invalid option"
+ exit;;
+ esac
+done
+
+if [ "$#" -eq 0 ]; then
+ help
+ exit
+fi
+
+pushd "$(dirname "$0")"
+if [ "$DB_URI" == "" ] || [ "$DB_SERVICE_NAME" == "" ]; then
+ err_help_exit "Error: you must include a database service name."
+fi
+if [ "$DEPLOY" == "plg" ]; then
+ deploy_prometheus
+ deploy_loki
+ deploy_grafana
+fi
+if [ "$DEPLOY" == "pg-exporter" ]; then
+ if [ "$DB_URI" == "" ]; then
+ err_help_exit "Error: you must provide a database uri when deploying a postgres exporter."
+ fi
+ deploy_pg_exporter $ENV $DB_URI $DB_SERVICE_NAME
+fi
+popd
diff --git a/tdrs-backend/plg/grafana/custom.ini b/tdrs-backend/plg/grafana/custom.ini
index b8124cdeb..7d8be7d57 100644
--- a/tdrs-backend/plg/grafana/custom.ini
+++ b/tdrs-backend/plg/grafana/custom.ini
@@ -1,13 +1,12 @@
+# TODO: Update the server config based on where we want PLG to live and how we specify the domain.
+
##################### Grafana Configuration Defaults #####################
-#
-# Do not modify this file in grafana installs
-#
# possible values : production, development
app_mode = production
# instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
-instance_name = ${HOSTNAME}
+instance_name = grafana
#################################### Paths ###############################
[paths]
@@ -38,17 +37,17 @@ min_tls_version = ""
http_addr =
# The http port to use
-http_port = 9400
+http_port = 8080
# The public facing domain name used to access grafana from a browser
-domain = localhost
+domain = app.cloud.gov
# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
enforce_domain = false
# The full public facing url
-root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/
+root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana
# Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
serve_from_sub_path = true
@@ -421,7 +420,7 @@ data_keys_cache_cleanup_interval = 1m
enabled = true
# snapshot sharing options
-external_enabled = true
+external_enabled = false
external_snapshot_url = https://snapshots.raintank.io
external_snapshot_name = Publish to snapshots.raintank.io
@@ -851,7 +850,7 @@ enabled = true
# 3. Composed by at least 1 lowercase character
# 4. Composed by at least 1 digit character
# 5. Composed by at least 1 symbol character
-password_policy = false
+password_policy = true
#################################### Auth Proxy ##########################
[auth.proxy]
@@ -1520,7 +1519,7 @@ enabled = true
#################################### News #############################
[news]
# Enable the news feed section
-news_feed_enabled = true
+news_feed_enabled = false
#################################### Query #############################
[query]
@@ -1938,7 +1937,7 @@ read_only_toggles =
#################################### Public Dashboards #####################################
[public_dashboards]
# Set to false to disable public dashboards
-enabled = true
+enabled = false
###################################### Cloud Migration ######################################
[cloud_migration]
diff --git a/tdrs-backend/plg/grafana/custom.local.ini b/tdrs-backend/plg/grafana/custom.local.ini
new file mode 100644
index 000000000..4f2bc8780
--- /dev/null
+++ b/tdrs-backend/plg/grafana/custom.local.ini
@@ -0,0 +1,1972 @@
+##################### Grafana Configuration Defaults #####################
+
+# possible values : production, development
+app_mode = development
+
+# instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
+instance_name = ${HOSTNAME}
+
+#################################### Paths ###############################
+[paths]
+# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
+data = data
+
+# Temporary files in `data` directory older than given duration will be removed
+temp_data_lifetime = 24h
+
+# Directory where grafana can store logs
+logs = data/log
+
+# Directory where grafana will automatically scan and look for plugins
+plugins = data/plugins
+
+# folder that contains provisioning config files that grafana will apply on startup and while running.
+provisioning = conf/provisioning
+
+#################################### Server ##############################
+[server]
+# Protocol (http, https, h2, socket)
+protocol = http
+
+# Minimum TLS version allowed. By default, this value is empty. Accepted values are: TLS1.2, TLS1.3. If nothing is set TLS1.2 would be taken
+min_tls_version = ""
+
+# The ip address to bind to, empty will bind to all interfaces
+http_addr =
+
+# The http port to use
+http_port = 9400
+
+# The public facing domain name used to access grafana from a browser
+domain = localhost
+
+# Redirect to correct domain if host header does not match domain
+# Prevents DNS rebinding attacks
+enforce_domain = false
+
+# The full public facing url
+root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana
+
+# Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
+serve_from_sub_path = true
+
+# Log web requests
+router_logging = false
+
+# the path relative working path
+static_root_path = public
+
+# enable gzip
+enable_gzip = false
+
+# https certs & key file
+cert_file =
+cert_key =
+cert_pass =
+
+# Certificates file watch interval
+certs_watch_interval =
+
+# Unix socket gid
+# Changing the gid of a file without privileges requires that the target group is in the group of the process and that the process is the file owner
+# It is recommended to set the gid as http server user gid
+# Not set when the value is -1
+socket_gid = -1
+
+# Unix socket mode
+socket_mode = 0660
+
+# Unix socket path
+socket = /tmp/grafana.sock
+
+# CDN Url
+cdn_url =
+
+# Sets the maximum time in minutes before timing out read of an incoming request and closing idle connections.
+# `0` means there is no timeout for reading the request.
+read_timeout = 0
+
+# This setting enables you to specify additional headers that the server adds to HTTP(S) responses.
+[server.custom_response_headers]
+#exampleHeader1 = exampleValue1
+#exampleHeader2 = exampleValue2
+
+[environment]
+# Sets whether the local file system is available for Grafana to use. Default is true for backward compatibility.
+local_file_system_available = true
+
+#################################### GRPC Server #########################
+[grpc_server]
+network = "tcp"
+address = "127.0.0.1:10000"
+use_tls = false
+cert_file =
+key_file =
+# this will log the request and response for each unary gRPC call
+enable_logging = false
+
+# Maximum size of a message that can be received in bytes. If not set, uses the gRPC default (4MiB).
+max_recv_msg_size =
+
+# Maximum size of a message that can be sent in bytes. If not set, uses the gRPC default (unlimited).
+max_send_msg_size =
+
+#################################### Database ############################
+[database]
+# You can configure the database connection by specifying type, host, name, user and password
+# as separate properties or as on string using the url property.
+
+# Either "mysql", "postgres" or "sqlite3", it's your choice
+type = postgres
+host = grafana-pg:5432
+name = grafana
+user = grafana
+# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
+password = something_secure
+# Use either URL or the previous fields to configure the database
+# Example: mysql://user:secret@host:port/database
+url =
+
+# Max idle conn setting default is 2
+max_idle_conn = 2
+
+# Max conn setting default is 0 (mean not set)
+max_open_conn =
+
+# Connection Max Lifetime default is 14400 (means 14400 seconds or 4 hours)
+conn_max_lifetime = 14400
+
+# Set to true to log the sql calls and execution times.
+log_queries =
+
+# For "postgres", use either "disable", "require" or "verify-full"
+# For "mysql", use either "true", "false", or "skip-verify".
+ssl_mode = disable
+
+# For "postgres", use either "1" to enable or "0" to disable SNI
+ssl_sni =
+
+# Database drivers may support different transaction isolation levels.
+# Currently, only "mysql" driver supports isolation levels.
+# If the value is empty - driver's default isolation level is applied.
+# For "mysql" use "READ-UNCOMMITTED", "READ-COMMITTED", "REPEATABLE-READ" or "SERIALIZABLE".
+isolation_level =
+
+ca_cert_path =
+client_key_path =
+client_cert_path =
+server_cert_name =
+
+# For "sqlite3" only, path relative to data_path setting
+path = grafana.db
+
+# For "sqlite3" only. cache mode setting used for connecting to the database
+cache_mode = private
+
+# For "sqlite3" only. Enable/disable Write-Ahead Logging, https://sqlite.org/wal.html. Default is false.
+wal = false
+
+# For "mysql" and "postgres". Lock the database for the migrations, default is true.
+migration_locking = true
+
+# For "mysql" and "postgres" only if migrationLocking is set. How many seconds to wait before failing to lock the database for the migrations, default is 0.
+locking_attempt_timeout_sec = 0
+
+# For "sqlite" only. How many times to retry query in case of database is locked failures. Default is 0 (disabled).
+query_retries = 0
+
+# For "sqlite" only. How many times to retry transaction in case of database is locked failures. Default is 5.
+transaction_retries = 5
+
+# Set to true to add metrics and tracing for database queries.
+instrument_queries = false
+
+#################################### Cache server #############################
+[remote_cache]
+# Either "redis", "memcached" or "database" default is "database"
+type = database
+
+# cache connectionstring options
+# database: will use Grafana primary database.
+# redis: config like redis server e.g. `addr=127.0.0.1:6379,pool_size=100,db=0,ssl=false`. Only addr is required. ssl may be 'true', 'false', or 'insecure'.
+# memcache: 127.0.0.1:11211
+connstr =
+
+# prefix prepended to all the keys in the remote cache
+prefix =
+
+# This enables encryption of values stored in the remote cache
+encryption =
+
+#################################### Data proxy ###########################
+[dataproxy]
+
+# This enables data proxy logging, default is false
+logging = false
+
+# How long the data proxy waits to read the headers of the response before timing out, default is 30 seconds.
+# This setting also applies to core backend HTTP data sources where query requests use an HTTP client with timeout set.
+timeout = 30
+
+# How long the data proxy waits to establish a TCP connection before timing out, default is 10 seconds.
+dialTimeout = 10
+
+# How many seconds the data proxy waits before sending a keepalive request.
+keep_alive_seconds = 30
+
+# How many seconds the data proxy waits for a successful TLS Handshake before timing out.
+tls_handshake_timeout_seconds = 10
+
+# How many seconds the data proxy will wait for a server's first response headers after
+# fully writing the request headers if the request has an "Expect: 100-continue"
+# header. A value of 0 will result in the body being sent immediately, without
+# waiting for the server to approve.
+expect_continue_timeout_seconds = 1
+
+# Optionally limits the total number of connections per host, including connections in the dialing,
+# active, and idle states. On limit violation, dials will block.
+# A value of zero (0) means no limit.
+max_conns_per_host = 0
+
+# The maximum number of idle connections that Grafana will keep alive.
+max_idle_connections = 100
+
+# How many seconds the data proxy keeps an idle connection open before timing out.
+idle_conn_timeout_seconds = 90
+
+# If enabled and user is not anonymous, data proxy will add X-Grafana-User header with username into the request.
+send_user_header = false
+
+# Limit the amount of bytes that will be read/accepted from responses of outgoing HTTP requests.
+response_limit = 0
+
+# Limits the number of rows that Grafana will process from SQL data sources.
+row_limit = 1000000
+
+# Sets a custom value for the `User-Agent` header for outgoing data proxy requests. If empty, the default value is `Grafana/
` (for example `Grafana/9.0.0`).
+user_agent =
+
+#################################### Analytics ###########################
+[analytics]
+# Server reporting, sends usage counters to stats.grafana.org every 24 hours.
+# No ip addresses are being tracked, only simple counters to track
+# running instances, dashboard and error counts. It is very helpful to us.
+# Change this option to false to disable reporting.
+reporting_enabled = true
+
+# The name of the distributor of the Grafana instance. Ex hosted-grafana, grafana-labs
+reporting_distributor = grafana-labs
+
+# Set to false to disable all checks to https://grafana.com
+# for new versions of grafana. The check is used
+# in some UI views to notify that a grafana update exists.
+# This option does not cause any auto updates, nor send any information
+# only a GET request to https://grafana.com/api/grafana/versions/stable to get the latest version.
+check_for_updates = true
+
+# Set to false to disable all checks to https://grafana.com
+# for new versions of plugins. The check is used
+# in some UI views to notify that a plugin update exists.
+# This option does not cause any auto updates, nor send any information
+# only a GET request to https://grafana.com to get the latest versions.
+check_for_plugin_updates = true
+
+# Google Analytics universal tracking code, only enabled if you specify an id here
+google_analytics_ua_id =
+
+# Google Analytics 4 tracking code, only enabled if you specify an id here
+google_analytics_4_id =
+
+# When Google Analytics 4 Enhanced event measurement is enabled, we will try to avoid sending duplicate events and let Google Analytics 4 detect navigation changes, etc.
+google_analytics_4_send_manual_page_views = false
+
+# Google Tag Manager ID, only enabled if you specify an id here
+google_tag_manager_id =
+
+# Rudderstack write key, enabled only if rudderstack_data_plane_url is also set
+rudderstack_write_key =
+
+# Rudderstack data plane url, enabled only if rudderstack_write_key is also set
+rudderstack_data_plane_url =
+
+# Rudderstack SDK url, optional, only valid if rudderstack_write_key and rudderstack_data_plane_url is also set
+rudderstack_sdk_url =
+
+# Rudderstack Config url, optional, used by Rudderstack SDK to fetch source config
+rudderstack_config_url =
+
+# Rudderstack Integrations URL, optional. Only valid if you pass the SDK version 1.1 or higher
+rudderstack_integrations_url =
+
+# Intercom secret, optional, used to hash user_id before passing to Intercom via Rudderstack
+intercom_secret =
+
+# Application Insights connection string. Specify an URL string to enable this feature.
+application_insights_connection_string =
+
+# Optional. Specifies an Application Insights endpoint URL where the endpoint string is wrapped in backticks ``.
+application_insights_endpoint_url =
+
+# Controls if the UI contains any links to user feedback forms
+feedback_links_enabled = true
+
+# Static context that is being added to analytics events
+reporting_static_context =
+
+#################################### Security ############################
+[security]
+# disable creation of admin user on first start of grafana
+disable_initial_admin_creation = false
+
+# default admin user, created on startup
+admin_user = admin
+
+# default admin password, can be changed before first start of grafana, or in profile settings
+admin_password = admin
+
+# default admin email, created on startup
+admin_email = admin@localhost
+
+# used for signing
+secret_key = SW2YcwTIb9zpOOhoPsMm
+
+# current key provider used for envelope encryption, default to static value specified by secret_key
+encryption_provider = secretKey.v1
+
+# list of configured key providers, space separated (Enterprise only): e.g., awskms.v1 azurekv.v1
+available_encryption_providers =
+
+# disable gravatar profile images
+disable_gravatar = false
+
+# data source proxy whitelist (ip_or_domain:port separated by spaces)
+data_source_proxy_whitelist =
+
+# disable protection against brute force login attempts
+disable_brute_force_login_protection = false
+
+# set to true if you host Grafana behind HTTPS. default is false.
+cookie_secure = false
+
+# set cookie SameSite attribute. defaults to `lax`. can be set to "lax", "strict", "none" and "disabled"
+cookie_samesite = lax
+
+# set to true if you want to allow browsers to render Grafana in a , 
+
+
+ 
+ 