Metrics

Current Monitoring and Metrics

What are Metrics Services?

In order to continuously monitor performance to help identify bottlenecks, slow processes, and areas that need optimization, we applied custom Metrics Services to Caseflow. It enables custom metrics collection, that can be tailored to the specific needs of Caseflow, making it adaptable to changing requirements over time. These allow for fine-grained data collection, enabling in-depth analysis and troubleshooting, which is crucial for diagnosing complex issues, and proactively addressing issues/errors before they impact the users.

We used the Goal-Question-Metric (GQM) approach to identify appropriate metrics for Caseflow by first identifying the information gaps we wanted to investigate. We searched the code to identify the factors that needed to be measured, such as code quality, development speed, and user satisfaction, including areas of user experience such as validation, failures, performance time, and errors that needed to be caught and defined.

Next, we identified key performance indicators (KPIs) to measure progress towards our goals. We defined the metrics that would be used to measure those KPIs and implemented them in the code to collect the data. The data is sent to Rails Logging and Datadog for analysis. We use these logs to identify areas for improvement and make changes to the code.

Why use Metrics Services?

This gives us the ability to store and view all metrics in one database which we can analyze without the use of external services or products, but these metrics can be integrated with external services or products in certain use cases.

Metric table/model

All metrics are saved to a table in our database. This table and other relevant metric dashboards can be viewed on Metabase. Ensure no PII/PHI is being saved in any metric field.

Field Name	Data Type	Description	Example
UUID	uuid	Unique ID for the metric, can be used to search within various systems for the logging	30a33868-2c1b-449c-8cec-3f265a7d0e43
User ID	bigint	The ID of the user who generated metric.	123
Metric Name	string	Name of Metric	caseflow.server.metric.request_latency
Metric Class	string	Class of metric, use reflection to find value to populate this	MetricService
Metric Group	string	Metric group: service, etc	service
Metric Message	string	Message or log for metric	BGS:fetch veteran info fo vbms id: XXXXXXXXX
Metric Type	string	Type of metric: ERROR, LOG, PERFORMANCE, etc	performance
Metric Product	string	Where in application: Queue, Hearings, Intake, VHA, etc	bgs
App Name	string	Application name: caseflow or efolder	caseflow
Metric Attributes	json	Store attributes relevant to the metric: OS, browser, etc	{"service":"bgs", "endpoint": "claims.find_by_vbms_file_number"}
Additional Info	json	additional data to store for the metric	{"file_number_type": "SSN"}
Sent To	string	Which system metric was sent to: Datadog, Rails Console, Javascript Console, etc	{rails_console, dynatrace}
Sent To Info	json	Additional information for which system metric was sent to	{"metric_group":"service","metric_name":"request_latency","metric_value":0.5868008640245534,"app_name":"caseflow","attrs":{"service":"bgs","endpoint":"claims.find_by_vbms_file_number","uuid":"30a33868-2c1b-449c-8cec-3f265a7d0e43"}}
Relevant Tables Info	json	Store information to tie metric to database table(s)	{}
Start Time	datetime	When metric recording started	2023-9-5, 10:30 AM
End Time	datetime	When metric recording stopped	2023-9-5, 10:30 AM
Duration	float	Time in milliseconds from start to end	110.51
Created At	datetime	When the metric was created in the database	2023-9-5, 10:30 AM

How to use Metric Services

To enable the Metrics Service, the metrics_monitoring feature toggle is required for the current user.

Front End

The following frontend Metrics Services will trigger a stopwatch, recording the duration of the process automatically. The timing data and metric data to be recorded will be sent to the backend via a POST request to /metrics/v2/logs, and is stored in the metrics database table. The service will send back the return value of the function that the metric was recording.

MetricsService.js

To measure frontend metrics that involve JavaScript processes and asynchronous calls:

Use recordMetrics() when you want to record javascript proccesses, such as a looping through a large array or parsing a long string of text.

To use recordMetrics() you will need to pass three arguments:

• The function to be recorded/tested

• An object containing the metric data fields for the table. The required field is data, which requires a JSON format. The message field is optional, and defaults to the uuid and data of the metric. The product field is optional, but if used is required as a product type located in the Metric Model, and otherwise defaults to "caseflow".

• The dedicated feature toggle that controls if the metrics are stored to the database. This defaults as true. Ensure it is included when calling recordMetrics() to avoid unneccesary data.

// Set up a large array
const testArray = [1,2,3...100];

// Function to loop through an array
function loopArr(arr) {
    for (let i = 0; i < arr.length; i++) {
        console.log(i)
    }
};

// Set up metric data with the fields (message, type, product, data)
const metricData = {
      message: "Testing duration of loopArr function with a large array",
      type: 'performance',
      product: 'reader',
      data: {
        "browser": "chrome",
        "OS": "Mac",
      },
    };


// Pass recordMetrics() the target function of loopArr(), metricData object, and dedicated feature toggle to enable this metric
recordMetrics(loopArr(testArray), metricData, <METRICS_FEATURE_TOGGLE>);

The use of recordAsyncMetrics() functions similarly to recordMetrics() method, but in place of a function, pass in a promise to record a metric. A metric data object in JSON format and a dedicated feature toggle are still required.

// Set the file name
const fileName = "helloworld.pdf"

// Set up a promise from a pdfJS LoadingTask
const promise = pdfjsLib.getDocument(fileName).promise;

// Set up metric data with the fields (data, type, message, product)
const metricData = {
      message: "Loading document using pdfJS",
      type: 'performance',
      product: 'reader',
      data: {
        "browser": "chrome",
        "OS": "Mac",
        "filename": `${fileName}`
      },
    };


// Pass recordAsyncMetrics() the promise, metricData object, and dedicated feature toggle
recordAsyncMetrics(promise, metricData, <METRICS_FEATURE_TOGGLE>);

ApiUtil.js

ApiUtil is used for logging successful/unsuccesful API REST calls.

To save metrics in the database while using the ApiUtil, pass in specific featureToggles in the requestOptions object:

• metricsLogRestError records the method, the URL, and the error message
• metricsLogRestSuccess records the method, and the URL

// Build requestOptions object with required feature toggles
let requestOptions = {
      responseType: 'application/json',
      metricsLogRestError: this.props.featureToggles.metricsLogRestError,
      metricsLogRestSuccess: this.props.featureToggles.metricsLogRestSuccess
    };

// Make API call with specified request options using ApiUtil.get 
const response = await ApiUtil.get("https://catfact.ninja/fact", requestOptions)

Back End

To enable backend Metrics Service, the metrics_monitoring is required for the current user. Using this service will trigger a stopwatch, recording the duration of the process automatically. The timing data and metric data will be saved to the metrics database table. The service will send back the return value of the function that the metric was recording.

MetricsService.record()

To use MetricsService.record() identify the method(s) for metrics collection, and wrap method(s) with MetricsService.record in a do/end block.

• The description details the method as the metric message (defaults to a blank field). Saves as metric_message in the database table.
• service can be used to record a specific third-party service if the method requires (defaults to "nil"). Saves as a part of metric_attributes in the database table.
• name can be specified as the endpoint of the specific third-party service (defaults to "unknown"). Saves as a part of metric_attributes in the database table.
• caller is the metric class, the metric params, and the current user (defaults to "nil"). This class saves to the metric_class, the user saves as the user to the database table.

# Set up a large array
array = (1..100).to_a

# Wrap service around target function passing in (description, service, name)
MetricsService.record("Printing Veteran document ID's from array",
    service: "vbms",
    name: "print_all_veteran_document_ids") do
      # Target function being measured
      array.each { |x| puts x }
end       

Rails Logging

Direct Rails Logging

Where in Code	Key ID(s)	Method Used	Description
Idt::Api::V1::BaseController .log_error()	error	Rails .logger.error(error)	Sends the error to the Rails logger
Idt::Api::V1::BaseController .log_error()	error	Rails.logger .error(error&.backtrace&.join("\n"))	Handles the error along with back trace
Reader::DocumentsController .index()	Appeal_id	Rails .logger	Some users couldn't access the documents under other profiles.
Services::metrics_service .record()	info	Rails .logger	Logs the description after the start and end of metric services
AdminController .verify_access()	info	Rails.logger .info()	User with specific roles couldn’t access the documents.
QueuController .verify_access()	info	Rails.logger .info()	Redirects the user associated with case details to search.
AnnotationController .rescue_from	error	Rails.logger .error	Validation Failed.
Idt::Api::V1::MpiController .veteran_updates()	info	Rails.logger .info	Logs veteran info like id, deceased time and indicator
Idt::Api::V1::JobsController .create()	info	Rails.logger .info	Displays message related to a job being pushed to queue
TeamManagementController .update()	info	Rails.logger .info	Logs message related to existing record updation
TeamManagementController .create_judge_team()	info	Rails.logger .info	A message with info stating that judge team is created for a specific user is displayed.
TeamManagementController .create_dvc_team()	info	Rails.logger .info	A success message is logged when a dvc team is created for a user
TeamManagementController .create_private_bar	info	Rails.logger .info	Used to log message when a private bar with parameters is being created
TeamManagementController .create_field_vso()	info	Rails.logger .info	Logs a message while field vso is created.
TeamManagementController .create_national_vso()	info	Rails.logger .info	Creating a Vso with parameters is logged.
Idt::Api::V1::BaseController .rescue_from	error	Rails.logger .error	Logs an IDT standard error with uuid
HearingsController .rescue_from	debug	Rails.logger .debug	Debugging message when a hearing cannot be found in VACOLS and caseflow.
ExplainAppealEventsConcern .request_issues_as_event_data()	warn	Rails.logger .warn	Unaccounted decision issues that are left will be logged as warning.
IntakesController .log_error()	error	Rails.logger .error	Intake error related to sentry event is displayed.
IssuesController .rescue_form ActiveRecord	error	Rails.logger .error	A failure of Issues Controller is being popped.

Sentry/Raven

Direct Raven calls

Where in Code	Key ID(s)	Method Used	Description
Idt::Api::V1::BaseController .log_error()	error	Raven .capture_exception()	Handles logging errors that occur within the class
ApplicationController .rescue_from StandardError	exception, error_uuid	Raven .capture_exception()	Logs the exception error and whether the exception was actionable
ApplicationController .on_external_error()	error, error_uuid	Raven .capture_exception()	Logs the error and its uuid when the error is not ignorable. Used when there is a BGS::ShareError or VBMS::ClientError
ApplicationController .rescue_from Caseflow...VacolsRepositoryError	exception, error_uuid	Raven .capture_exception()	Logs the exception and its uuid when there is a Vacols Repository Errors
AppealsController .fetch_notification_list()	error, error_uuid	Raven .capture_exception()	Logs standard errors that occur when fetching the notification list
AppealsController .render_error()	Error, appeal.type appeal.id	Raven .capture_exception()	Used to create a standardized error message for the controller
ClaimReviewController .render_error()	error, error_uuid	Raven .capture_exception()	Used to create a standardized error message for the controller
DecisionReviewsController .update()	Error, error_uuid	Raven .capture_exception()	Logs errors that occur when updating a Decision Review
IntakesController .log_error()	error	Raven .capture_exception()	Used to create a standardized error message for the controller
IntakesController .index_props()	error	Raven .capture_exception()	Logs standard errors that occur in the method
TasksController .update()	error	Raven .capture_exception()	Logs errors from several custom error types while updating a Task: Caseflow::Error::InvalidEmailError AssignHearingDispositionTask::HearingAssociationMissing Caseflow::Error::VirtualHearingConversionFailed
Hearings::HearingDayController .log_error()	error	Raven .capture_exception()	Used to create a standardized error message for the controller
Idt::AuthenticationsController .index()	error	Raven .capture_exception()	Logs errors when encountering Caseflow::Error::InvalidOneTimeKey
Test::UsersController .show_error()	error, error_uuid	Raven .capture_exception()	Used to create a standardized error message for the controller Note: Appears to be unused
ApplicationJob .capture_exception()	error	Raven .capture_exception()	Used to create a standardized error message for the job if the error is not ignorable
DispatchEmailJob .send_email()	error	Raven .capture_exception()	Logs errors encountered while sending emails
SetAppealAgeAodJob .log_error()	error	Raven .capture_exception()	Used to create a standardized error message for the job
StatsCollectorJob .log_error()	error, collector_name	Raven .capture_exception()	Used to create a standardized error message for the job
UpdateAppellantRepresentationJob .log_error()	error	Raven .capture_exception()	Used to create a standardized error message for the job
UpdateCachedAppealsAttributesJob .log_error()	error	Raven .capture_exception()	Used to create a standardized error message for the job
VANotifyStatusUpdateJob .get_current_status()	error, error_uuid	Raven .capture_exception()	Logs errors when type of VA Notify Status type is neither an Email nor an SMS
VANotifyStatusUpdateJob .get_current_status()	error, error_uuid	Raven .capture_exception()	Logs error when job encounters a Caseflow::Error::VANotifyApiError
WarmBgsCachesJob .warm_people_caches()	error	Raven .capture_exception()	Logs standard errors encountered during the function
WarmBgsCachesJob .warm_participant_caches()	error	Raven .capture_exception()	Logs standard errors encountered during the function
WarmBgsCachesJob .warm_ro_participant_caches()	error	Raven .capture_exception()	Logs standard errors encountered during the function
WarmBgsCachesJob .warm_veteran_attribute_caches()	error	Raven .capture_exception()	Logs standard errors encountered during the function
Hearings::GeomatchAndCacheAppealJob retry_on()	exception	Raven .capture_exception()	Logs an error when the job has retried 10 times
Hearings::SendEmail .send_email()	error	Raven .capture_exception()	Logs standard and custom errors encountered while sending an email
Hearings::SendEmail .create_sent_hearing_email_event()	error	Raven .capture_exception()	Logs standard errors encountered while creating the sent hearing email event
Hearings::SendSentStatusEmail .call()	error	Raven .capture_exception()	Logs standard and custom errors encountered while sending an email
VirtualHearings::CreateConferenceJob retry_on()	exception	Raven .capture_exception()	Logs an error when the job has encountered the PexipApiError 10 times
VirtualHearings::CreateConferenceJob .send_emails()	exception	Raven .capture_exception()	Logs standard errors encountered while sending an email
VirtualHearings::CreateConferenceJob .generate_links_and_pins()	exception	Raven .capture_exception()	Logs standard errors encountered while generating links and pins for virtual hearings
VirtualHearings::DeleteConferencesJob retry_on()	exception	Raven .capture_exception()	Logs an error when the job has encountered the DeleteConferencesJobFailure error 5 times
DecisionReview .ratings_with_issues_or_decisions()	error	Raven .capture_exception()	Logs an error when the model encounters the PromulgatedRating::LockedRatingError
DecisionReviewIntake . after_validated_pre_start!()	error	Raven .capture_exception()	Logs an error when the model is running during start and encounters the error: EndProductEstablishment::EstablishedEndProductNotFound
DistributedCase .flag_redistribution()	Case id, task.appeal.uuid	Raven.capture_message()	Logs a message when a distributed case already exists for the same appeal
EstablishClaim .try_prepare_with_decision!()	error, claim id	Raven .capture_exception()	Logs an error when the model encounters a VBMS::ClientError when trying to prepare a decision
HearingDay .log_error()	error, hearing_day.id	Raven .capture_exception()	Used to create a standardized error message for the model
LegacyHearing .vacols_hearing_exists?()	error	Raven .capture_exception()	Logs an error when the model encounters the VacolsRecordNotFound error
PromulgatedRating .retry_fetching_rating_profile()	error	Raven .capture_exception()	Logs an error when retrying a fetch and encountering the errors BGS::ShareError Rating::NilRatingProfileListError
RampRefiling .create_contentions_on_new_end_product!()	error	Raven .capture_exception()	Logs standard errors encountered by the model while performing this task
Veteran .handle_bgs_share_error()	error	Raven .capture_exception()	Logs an error unless the error message is a Sensitive File
AppealConcern .timezone_identifier_for_address()	error	Raven .capture_exception()	Logs a standard error encountered by the model while trying to identify the time zone for an address
AssociatedVacolsModel .check_and_load_vacols_data!()	LazyLoadingDisabledError	Raven .capture_exception()	Logs an error encountered while trying to load vacols data
HearingConcern .rescue_and_check_toggle_veteran_date_of_death_info()	error	Raven .capture_exception()	Logs standard and custom errors encountered by the model while checking the veteran date of death info
ConferenceLink .generate_links_and_pins()	error	Raven .capture_exception()	Logs custom errors while the model is trying to generate links and pins: VirtualHearings::LinkService::PINKeyMissingError VirtualHearings::LinkService::URLHostMissingError VirtualHearings::LinkService::URLPathMissingError
SentHearingEmailEvent .invalid_reported_status_failure()	reported_status	Raven .capture_exception()	Logs an error message when an invalid reported status is entered
SentHearingEmailEvent .email_already_sent_failure()	external_message_id	Raven .capture_exception()	Logs an error message when a previous email has failed to be delivered to the email addesss provided
WorkQueue::TaskColumnSerializer :veteran_appellant_deceased	error	Raven .capture_exception()	Logs an error when the model encounters the BGS::PowerOfAttorneyFolderDenied error
InformalHearingPresentationTask .update_to_new_poa()	error	Raven .capture_exception()	Log standard errors encountered while updating power of attorney
ScheduleHearingTask .cancel_parent_task()	parent.id	Raven .capture_exception()	Logs an error when trying to cancel a parent HearingTask that still has active child tasks
HearingRepository .hearings_for()	Error, vacols_record.hearing_pkseq	Raven .capture_exception()	Logs an error when a regional office for a case hearing is not found (RegionalOffice::NotFoundError)
DocumentFetcher .warn_about_same_vbms_document_id()	error_message	Raven .capture_exception()	Logs a DuplicateVbmsDocumentIdError
RedistributedCase .alert_case_not_found()	error, case_id	Raven .capture_exception()	Logs an error when a case is not found
RedistributedCase .alert_existing_distributed_case_not_unique()	error, case_id, judge.css_id	Raven .capture_exception()	Logs an error when another DistributedCase already exists
VaDotGovAddressValidator .remove_missing_facilities()	cacility missing_ids, error	Raven .capture_exception()	Logs an error when given facility ids are not found
VirtualHearings::ResendVirtualHearingEmailsService .reset_sent_status_and_send()	error	Raven .capture_exception()	Logs standard and custom errors encountered during the this service's method
VirtualHearings::ResendVirtualHearingEmailsService . should_resend_email?()	error	Raven .capture_exception()	Logs standard and custom errors encountered while determining if an email should be resent
LegacyAppeal .assigned_to_location()	Legacy appeal external_id	Raven.capture_message()	Logs a message when the Legacy Appeal only has closed tasks left
LegacyAppeal .assigned_to_location()	Legacy appeal external_id	Raven.capture_message()	Logs a message when the Legacy Appeal is open but has no tasks
Task .put_on_hold_due_to_new_child_task()	Task id	Raven.capture_message()	Logs a message when cloning a task to create a child task
JudgeAssignTask .begin_decision_review_phase()	Message, time	Raven.capture_message()	Logs a message that notes the time and that the type from Assign to DecisionReview is still being changed
RootTask .when_child_task_created()	child_task.id, root_task.id	Raven.capture_message()	Logs a message that a child task has been created from a root task but didn’t update RootTask status
HearingRepository .hearings_for()	vacols_ids	Raven.capture_message()	Logs a message that a hearings_for has been sent for a non-unique vacols id
InitialTasksFactory .warn_poa_not_a_representative()	poa_participant_id, appeal.id	Raven.capture_message()	Logs a message when POA is not a representative
HearingBadge.jsx componentDidMount()	error	Raven.captureException()	Logs and displays errors which occur while getting hearings for an appeal

Indirect Raven/Sentry calls

Where in Code	Key ID(s)	Method Used	Description
Idt::Api::V1::BaseController rescue_from	error	BaseController .log_error()	Used multiple times to ensure errors are logged, including custom error types
AppealsController .update_power_of_attorney()	error	AppealsController .render_error()	Logs standard errors that occur when updating the power of attorney
ClaimReviewController .edit()	error	ClaimReviewController .render_error()	Logs standard errors that occur when editing a claim review
IntakesController .create()	error	IntakesController .log_error()	Logs standard errors that occur when creating Intakes
IntakesController .review()	error	IntakesController .log_error()	Logs standard errors that occur when rendering an Intake review
IntakesController .complete()	error	IntakesController .log_error()	Logs standard errors what occur when marking an Intake as complete
Hearings::HearingDayController .show()	error	Hearings::HearingDayController .log_error()	Logs errors from several custom error types while displaying a Hearing Day: PINKeyMissingError, URLHostMissingError, URLPathMissingError
ApplicationJob rescue_from	error	ApplicationJob .capture_exception()	Logs error when encountering custom errors during the execution of the job: Caseflow::Error::TransientError, VBMS::ClientError, BGS::ShareError
SetAppealAgeAodJob .perform()	error	SetAppealAgeAodJob .log_error()	Logs errors encountered during the execution of the job
StatsCollectorJob .perform()	error	StatsCollectorJob .log_error()	Logs standard errors encountered while executing the job
StatsCollectorJob .run_collectors()	error	StatsCollectorJob .log_error()	Logs standard errors encountered while starting up the collectors
UpdateAppellantRepresentationJob .perform()	error	UpdateAppellantRepresentationJob()	Logs standard errors encountered while executing the job
UpdateCachedAppealsAttributesJob .perform()	error	UpdateCachedAppealsAttributesJob .log_error()	Logs standard error encountered while executing the job
HearingDay .generate_link_on_create()	error	HearingDay .log_error()	Logs standard errors encountered while the model is creating a link during creation

Datadog

Direct Datadog Calls

Where in Code	Key ID(s)	Method Used	Description
DataDogService .increment_counter()	stat_name	Datadog::Statsd .increment()	Updates a datadog counter
DataDogService .emit_gauge()	stat_name	Datadog::Statsd .gauge()	Sends given statistics to datadog
DataDogService .histogram()	stat_name	Datadog::Statsd .histogram()	Creates a datadog histogram

DataDogService Calls

Where in Code	Key ID(s)	Method Used	Description
MetricsService .record()	app, service	DataDogService .emit_gauge()	Records the time it takes for a given service to run
MetricsService .increment_datadog_counter()	Metric_name, service, endpoint_name	DataDogService .increment_counter()	Increments the count of the given counter
Config.ru emit_datadog_point()	threads	DataDogService .emit_gauge()	Records count of puma threads
ApplicationController .handle_non_critical_error()	endpoint, error_type, err.code	DataDogService .increment_counter()	Keeps count of errors based on error type and code
CollectDataDogMetrics .emit_datadog_point()	Db_name, connection type	DataDogService .emit_gauge()	Standardizes datadog message for tracking database connection counts
Metrics::V1::HistogramController .create()	metric	DataDogService .histogram()	Creates a datadog histogram based on a given metric
CaseflowJob .datadog_report_runtime()	metric_group_name	DataDogService .record_runtime()	Records run time for caseflow jobs
CaseflowJob .datadog_report_time_segment()	segment	DataDogService .emit_gauge()	Records run time for a given caseflow job segment
DispatchEmailJob .external_message_id()	“Email.sent”, email type	DataDogService .increment_counter()	Keeps count of how many emails are sent
UpdateAppellantRepresentationJob .increment_task_count()	METRIC_GROUP_NAME, task_effect, appeal_id	DataDogService .increment_counter()	Keeps track of tasks done for an appeal
UpdateCachedAppealsAttributesJob .increment_vacols_update_count()	APP_NAME, METRIC_GROUP_NAME	DataDogService .increment_counter()	Keeps track of vacols update counts
UpdateCachedAppealsAttributesJob .increment_appeal_count()	APP_NAME, METRIC_GROUP_NAME	DataDogService .increment_counter()	Keeps track of appeal counts
UpdateCachedAppealsAttributesJob .record_success_in_datadog()	APP_NAME, METRIC_GROUP_NAME	DataDogService .increment_counter()	Keeps track of successful runs of the job
UpdateCachedAppealsAttributesJob .record_error_in_datadog()	APP_NAME, METRIC_GROUP_NAME	DataDogService .increment_counter()	Keeps track of unsuccessful runs of the job
WarmBgsCachesJob .warm_poa_for_oldest_cached_records()	Hearings::SendEmail.external_message_id(), DATADOG_METRICS.HEARINGS.APP_NAME, ATADOG_METRICS.HEARINGS.VIRTUAL_HEARINGS_GROUP_NAME	DataDogService .increment_counter()	Keeps track of number of emails sent
Hearings::SendSentStatusEmail .log_to_datadog()	message, hearing.id, hearing.class.name	DataDogService .increment_counter()	Increments a counter to keep track results of sending emails
Memberships::SendMembershipRequestMailerJob .external_message_id()	“Email.error”, requestor, requests	DataDogService .increment_counter()	Increments error count for failed emails
VirtualHearings::CreateConferenceJob .create_conference()	"Created_conference.failed”, hearing_id	DataDogService .increment_counter()	Keeps track of failed jobs for creating a virtual hearing conference
VirtualHearings::CreateConferenceJob .create_conference()	"Created_conference.successful", hearing_id	DataDogService .increment_counter()	Keeps track of successful jobs for creating a virtual hearing conference
VirtualHearings::DeleteConferencesJob .count_deleted_and_log()	deleted_conferences.successful	DataDogService .increment_counter()	Keeps track of successful jobs for deleting a virtual hearing conference
VirtualHearings::DeleteConferencesJob .count_deleted_and_log()	deleted_conferences.failed	DataDogService .increment_counter()	Keeps track of failed jobs for deleting a virtual hearing conference
LegacyAppeal .veteran_file_number()	database_disagreement, external_id	DataDogService .increment_counter()	Keeps track of number of disagreements between vacols and caseflow file numbers
AppealConcern .timezone_identifier_for_address()	error.country_code	DataDogService .increment_counter()	Keeps track of Ambiguous Time zone Error count
BaseHearingUpdateForm .create_or_update_virtual_hearing()	“cancelled_virtual_hearing.successful”	DataDogService .increment_counter()	Keeps track of number of successfully canceled virtual hearings
BaseHearingUpdateForm .create_or_update_virtual_hearing()	“updated_virtual_hearing.successful”	DataDogService .increment_counter()	Keeps track of number of successfully updated virtual hearings
BaseHearingUpdateForm .create_or_update_virtual_hearing()	“created_virtual_hearing.successful”	DataDogService .increment_counter()	Keeps track of number of successfully created virtual hearings
BusinessMetrics .record()	app_name, service	DataDogService .increment_counter()	Used to keep track of the count of a given service
DataDogService .record_runtime()	app_name, metric_group, metric_name	DataDogService .emit_gauge()	Formats information to send to datadog
GeomatchService .record_geomatched_appeal()	appeal.external_id	DataDogService .increment_counter()	Keeps track of number of geomatched appeals
Hearings::ReminderService .send_to_datadog()	Type, hearing.id	DataDogService .increment_counter()	Keeps track of how many reminders of a given type have been created

MetricsService Datadog Calls

Where in Code	Key ID(s)	Method Used	Description
MetricsService .record()	service	MetricsService. increment_datadog_counter()	Increments the counter for request_errors for a service
MetricsService .record()	service	MetricsService. increment_datadog_counter()	Increments the counter for request_attempts for a service
AppealsController .show()	appeal_id	MetricsService. record()	Record time it takes to load or create an appeal
LegacyTasksController .index()	user_id	MetricsService. record()	Record time it takes to load all appeal tasks for a user
Idt::Api::V2::AppealsController .reader_appeal()	appeal_id	MetricsService. record()	Record time it takes to load appeal data
Idt::Api::V2::AppealsController .appeal_documents()	appeal_id	MetricsService. record()	Record time it takes to load appeal document data
Reader::AppealController .show()	vacols_id	MetricsService. record()	Record time it takes to get appeal info related to a Vacols id
Reader::DocumentsController .index()	appeal_id	MetricsService. record()	Record time it takes to get document data for an appeal
FetchAllActiveAmaAppealsJob .add_record_to_appeal_states_table()	appeal_id	MetricsService. record()	Record time it takes to update a record in the Appeal States Table
FetchAllActiveAmaAppealsJob .add_record_to_appeal_states_table()	appeal_id	MetricsService. record()	Record time it takes to create a record in the Appeal States Table
FetchAllActiveLegacyAppealsJob .add_record_to_appeal_states_table()	appeal_id	MetricsService. record()	Record time it takes to update a record in the Appeal States Table
FetchAllActiveLegacyAppealsJob .add_record_to_appeal_states_table()	appeal_id	MetricsService. record()	Record time it takes to create a record in the Appeal States Table
QuarterlyNotificationsJob .perform()	appeal_id	MetricsService. record()	Record time it takes to create a Quarterly Notification
AttorneyCaseReview .update_in_vacols!()	task_id	MetricsService. record()	Record time it takes to reassign a case
JudgeCaseAssignmentToAttorney .assign_to_attorney!()	vacols_id	MetricsService. record()	Record time it takes to assign a case to an attorney
JudgeCaseAssignmentToAttorney .reassign_to_attorney()	vacols_id	MetricsService. record()	Record time it takes to reassign a case to an attorney
JudgeCaseReview .update_in_vacols!()	task_id	MetricsService. record()	Record time it takes to update a Judge Case Review
AppealCancelled .update_appeal_state_when_appeal_cancelled()	appeal.id	MetricsService. record()	Record time it takes to set an appeal’s canceled state to true
AppealDocketed .create_tasks_on_intake_success!()	Appeal id	MetricsService. record()	Record time it takes to send a notification to VA notify
AppealDocketed .docket_appeal()	appeal.id	MetricsService. record()	Record time it takes to send a notification to VA notify
AppealDocketed .update_appeal_state_when_appeal_docketed()	appeal.id	MetricsService. record()	Record time it takes to set an appeal’s docketed status to true
HearingPostponed .update_appeal_states_on_hearing_postponed()	appeal.id	MetricsService. record()	Record time it takes to update an appeal’s HEARING_POSTPONED status
HearingPostponed .update_appeal_states_on_hearing_postponed()	appeal.id	MetricsService. record()	Record time it takes to update an appeal’s HEARING_POSTPONED status
HearingScheduled .update_appeal_states_on_hearing_scheduled()	appeal.id	MetricsService. record()	Record time it takes to update an appeal’s HEARING_SCHEDULED status
HearingScheduledInError .update_appeal_states_on_hearing_scheduled_in_error()	appeal.id	MetricsService. record()	Record time it takes to update an appeal’s SCHEDULED_IN_ERROR status
HearingScheduledInError .update_appeal_states_on_hearing_scheduled_in_error()	appeal.id	MetricsService. record()	Record time it takes to update an appeal’s SCHEDULED_IN_ERROR status
HearingWithdrawn .update_appeal_states_on_hearing_withdrawn()	appeal.id	MetricsService. record()	Record time it takes to update an appeal’s HEARING_WITHDRAWN status
HearingWithdrawn .update_appeal_states_on_hearing_withdrawn()	appeal.id	MetricsService. record()	Record time it takes to update an appeal’s HEARING_WITHDRAWN status
IhpTaskCancelled .update_appeal_state_when_ihp_cancelled()	appeal.id	MetricsService. record()	Record time it takes to update an appeal’s VSO_IHP statuses to false
IhpTaskComplete .update_from_params()	appeal.id	MetricsService. record()	Record time it takes to send a notification to VA Notify
IhpTaskComplete .update_appeal_state_when_ihp_completed()	appeal.id	MetricsService. record()	Record time it takes to update an appeal’s VSO_IHP statuses
IhpTaskPending .create_ihp_tasks!()	parent.appeal.id	MetricsService. record()	Record time it takes to send a notification to VA Notify
IhpTaskPending .create_from_params()	appeal.id	MetricsService. record()	Record time it takes to send a notification to VA Notify
IhpTaskPending .update_appeal_state_when_ihp_created()	appeal.id	MetricsService. record()	Record time it takes to update an appeal’s VSO_IHP statuses
PrivacyActCancelled .update_appeal_state_when_privacy_act_cancelled()	appeal.id	MetricsService. record()	Record time it takes to update an appeal’s PRIVACY_ACT_PENDING status to false
PrivacyActComplete .update_appeal_state_when_privacy_act_complete()	appeal.id	MetricsService. record()	Record time it takes to update an appeal’s PRIVACY_ACT statuses
PrivacyActPending .update_appeal_state_when_privacy_act_created()	appeal.id	MetricsService. record()	Record time it takes to update an appeal’s PRIVACY_ACT statuses
VACOLS::Case .aod()	vacols_ids	MetricsService. record()	Record time it takes to query cases that match the criteria
VACOLS::Case .remand_return_date()	vacols_ids	MetricsService. record()	Record time it takes to query cases that match the criteria
VACOLS::Case .previous_active_location()	bfkey	MetricsService. record()	Record time it takes to query cases that match the criteria
VACOLS::CaseAssignment .exists_for_appeals()	vacols_ids	MetricsService. record()	Record time it takes to query case assignments
VACOLS::CaseHearing .create_hearing!()	“create_hearing!”	MetricsService. record()	Record time it takes to create a hearing
VACOLS::CaseHearing .update_hearing!()	hearing_pkseq	MetricsService. record()	Record time it takes to update a hearing
VACOLS::CaseIssue .descriptions()	vacols_ids	MetricsService. record()	Record time it takes to query case issue descriptions
VACOLS::Correspondent .update_veteran_nod()	Veteran_ssn, veteran_pat	MetricsService. record()	Record time it takes to update a veteran’s NOD
VACOLS::Note .generate_primary_key()	bfkey	MetricsService. record()	Record time it takes to generate the primary key for a note
VACOLS::Note .create!()	note[case_id]	MetricsService. record()	Record time it takes to create a note
VACOLS::Note .delete!()	note[case_id]	MetricsService. record()	Record time it takes to delete a note
VACOLS::Representative .update_vacols_rep_type!()	case_id	MetricsService. record()	Record time it takes to update the rep type for a case
VACOLS::Representative .update_vacols_rep_table!()	bfkey	MetricsService. record()	Record time it takes to update the rep name
VacolsLocationBatchUpdater .call()	“batch_update_vacols_location”	MetricsService. record()	Record time it takes to do a batch update on records in tables
AppealRepository .vacols_records_for_appeals()	“eager_load_legacy_appeals_batch”	MetricsService. record()	Record time it takes to load legacy appeals
AppealRepository .load_vacols_data()	appeal.vacols_id	MetricsService. record()	Record time it takes to load case records
AppealRepository .appeals_by_vbms_id()	“appeals_by_vbms_id”	MetricsService. record()	Record time it takes to load cases that match the query
AppealRepository .appeals_by_vbms_id_with_preloaded_status_api_attrs()	“appeals_by_vbms_id_with_preloaded_status_api_attrs”	MetricsService. record()	Record time it takes to load cases that match the query
AppealRepository .appeals_ready_for_hearing()	“appeals_ready_for_hearing”	MetricsService. record()	Record time it takes to load cases that match the query
AppealRepository .load_vacols_data_by_vbms_id()	appeal.vbms_id	MetricsService. record()	Record time it takes to load cases that match the query
AppealRepository .remands_ready_for_claims_establishment()	“remands_ready_for_claims_establishment”	MetricsService. record()	Record time it takes to load cases that match the query
AppealRepository .amc_full_grants()	outcoded_after	MetricsService. record()	Record time it takes to load cases that match the query
AppealRepository .certify()	appeal.vacols_id	MetricsService. record()	Record time it takes to verify and save a case record
AppealRepository .load_user_case_assignments_from_vacols()	css_id	MetricsService. record()	Record time it takes to load active cases for the user
AppealRepository .docket_counts_by_priority_and_readiness()	“docket_counts_by_priority_and_readiness”	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .genpop_priority_count()	“genpop_priority_count”	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .not_genpop_priority_count()	"not_genpop_priority_count"	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .priority_ready_appeal_vacols_ids()	"priority_ready_appeal_vacols_ids"	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .nod_count()	"nod_count"	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .regular_non_aod_docket_count()	"regular_non_aod_docket_count"	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .latest_docket_month()	"latest_docket_month"	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .docket_counts_by_month()	"docket_counts_by_month"	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .age_of_n_oldest_genpop_priority_appeals()	"age_of_n_oldest_genpop_priority_appeals"	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .age_of_n_oldest_nonpriority_appeals_available_to_judge()	"age_of_n_oldest_nonpriority_appeals_available_to_judge"	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .age_of_oldest_priority_appeal()	"age_of_oldest_priority_appeal"	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .age_of_oldest_priority_appeal_by_docket_date()	"age_of_oldest_priority_appeal"	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .age_of_n_oldest_priority_appeals_available_to_judge()	"age_of_n_oldest_priority_appeals_available_to_judge"	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .nonpriority_decisions_per_year()	"nonpriority_decisions_per_year"	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .distribute_priority_appeals()	"distribute_priority_appeals"	MetricsService. record()	Record time it takes to return the CaseDocket attribute
AppealRepository .distribute_nonpriority_appeals()	"distribute_nonpriority_appeals"	MetricsService. record()	Record time it takes to return the CaseDocket attribute
HearingRepository .load_vacols_data()	hearing.vacols_id	MetricsService. record()	Record time it takes to load a case hearing
IssueRepository .create_vacols_issue!()	vacols_id	MetricsService. record()	Record time it takes to create a vacols issue
IssueRepository .update_vacols_issue!()	vacols_id, vacols_sequence_id	MetricsService. record()	Record time it takes to update a vacols issue
IssueRepository .delete_vacols_issue!()	vacols_id, vacols_sequence_id	MetricsService. record()	Record time it takes to delete a vacols issue
PowerOfAttorneyRepository .load_vacols_data()	poa.vacols_id	MetricsService. record()	Record time it takes to load vacols data for a POA
QueueRepository .tasks_for_user()	“fetch user tasks”	MetricsService. record()	Record time it takes to fetch tasks for a user
QueueRepository .tasks_for_appeal()	“fetch appeal tasks”	MetricsService. record()	Record time it takes to fetch tasks for an appeal
QueueRepository .appeals_by_vacols_ids()	“fetch appeals and associated info for tasks”	MetricsService. record()	Record time it takes to fetch appeals and related info
AppealFinder .find_appeals_with_file_numbers()	file_numbers	MetricsService. record()	Record time it takes to fetch appeal info for the given file numbers
AppealFinder .find_appeals_for_vso_user()	“Get vso appeals information for veterans”	MetricsService. record()	Record time it takes to fetch appeals for a given participant
ExternalApi::BgsAddressFinder .fetch_addresses()	participant_id	MetricsService. record()	Record time it takes to get the address for a participant
ExternalApi::BGSService .get_end_products()	vbms_id	MetricsService. record()	Records time it takes get end products for a given vbms
ExternalApi::BGSService .cancel_end_product()	veteran_file_number, end_product_code, end_product_modifier, payee_code, benefit_type	MetricsService. record()	Records time it takes to cancel a given end product
ExternalApi::BGSService .update_benefit_claim()	veteran_file_number, payee_code, claim_date, benefit_type_code, modifier, new_code	MetricsService. record()	Records time it takes to update a benefit claim
ExternalApi::BGSService .fetch_veteran_info()	vbms_id	MetricsService. record()	Records time it takes to fetch a veteran’s info based on a given vbms
ExternalApi::BGSService .fetch_person_info()	participant_id	MetricsService. record()	Records time it takes to fetch a person’s info based on a given participant id
ExternalApi::BGSService .fetch_person_by_ssn()	ssn	MetricsService. record()	Records time it takes to fetch a person’s info based on a given ssn
ExternalApi::BGSService .fetch_poa_by_file_number_by_claimants()	file_number	MetricsService. record()	Records time it takes to fetch the poa for a file
ExternalApi::BGSService .fetch_poa_by_file_number_by_org()	file_number	MetricsService. record()	Records time it takes to fetch the poa for a file
ExternalApi::BGSService .fetch_poas_by_participant_id()	participant_id	MetricsService. record()	Records time it takes to fetch the poas that a participant holds
ExternalApi::BGSService .fetch_poas_by_participant_ids()	participant_ids	MetricsService. record()	Records time it takes to fetch the poas that a group of participant hold
ExternalApi::BGSService .fetch_limited_poas_by_claim_ids()	claim_ids	MetricsService. record()	Records time it takes to fetch the poas associated with a a set of claims
ExternalApi::BGSService .fetch_poas_list()	“fetch list of poas”	MetricsService. record()	Records time it takes to fetch all powers of attorney
ExternalApi::BGSService .get_security_profile()	“security profile”	MetricsService. record()	Records time it takes to get the security profile of a user and/or station
ExternalApi::BGSService .can_access?()	vbms_id	MetricsService. record()	Records time it takes to determine if the user can access a file
ExternalApi::BGSService .fetch_ratings_in_range()	participant_id	MetricsService. record()	Records time it takes to get all ratings of a participant within a given date range
ExternalApi::BGSService .fetch_rating_profile()	participant_id	MetricsService. record()	Records time it takes to get a participant’s rating profile on a given date
ExternalApi::BGSService .fetch_rating_profiles_in_range()	participant_id	MetricsService. record()	Records time it takes to get a participant’s rating profiles within a given date range
ExternalApi::BGSService .fetch_claimant_info_by_participant_id()	participant_id	MetricsService. record()	Records time it takes to fetch claimant info
ExternalApi::BGSService .find_all_relationships()	participant_id	MetricsService. record()	Records time it takes to fetch relationships for a given participant
ExternalApi::BGSService .get_participant_id_for_css_id_and_station_id()	css_id, station_id	MetricsService. record()	Records time it takes to find the participant id for a user
ExternalApi::BGSService .find_claimant_letters()	document_id	MetricsService. record()	Records time it takes to fetch the claimant letter for a given document
ExternalApi::BGSService .manage_claimant_letter_v2!()	program_type_cd, claimant_participant_id	MetricsService. record()	Records time it takes to create a claimant letter
ExternalApi::BGSService .generate_tracked_items!()	claim_id	MetricsService. record()	Records time it takes to generate tracked items for a claim
ExternalApi::BGSService .find_contentions_by_claim_id()	claim_id	MetricsService. record()	Records time it takes to find a veteran’s contentions for a claim
ExternalApi::BGSService .find_current_rating_profile_by_ptcpnt_id()	participant_id	MetricsService. record()	Records time it takes to get the current rating profile of a participant
ExternalApi::BGSService .pay_grade_list()	“fetch list of pay grades”	MetricsService. record()	Records time it takes to fetch the pay grade list
ExternalApi::BgsVeteranStationUserConflict .employee_dtos()	veteran_participant_id	MetricsService. record()	Records time it takes to fetch a dtos employee by participant id
ExternalApi::BgsVeteranStationUserConflict .sensitivity_level()	veteran_participant_id	MetricsService. record()	Records time it takes to fetch the sensitivity level by participant id
ExternalApi::EfolderService .send_efolder_request()	url	MetricsService. record()	Records time it takes to resolve a GET request to eFolder
ExternalApi::GovDeliveryService .send_gov_delivery_request()	url, method.to_s.upcase	MetricsService. record()	Records time it takes to send a gov delivery request
ExternalApi::MPIService .search_people_info()	last_name, first_name, middle_name, ssn, date_of_birth, gender, address, telephone	MetricsService. record()	Records time it takes to find a person based on the given criteria
ExternalApi::MPIService .fetch_person_info()	icn	MetricsService. record()	Records time it takes to find a person’s info based on an icn
ExternalApi::PexipService .send_pexip_request()	host, method.to_s.upcase, url	MetricsService. record()	Records the time it takes to send a pexip request
ExternalApi::VADotGovService .send_va_dot_gov_request()	method.to_s.upcase, url	MetricsService. record()	Records time it takes to send a VA.gov request
ExternalApi::VANotifyService .send_va_notify_request()	method.to_s.upcase, url	MetricsService. record()	Records time it takes to send a VA notify request
ExternalApi::VBMSRequest .call()	request.class, request.id	MetricsService. record()	Records time it takes to send a VBMS request
VBMSCaseflowLogger .send_and_log_request()	vbms_id	MetricsService. record()	Records time it takes to send a VBMS request
VBMSCaseflowLogger .call_and_log_service()	Vbms_id, service.class	MetricsService. record()	Records time it takes to call a VBMS service
config/initializers/warmup_vacols.rb	initial_pool_size	MetricsService. record()	Records the time it takes to warm up the vacols connections

CaseflowJob Datadog Calls

Where in Code	Key ID(s)	Method Used	Description
DataIntegrityChecksJob .perform()	Segment klass.underscore	CaseflowJob. datadog_report_time_segment()	Records runtime for each data integrity Checker that is run
ETLBuilderJob .sweep_etl()l	“etl_sweeper”	CaseflowJob. datadog_report_time_segment()	Records runtimes of ETL sweeper jobs
ETLBuilderJob .build_etl()	“etl_builder”	CaseflowJob. datadog_report_time_segment()	Records runtimes of ETL builder jobs
NightlySyncsJob .sync_vacols_users()	"sync_users_from_vacols"	CaseflowJob. datadog_report_time_segment()	Records runtimes of Vacols user sync jobs
NightlySyncsJob .sync_vacols_cases()	“sync_cases_from_vacols”	CaseflowJob. datadog_report_time_segment()	Records runtimes of Vacols case sync jobs
NightlySyncsJob .sync_bgs_attorneys()	“sync_bgs_attorneys”	CaseflowJob. datadog_report_time_segment()	Records runtimes of BGS attorney sync jobs
PushPriorityAppealsToJudgesJob .perform()	"priority_appeal_push_job"	CaseflowJob. datadog_report_time_segment()	Records runtimes of pushing priority appeals to judges
UpdateAppellantRepresentationJob .log_error()	METRIC_GROUP_NAME	CaseflowJob. datadog_report_runtime()	Records runtime of the metric group when an error occurs
UpdateCachedAppealsAttributesJob .perform()	“Cache_ama_appeals”, ama_appeals_start	CaseflowJob. datadog_report_time_segment(	Records runtime for ama appeals caching jobs
UpdateCachedAppealsAttributesJob .perform()	“cache_legacy_appeals”, legacy_appeals_start	CaseflowJob. datadog_report_time_segment(	Records runtime for legacy appeals caching
UpdateCachedAppealsAttributesJob .cache_legacy_appeals()	“cache_legacy_appeal_postgres_data”, cache_postgres_data_start	CaseflowJob. datadog_report_time_segment()	Records
UpdateCachedAppealsAttributesJob .cache_legacy_appeals()	“cache_legacy_appeal_vacols_data”, cache_vacols_data_start	CaseflowJob. datadog_report_time_segment()	Records
UpdateCachedAppealsAttributesJob .log_error()	METRIC_GROUP_NAME	CaseflowJob. datadog_report_runtime()	Records runtime of the job

Other Indirect Datadog Calls

Where in Code	Key ID(s)	Method Used	Description
CollectDataDogMetrics .collect_postgres_metrics()	"postgres", status count	CollectDataDogMetrics .emit_datadog_point()	Keeps count of active/dead/idle postgres connections
UpdateAppellantRepresentationJob .perform()	appeal.id	UpdateAppellantRepresentationJob .increment_task_count()	Keeps track of new/closed/error task counts for appeals
UpdateCachedAppealsAttributesJob .perform()	n/a	UpdateCachedAppealsAttributesJob .record_success_in_datadog()	Increments successful job count
UpdateCachedAppealsAttributesJob .log_error()	n/a	UpdateCachedAppealsAttributesJob .record_error_in_datadog()	Keeps track of number of errors the job encounters
Hearings::SendSentStatusEmail .log()	Email message	Hearings::SendSentStatusEmail .log_to_datadog()	Passes through the email message to datadog incrementer
Hearings::SendSentStatusEmail .call()	“sent admin email”	Hearings::SendSentStatusEmail .log()	Increments sent email count
Hearings::SendSentStatusEmail .send_email()	“failure to send email”	Hearings::SendSentStatusEmail .log()	Increments sent email failure count
Hearings::SendSentStatusEmail .email_missing?()	“email missing”	Hearings::SendSentStatusEmail .log()	Increments missing email count