Skip to content

Incident Response

Jump to bottom
Bianca Rivera Alvelo edited this page Oct 24, 2024 · 30 revisions

Introduction

The VRO Platform is a moderately complex system that integrates with other systems and is hosted on a shared infrastructure. Incidents of unplanned service failures and disruptions are inevitable. The first objective of the incident response process is to restore a normal service operation as quickly as possible and minimize the incident's impact. This document describes the core steps for responding to incidents.

Definition

The VRO team defines an incident as an unplanned event or occurrence that disrupts normal operations, services, or functions on the VRO platform. These negatively impact the availability, performance, security, or functionality of a VRO service and require immediate attention to mitigate its effects and restore normalcy. Incidents can vary widely in scope and severity and can be caused by factors in or out of the VRO team's control.

Incident Type Description
Service Outages Complete or partial unavailability of infrastructure services.
Performance Degradation Noticeable slowdown or inefficiency in infrastructure services.
Security Breaches Unauthorized access, data breaches, or vulnerabilities affecting infrastructure integrity, confidentiality, or availability.
Operational Failures Failures in deployment pipelines, configuration management, or automated processes impacting normal operations.
Resource Exhaustion Over-utilization or exhaustion of resources leading to degraded service.
Unexpected Behavior Anomalies or unexpected behaviors in infrastructure services affecting development, testing, or deployment activities.

Root Cause

The root cause of an incident is investigated and attributed to the team responsible for the issue’s origin - not necessarily the team addressing it - to ensure accurate reporting and unbiased resolution efforts. The GitHub issue for an incident should be annotated with the RC label corresponding to the root cause and should also be documented in the Incident Report.

Note

During the resolution process of an incident, the root cause is not included in updates or communications with partner teams to avoid bias and ensure impartiality.

Root Cause Type Description
VRO An issue on the VRO platform that ties directly to our team's scope of responsibilities. These incidents are tracked to measure our MTTR (Mean Time to Resolve) metric.
Partner Team Application or External VA An issue with the partner team application controlled by the partner team, or due to a VA external system not functioning appropriately
LHDI An issue on the LHDI platform. These are not within the VRO team's control, but the VRO team reports them to LHDI and works to resolve them in partnership with LHDI.

Role and Responsibilities

As a default, incident response is the responsibility of the VRO primary on-call engineer, which rotates with every VRO sprint period. Throughout the process, they might personally conduct each step or delegate tasks as needed; regardless, a single individual should be identified as being in charge of the incident response. If this responsibility needs to be transferred while an incident is active, then this handoff should be explicitly communicated. Working hours are from 9am - 5pm ET. Any incidents that occur or are reported outside of these hours will be addressed when the on-call engineer resumes their working schedule.

See the On Call Responsibilities page for more information.

Process

The Report a VRO Incident Slack Workflow is the intake form and process that should be used to escalate any problems discovered. It is bookmarked in the #benefits-vro-support channel and the #benefits-vro-on-call channel. This workflow should be used for reporting any incidents discovered internally (i.e., by the on-call engineer and other VRO team members) and externally (i.e., by a partner team or third party).

Incident Response flowchart

Generated using draw.io. Source file: incidentResponse.drawio.txt (remove the .txt extension to use in draw.io)

Demo Videos: How to use the 'Report a VRO Incident' Slack workflow

Below are two videos demonstrating the steps and tasks the VRO on-call engineer and the partner teams go through when using the workflow.

Note

There are some discrepancies between the language and formatting of the steps in the current iteration of the workflow steps and the version presented in this video.

Partner Team View
ReportIncidents_PartnerTeamView.mp4

0:00: Find the Incident Report bookmark in #benefits-vro-support. 0:07: Fill out the form. 0:56: Observe the automated post. 1:08: Observe the acknowledgment from the responding engineer. 1:24: Post additional comments in the thread. 1:44: Receive status updates in the thread. 2:03: Receive notification when the incident is resolved.

VRO Team View
ReportIncidents_VROTeamView.mp4

0:01: Step 0: React with 👀 on the Incident Report Slack post. 0:07: Step 0: Click the Acknowledge button on the PagerDuty post. 0:14: Step 1: Post an initial update 0:20: Step 2: Post internal notes to #benefits-vro-on-call 0:40: Step 2: Post general updates to #benefits-vro-support 1:12: Click the Next Step button as tasks are completed. 1:38: Step 3: Update the GitHub issue and the Incidents epic 2:51: Click the Next Step button as tasks are completed. 3:03: Step 5: Log the incident on the wiki 4:10: Step 5: Close the GitHub issue 4:20: Click the Next Step button as tasks are completed. 4:26: Step 5: React with :support-complete: on the Incident Report slack post

Step 0: Acknowledge

Description Purpose Estimated time to complete SLA
Communication to the person escalating a potential incident that VRO will begin an investigation. Reduce the likelihood of uncoordinated troubleshooting efforts, reduce panic, and establish consistent data points for calculating incident metrics 2 minutes within 60 minutes of the report

Tasks

Tasks for this step vary depending on the source or person reporting the incident.

  • 'Report a VRO Incident' Slack workflow (0:01 - 0:13)

    • React with 👀 on the Incident Report post in #benefits-vro-support
    • Click the Acknowledge button on the PagerDuty post in #benefits-vro-on-call

  • Non-workflow Slack post

    • React with 👀 to the post
    • Use the Report a VRO Incident Slack workflow
    • React with 👀 on the post in #benefits-vro-support

  • Email message or Slack DM from a known partner or stakeholder

    • Reply: "Thank you for reporting this incident. If you have access to the VA OCTO slack, the resolution of this incident will be documented in the #benefits-vro-support channel."
    • Use the Report a VRO Incident Slack workflow
    • React with 👀 on the Incident Report post in #benefits-vro-support

  • Email message from an unknown party

Step 1: Triage

Description Purpose Estimated time to complete SLA
Sharing a brief assessment that determines an initial severity level (SEV) and the affected systems and next steps. Gain situational awareness, respond with appropriate urgency, and provide a prompt update 10 minutes within 30 minutes of Step 0

Tasks

  • Post an initial update in the thread of the automated workflow message on the #benefits-vro-support channel. Use a message appropriate for the assessed Severity level (see language below).

Important

  • Updates posted on the #benefits-vro-support channel are for communicating general updates that foster shared situational awareness to an external audience (i.e., partner teams and VA stakeholders)
  • Updates posted on the #benefits-vro-on-call channel are for documenting more granular details that are necessary or relevant only to an internal audience (i.e., the VRO team).

Initial update by severity level:

SEV 1: Critical - Core functionality is unavailable or buggy

Examples: a VRO app appears offline; a VRO app's transactions are failing; a VRO data service appears unresponsive; inaccurate data is transmitted

Priority: immediate investigation

Initial Message:

  • Confirmation that investigation is underway
  • The next updates will be sent 30 minutes after an investigation has begun

Sample language:

"We are currently investigating this incident. We will provide detailed updates of our findings and actions every ~30 minutes until the issue is resolved."

SEV 2: High - Core functionality is degraded

Examples: increased latency; increased retry attempts

Priority: immediate investigation

Initial Message:

  • Confirmation that investigation is underway
  • The next updates will be sent 30-60 minutes after an investigation has begun

Sample language:

"We are currently investigating this incident. We will provide detailed updates of our findings and actions every ~30 - 60 minutes until the issue is resolved."

SEV 3: Medium - Core functionality metrics are affected, but without noticeable performance degradation

Examples: sustained increase in CPU utilization; sustained increased in open database connections

Priority: investigation within the next business day; continued passive monitoring in the meantime

Initial Message:

  • Confirmation that investigation will begin in the next business day
  • The next updates will be sent every 1 hour after an investigation has begun

Sample language:

"We will begin our investigation into this incident within the next business day. We will provide detailed updates of our findings and actions every hour until the issue is resolved."

SEV 4: Low - Non-core functionality is affected

Examples: gaps or increased latency in transmitting data to an analytics platform

Priority: investigation within the next 1-2 business days is limited to solely identifying the root cause; continued passive monitoring in the meantime

Initial Message:

  • Confirmation of when the investigation begins
  • The next updates will be sent as needed

Sample language:

"We will begin our investigation into this incident at our earliest convenience. We will provide updates on our findings and actions as needed."

All subsequent update messages should include the following information:

  • Current status and progress made so far (including the root cause of the issue if identified)
  • Specific actions taken since the last update
  • Any changes to the estimated resolution time
  • Next update time

Tip

  • Use succinct and specific language; avoid jargon or acronyms that non-VRO team members may not be familiar with.
  • Use Slack's /remind feature to alert you to when the next update is due. For example: /remind me in 30 minutes to post an update.

Step 2: Contain/Stabilize

Description Purpose Estimated time to complete SLA
Work to prevent further damage and provide status updates. Containing the situation might provide more immediate relief than implementing a remediation varies SEV 1, SEV 2: immediately after Step 1; SEV 3, SEV 4: as soon as VRO can prioritize

Tasks

  • Post internal notes to #benefits-vro-on-call
  • Post updates in the thread of the #benefits-vro-support message within the frequency defined for the respective Severity.
  • Click the Next Step button on the Incident Response workflow.

Considerations: Is there a configuration change to prevent requests to the buggy system? Would an increase in computing resources temporarily stabilize the system?

Step 3: Remediate (short-term)

Description: Get the system back to a minimally acceptable operating status.

Purpose: Reduce likelihood that the incident will reoccur in the short-term; increase likelihood that the system will be stable.

Estimated time to complete: varies

SLA: SEV 1, SEV 2: top priority for the on-call engineer; SEV 3, SEV 4: as soon as VRO can prioritize

Considerations: Should compute resources be recalibrated? Would a rollback or roll-forward of code/configuration would be appropriate and feasible?

Tasks:

  1. Post general updates to the Incident Report thread in #benefits-vro-support within the frequency defined for the respective Severity.
  2. Update the GitHub issue that was created by the Incident Report workflow.
  • Add blue VRO-team label
  • Add root cause label
    • RC VRO
    • RC LHDI
    • RC Partner Team or External VA
  • Assign to the engineer(s) responding to the incident
  • Include SEV level
  • Add to the current sprint
  • Add to the Incidents Epic
  • Arrange to discuss the incident as a 16th minute item during Daily Scrum
  1. Click the Next Step button on the Incident Response workflow.
Step 4: Monitor

Step 4: Monitor

Description: Look for data points that indicate the incident is under control. As needed, return to Step 2 and Step 3.

Purpose: gain confidence that the incident is under control

Estimated time to complete: at minimum: 30 minutes

SLA: n/a

Tasks:

  1. (as needed) Post internal notes to #benefits-vro-on-call.
  2. Post general updates to the Incident Report thread in #benefits-vro-support within the frequency defined for the respective Severity.
Step 5: Log the incident

Step 5: Log the incident

Description: Document the incident in the VRO wiki's Incident Reports.

Purpose: build a record of incidents that can reveal patterns, inform engineering decisions, and be a general resource

Estimated time to complete: 30 minutes

SLA: within 8 business hrs of the incident's resolution

Considerations: Account for these details: how the incident was detected, including timestamp; severity level; corrective measures taken; timestamp of when system returned to operating status; "red herrings" that were encountered; follow-up tasks

Tasks:

  1. Create an Incident Report on the wiki page.
  2. Close the GitHub issue that was created by the Incident Report workflow.
  3. React with :support-complete: on the Incident Report post in #benefits-vro-support.
Step 6: Post-incident review

Step 6: Post-incident review

Description: As a more in-depth analysis, assess what happened, what went well, what did not go well, and measures to prevent a recurrence. Describe troubleshooting measures, including log snippets and command line tools. Share this document with the VRO team and with partner teams. Expected for SEV 1 and SEV 2 incidents; at team's discretion for SEV 3 and SEV 4 incidents.

Purpose: leverage the incident as a learning opportunity; surface further corrective measures

Estimated time to complete: 4 hours

SLA: within 5 business days of the incident's resolution

Considerations: Follow principles of blameless post-mortems:

focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior.

Tasks:

  1. Create a Post-incident review on the private wiki.
Step 7: Discuss longer-term remediation

Step 7: Discuss longer-term remediation

Description: Determine measures that would reduce the likelihood of this incident recurring and/or give the team better visibility into conditions that led to this incident. Propose these to the Product Manager for consideration.

Purpose: Consider remediation measures that could not be achieved in the short term response.

Estimated time to complete: varies

SLA: within 2 sprint cycles of the incident's resolution

Tasks:

  1. Document recommended remediation steps.
  2. Share these with the Product Manager.
  3. Be prepared to present the steps during a backlog refinement session.
Clone this wiki locally