Set up Data to Demonstrate Progress on Time to Deploy Content

[gracekretschmer-metrostar]

User Story or Problem Statement

As the PO and Engineering Lead for CMS, I need to demonstrate progress in the time to deploy content as Next Build goes into production.

Description or Additional Context

Ahead of the removal of events and event listing from content build and based upon the findings from the UX research, update the Next Build and Content Build dashboard, the CMS team needs to update the dashboards to demonstrate progress in the Next Build work.

Acceptance Criteria

• Data in data dog that demonstrates progress on the time to deploy content.

[gracekretschmer-metrostar]

Next steps: get through next build launch and revisit this ticket to see if it needs additional refinement.

[timcosgrove]

This represents the first round of data on this.

Assumptions

• We report time for a deploy cycle
  • For Content Build, the next one begins when the previous one ends, so we take an average of cycle times, which can vary
  • For Next Build, we currently run on a 30 minute cycle, regardless of the actual release time, so we are using 30 minutes here.
• Improvement is reported (old / new), so as a multiple: "The new system is 1.6x faster"

Event & Event Listing publishing time changes

Prior average cycle: 62.94 min
Current cycle: 30.00 min

Improvement: 62.94 / 30 = 2.1x faster

Other content publishing time changes

Prior average cycle: 62.94 min
Current average cycle: 44.64 min

Improvement: 62.94 / 44.64 = 1.4x faster

[timcosgrove]

Data sources for content timing

Content Build timing data can most easily be seen from the CMS overview dashboard: https://vagov.ddog-gov.com/dashboard/vnk-g4s-fru/cms-prod--staging?fromUser=true&refresh_mode=sliding&from_ts=1733955026059&to_ts=1734559826059&live=true

The specific time where Content Build timing data changed is here: https://vagov.ddog-gov.com/dashboard/vnk-g4s-fru/cms-prod--staging?fromUser=true&refresh_mode=paused&from_ts=1733956959508&to_ts=1734558088491&live=false

Note that Content Build reports its total time for the Content Release to take place, and that Content Releases for Content Build are lined up continuously. This means this average number can be used to report time-to-publish.

For Next Build, the Content Release data can be viewed on the Next Build dashboard: https://vagov.ddog-gov.com/dashboard/27n-3bm-skx/next-build-content-release?fromUser=false&refresh_mode=sliding&from_ts=1734545643666&to_ts=1734560043666&live=true

However, Next Build does not release back-to-back the way Content Build does. Next Build currently runs a content release every 30 minutes.

This means that time-to-publish is 30 minutes for Next Build content, regardless of the time it takes the Next Build content release to deploy.

[timcosgrove]

Notes for the future

If we are measuring actual time-to-publish - that is, time from when an editor hits 'publish' on their content to the time when it is available to the Veteran - then this current data does not directly represent that. We only measure the length of time it takes for a content release process to take place, and we infer from that.

In order to properly measure time-to-publish, a few things would need to be in place

1. We would need to track the exact time when a given piece of content is published
2. The content release system, whether Next Build or Content Build, would need to report back to the CMS that content has reached va.gov
3. The CMS would need to calculate the difference between those values and report it to DataDog as a timing number.

This is a non-trivial effort. It would be good to have these numbers at hand. Also, of these, 1 essentially already exists in the form of the workflow system, and 2 would have other value (it could be valuable to report back to the CMS that a piece of content is on production - that is valuable information to editors.

Still, we need to weigh the need for this information against the effort that it would take to collect the information.