AB test - replace some content on UTR page #2947
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
richardTowers
force-pushed
the
ab-test-video-killed-the-utr-star
branch
from
159f475 to
128cde5
Compare
richardTowers
force-pushed
the
ab-test-video-killed-the-utr-star
branch
from
57e7ef7 to
25e5436
Compare
HMRC would like to see whether a link to a video performs better than some descriptive help text. We've agreed to run an AB test to compare these variants. There's no elegant way to AB test bits of content in GOV.UK, so we've had to resort to the fragile and ugly approach of find / replacing the content. As per the comments in the code, this will break if the content in the content item changes, and we'll have to update the hardcoded replacement if the content in the B variant needs to change. Nevertheless, we think this is a pragmatic thing to do as a one off, as it keeps an immportant stakeholder happy, helps us learn about video content (which is a strategic priority), and helps us learn about the use case for content AB tests in mainstream content.
We don't want to do the replacement in the B variant if the list item on the page has any text after the link to the HMRC app. For example, if the list item was - in the HRMC app which is really good We would not want the replacement to be - in the HMRC app - watch a video about finding your UTR number in the app _which is really good_
richardTowers
force-pushed
the
ab-test-video-killed-the-utr-star
branch
from
25e5436 to
366be79
Compare
richardTowers
changed the title
richardTowers
commented
Oct 6, 2023
hannako
reviewed
Oct 6, 2023
hannako
reviewed
Oct 6, 2023
|
Just popping this back in draft because I'm going on holiday. It's not ready to merge until:
|
As suggested by @hannako, we can make the AB test replacement less fragile by looking for a specific replacement placeholder in the content item, instead of replacing a particular bit of content. I've gone for a [mustache](https://mustache.github.io/) style replacement string of {{ab_test_find_utr_number_video_links}}. There's a possible future thought of doing other kinds of templating, so I think starting to socialise this syntax is a good idea. I've checked in the govspeak preview app, and the `{{...}}` syntax is ignored by govspeak so there should be no issue entering it in the publishing app. I've pulled the replacement strings out into the translation file so that they're easier to review / keep up to date. I've also switched to three A / B / Z variants, so we can do splits like 5% / 5% / 90% where we're measuring the difference between the two 5% samples and ignoring the 90% sample. This test is probably going to be 50% / 50%, so not strictly necessary, but this seems to be the idiomatic way to do it.
richardTowers
force-pushed
the
ab-test-video-killed-the-utr-star
branch
from
c3bc3ca to
0f30c12
Compare
The tests insist on this 😴
I did this quickly in vim, by:
1) Opening all the files with `vim config/locales/*.yml`
2) Copying the YAML fragment to add
3) Pasting it after line 2 in all files by running:
:argdo execute "normal! 2Gp" | w
4) Making sure the en.yml file only had one set of keys
richardTowers
force-pushed
the
ab-test-video-killed-the-utr-star
branch
from
97e613f to
ebf40f7
Compare
This isn't required for GA4, but just to make sure our test is compatible with the older UA code I've picked a custom dimension from the spreadsheet (linked in a comment).
In the situation where this code is deployed, but the content item
doesn't yet contain the placeholder to be replaced, we don't want to do
any of the AB testing code (i.e. setting / reading of cookies, sending
events to google analytics).
This means this code can safely be deployed to production, and the AB
test will only begin once the magic placeholder {{ab_test_find_utr_number_video_links}}
appears in the content item.
richardTowers
force-pushed
the
ab-test-video-killed-the-utr-star
branch
from
6598eb4 to
d272c76
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://trello.com/c/jKWA7Nu3/543-ab-test-content-for-find-your-utr-number
HMRC would like to see whether a link to a video performs better than
some descriptive help text. We've agreed to run an AB test to compare
these variants.
There's no elegant way to AB test bits of content in GOV.UK, so we've
had to resort to a workaround where we look for a particular replacement
string in the content (in this case
{{ab_test_find_utr_number_video_links}})and replace it with a value from the translations file, depending on which variant
the user should see.
The code will only be executed on the /find-utr-number page, and only if the placeholder
string appears in the content item's body. This will allow us to deploy this code to production,
and treat the introduction of the placeholder string in the content item as the trigger to
start the test. We can try this out in the draft stack in integration and production before
running it in the live content item.
We think this is a pragmatic thing to do as a one off, as
it keeps an immportant stakeholder happy, helps us learn about video
content (which is a strategic priority), and helps us learn about the
use case for content AB tests in mainstream content.