A/B test IOU invite notifications

[danielrvidal]

Problem: Users often get notifications from Expensify and never log in. We know this happens, but we don’t know how often it happens because we don’t track it. Thus, we don’t know how big of a problem this actually is, which is a problem in itself.

Solution: Start measuring and a/b testing CRITICAL emails that drive conversion of new users. We had a long thread where we ideated on a new email format for invites (and in general) that look much more like the full chat UI than our current emails. The bottom-up flow for IOUs is our most important invite email because it brings in the decision maker who can bring on more users. Thus, let’s create a baseline of how often people are opening and clicking the CTA in the current email (image 1), and then let’s test the updated version (image 2) to validate the proposed email format converts better. If it does, we’d look to update all of our emails with this formatting.

cc @shawnborton who helped create the mock ups for the invites.

Here are the next steps:

1. @anmurali and I are getting the data for a baseline with the current IOUs.
2. @shawnborton and I can finalize the mock up for the updated IOU.
3. We'll get the updated email built. @justinpersaud, have an interest as I think you are helping with the a/b testing?
4. We'll start testing.

[justinpersaud]

Sure, I am prioritizing my time with fighting spam in #fireroom-2024-05-23-spam right now but I should be able to help with the A/B stuff on the side. Anu was already asking some questions about it.

[shawnborton]

Just let me know how I can be helpful with mocks!

[danielrvidal]

Sounds good, I'll work with @shawnborton early next week!

[danielrvidal]

I was talking with @jamesdeanexpensify and @trjExpensify about how our proposal that landed in that thread is actually slightly different from the text currently live in the product.

The text that is in the proposed mock is from this doc: https://docs.google.com/document/d/19TNWIrcPh4X-sr5ycWYYb-LfSPAb8fVx005asviM2II/edit?pli=1#heading=h.sv15vxk1s1ee

So I think @jamesdeanexpensify is going to create a GH to go forward with those updates. So we can assume the UI from the DM is going to use the text that is in the proposed mock included. So I don't know if we need new mocks, we can just go with the one @shawnborton actually already created.

[melvin-bot]

@danielrvidal, @anmurali Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

[danielrvidal]

@shawnborton @jamesdeanexpensify @anmurali I'm thinking we should pause this one until @jamesdeanexpensify gets the copy updated for the DM chat. Otherwise we'd be testing something that is going to change.

@jamesdeanexpensify is working on that here: #47427

I'll ask @jamesdeanexpensify what the next steps are on that issue though just so we can align on timing/expectations.

[shawnborton]

That works for me.

[danielrvidal]

Waiting on the f1 room to open then I'm going to drop an update in there.

[danielrvidal]

I created an updated here. I'll be creating the IOU here shortly:
https://expensify.slack.com/archives/C07HPDRELLD/p1724245865161379

[danielrvidal]

We're getting the text updated so we'll start working on the updates this week.

[jamesdeanexpensify]

@danielrvidal we might need to push a bit harder on getting this issue assigned (either in #whatsnext, the F1 room, or both). And maybe we should make it a Daily?

[danielrvidal]

Yea, we need to figure out who is curating the room so this could be called out as unassigned in the weekly update. Then we could also tag hot picks to make the engineers more aware.

[danielrvidal]

Moving this to weekly as we need to get the beginningOfChatHistory updated first.

[danielrvidal]

This is in review. Moving to weekly.

[danielrvidal]

I sent out the doc two days ago,

[danielrvidal]

Detailed is going out shortly for the trackingKey's so we'll be conducting this soon.

[danielrvidal]

We're still working on this. Subject lines detailed is being done.

[danielrvidal]

Subject lines test is getting deployed on Monday.

[anmurali]

Latest is here
@danielrvidal is testing it.

[danielrvidal]

Update here on the subject line testing. https://expensify.slack.com/archives/C07HPDRELLD/p1731957768195549?thread_ts=1730161250.824289&cid=C07HPDRELLD

[danielrvidal]

This is still being discussed here: https://expensify.slack.com/archives/C07HPDRELLD/p1733262477107509?thread_ts=1730161250.824289&cid=C07HPDRELLD

[danielrvidal]

Our latest update here: https://expensify.slack.com/archives/C07HPDRELLD/p1733862564966199?thread_ts=1730161250.824289&cid=C07HPDRELLD

@anmurali is creating another issue to store tags as NVPs.

[danielrvidal]

We're waiting for data