Candidate generation (with attention check)

[glciampaglia]

Currently we are ranking all available tweets in the home timeline. Since the feed reading task will consist of 5 screens of 10 tweets each, we need to identify 50 tweets. To do so, we take the top k tweets with links to Newsguard domains (k < 50) and mix them with 50 - k tweets that do not have links to NG domains (see #47). However, our outcomes of interest are about changes in engagement with low-quality NG domains due to re-ranking, so this introduces the problem that if a low-quality NG domain is re-ranked outside of the top k, it will never be shown.

To avoid this, we need to decouple the ranking phase from the candidate generation phase, which is where we identify all the tweets that will be ranked. The former depends on the treatment (relevance, relevance + diversity, reverse chronological), while the latter is independent of treatment.

One possible way to do so is to identify a set of tweets with NG domains so that the proportion of low- to high-quality sources matches the baseline frequency in the home timeline of the user. We could select the most recent ones, or the ones with most engagements, or a mix of the two. We need to discuss which one to choose.

[glciampaglia]

We are not going to do attention checks anymore, moving to utopia and archiving.