feat(pipeline) : Simplify notify_rgpd_contacts #326
Conversation
pipeline/dbt/models/intermediate/int__union_contacts__enhanced.sql
Outdated
Show resolved
Hide resolved
pipeline/dbt/models/intermediate/int__union_contacts__enhanced.sql
Outdated
Show resolved
Hide resolved
| ), | ||
|
|
||
| final AS ( | ||
| SELECT | ||
| contacts.source || '-' || contacts.id AS "_di_surrogate_id", | ||
| contacts.id AS "id", | ||
| contacts.source AS "source", | ||
| contacts.courriel AS "_courriel_original", |
There was a problem hiding this comment.
j'ai l'impression que ce champ a 2 fonctions ici :
- permettre d'avoir la vue avant/après
- permettre de récupérer la liste des mails à charger dans brevo
Pour 1. , ça n'est pas nécessaire. Il suffira de faire une jointure avec int__union_contacts.
Pour 2., cf commentaire plus haut
There was a problem hiding this comment.
Pas totalement certain de comprendre ta remarque, cf. moi aussi plus haut :)
| """ | ||
| SELECT | ||
| json_build_object('email', _courriel_original), | ||
| rgpd_notice_has_hardbounced | ||
| FROM public_intermediate.int__union_contacts__enhanced | ||
| GROUP BY _courriel_original, rgpd_notice_has_hardbounced | ||
| ORDER BY _courriel_original | ||
| """ |
There was a problem hiding this comment.
pourquoi utiliser rgpd_notice_has_hardbounced ?
utiliser int__union_contacts serait moins compliqué (pas de rétroaction)
There was a problem hiding this comment.
On utilise (détourne ?) le flag hardbounced pour savoir si on a déjà envoyé un email ou non.
- True/False : oui, un email a déjà été envoyé
- None : on n'a jamais contacté cet email et donc on le met dans la liste des "nouveaux"
On n'a pas cette info au niveau int__union_contacts qui provient exclusivement des sources mes-aides, mediation numérique et dora; il faut donc faire un JOIN mais je voulais éviter de refaire dans un DAG un JOIN complexe qui par ailleurs est déjà fait dans le contacts__enhanced.
Mais bon, j'ai fait une proposition de fix, tu me diras si ça te convient mieux.
vperron
force-pushed
the
vperron/retry-hardbounced
branch
from
5902e9f to
ca6277e
Compare
|
Pas de migration de données à prévoir sur Brevo. |
vmttn
force-pushed
the
vperron/retry-hardbounced
branch
from
ca6277e to
7b39693
Compare
|
lgtm j'ai simplement rebased |
After an entire year of operation, we noticed that our sources never change their contact emails. Ever. Even if someday this happens, we can probably suppose that it's till going to be quite rare. In those exceptional cases, it means that the structure would get a new RGPD notification. But it does make the code way easier to maintain and more robust: - now we will never miss an email that would have been "forgotten" by Brevo (due to rate limits, payment issues, ...) - it can also be seen as a feature to send a new notification if the email changes, for instance when the previous one was incorrect. Moreover, we don't need to maintain the `contact_uid` list that was sent to brevo.
vperron
force-pushed
the
vperron/retry-hardbounced
branch
from
7b39693 to
fd8213c
Compare
After an entire year of operation, we noticed that our sources never change their contact emails. Ever.
Even if someday this happens, we can probably suppose that it's till going to be quite rare.
In those exceptional cases, it means that the structure would get a new RGPD notification.
But it does make the code way easier to maintain and more robust:
Moreover, we don't need to maintain the
contact_uidlist that was sent to brevo.