Automated retries

[ekzyis]

Description

close #1492 based on #1785, #1787

All failed invoices are returned only once to exactly one client via a new query failedInvoices. To avoid an invoice being returned but never retried by a client for any reason, a visibility timeout as described in #1492 (comment) is used: for each returned invoice, we set a new Invoice.lockedAt column to the current time and then unset it when we can assume that the client did not and will not attempt a retry of this invoice.

To not retry invoices that the user manually canceled (closing the QR code), a new boolean column Invoice.userCancel was added. This is only set if the invoice was canceled via closing the QR code.

TODO:

• retry returned failed invoices on client
• count retries so we can stop returning them after we hit a limit
• show failed invoices that have exhausted retries in notifications
• don't show failed invoices that haven't exhausted retries in notifications
• don't retry invoices that have been manually cancelled?

added userCancel column, see #1785

• don't return failed invoices to clients that don't have send wallets

client only polls when it has send wallets

• don't return intermediate failed invoices that will be retried due to sender or receiver fallbacks

added "cancelledAt" < now() - interval '${WALLET_RETRY_BACKOFF_MS} milliseconds' filter

• ⚠️ make sure when this is deployed, this will not start retrying old failed invoices ⚠️

Screenshots

Additional Context

tbd

Checklist

Are your changes backwards compatible? Please answer below:

tbd

On a scale of 1-10 how well and how have you QA'd this change and any features it might affect? Please answer below:

tbd

For frontend changes: Tested on mobile, light and dark mode? Please answer below:

tbd

Did you introduce any new environment variables? If so, call them out explicitly here:

tbd