Real-time Sync: Add persistency layer

[hydra-yse]

Closes #501.

[dangeross]

Sorry for the late review

[dangeross]

ACK, needs cargo fmt

[dangeross]

I see that when restoring the app (data is already pushed to sync server), the SDK pulls in the swap data and because it's in initially the Created state these swaps are then tracked in the swap update loop (see attached logs). Should we consider adding a state to mark that the swap needs recovering without adding it to the update loop?

1733127914391.app.log

[roeierez]

We have is_local on the swaps. The main question is do we need to exclude them totally from the update loop (and only count on the onchain data) or should we add them to the update loop but without initiating transactions (claim). What do you think?

[hydra-yse]

Currently they are considered in the update loop (as Roei said using is_local from the sync_state table) so state updates are pushed outwards (and merged if necessary), then with either Boltz status/restore they're set to the right final state. May be unnecessary to push outwards and let each instance settle its own final state.

Regarding claiming, I think there was discussion about only letting the initiator broadcast. I think it's the only non-racy way to do it (since the claim checks I've added assume the chain service already has the other instance's tx indexed, which may or may not be true in some edge cases), so if this mitigation is not enough I agree we should just stick to the easy approach.

Thinking of the frequency at which this may happen, the scenario would be if the sync-out and sync-in between clients happens very quickly, so that they both receive the Boltz lockup status update at the same time and try claiming simultaneously. Not that uncommon if you ask me, so I'm inclined to the easier approach too.

Another idea could also be to maybe track the claims for a few confirmations to see whether the claim info actually matches, that way if two instances claim we can still settle on which one got confirmed (via script history), but I think that may just be unnecessary overhead, let me know what you think.

[dangeross]

I'm wondering if we first set them to a Recover state, then after the recovery/chain sync if the swap is ongoing then add it the the update loop for tracking.

[roeierez]

I think we should start simple means:

1. Not claiming if it is not a local swap
2. We can still track all swaps in the swapper web socket loop
3. We can let the recover run as currently in its own interval. @dangeross comment is valid IMO and it would be nice to run the swaps retrieved from the sync through the recovery process before persisting them.

[hydra-yse]

Best way I see of going about that is to persist them instantly anyway, but as Ross mentioned with a Recovered/new state that allows them to be tracked explicitly. It's also easier to debug what's happening that way

[dangeross]

From testing I think Recover state is helpful. The SDK is behaving differently depending if the rt-sync pull is completed before we start to track swaps. The Recover state will also allow us to start tracking the swap if needed.

I also see an issue sometimes with the initial chain sync from a fresh state, if the rt-sync pulls after the first chain sync, where the SDK events are suppressed, in the next chain sync all the pulled rt-sync swaps events are emitted.

[roeierez]

Good points @hydra-yse as Ross mentioned let's add the recovery to the sync in pipeline just before persisting the swaps. Results will be more predictable this way and we will avoid weird behaviors.

[hydra-yse]

Done in 79b3d8d