Arrangement backfill fails recovery test, get_row could not find vnode=0

[kwannoel]

Branch to reproduce: #14931 (please fork the branch if you want to do any debugging, to make sure we always have a reproducible case).

See #14888 (comment).

[kwannoel]

Here the root cause:

Currently we will deregister the table or mv anyway before sending a barrier to stop the actors, and then the metadata update will be broadcast to CN and cause inconsistency before the actors get dropped.

And the solution:

correct. I think we can try to send and collect the barrier, and then deregister the table after the stop actor barrier is successfully collected or a recovery is triggered.

[kwannoel]

Seems like it can happen when barrier latency is high, and we drop the sink.

It occurs in arrangement backfill recovery test, when we create a sink and drop it subsequently when there's high barrier latency.

See e2e_test/streaming/rate_limit/snapshot_amplification.slt

[kwannoel]

https://risingwave-labs.slack.com/archives/C05AZUR6Q5P/p1707201627019669?thread_ts=1706762317.774449&cid=C05AZUR6Q5P slack disccussion

[kwannoel]

This is edge case which can happen when drop sink under high barrier. We can fix it in next release.