sidechain paseo stopped producing blocks for several hours due to heavy load on shielding target

[brenzi]

2024-11-15 03:18:59.298 [Sidechain] imported block (number: 1474759, tcalls: 0, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 1538453
2024-11-15 03:21:56.994 [Sidechain] imported block (number: 1474760, tcalls: 1, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 1538453

gap 

2024-11-15 05:29:32.488 [Sidechain] imported block (number: 1474761, tcalls: 415, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 1538580
2024-11-15 05:37:47.445 [Sidechain] imported block (number: 1474762, tcalls: 1575, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 1539091
2024-11-15 05:37:50.709 [Sidechain] imported block (number: 1474763, tcalls: 10, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 1539094
2024-11-15 05:37:51.322 [Sidechain] imported block (number: 1474764, tcalls: 0, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 1539094
2024-11-15 05:37:52.641 [Sidechain] imported block (number: 1474765, tcalls: 2, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 1539095	

no suspicious errors or warnings, no service panic/restart

tons of transfers happening on paseo relay until 04:03:
2024-11-15 03:22:05.051
[L1Event:TargetA] BalanceTransfer { from: 9c4381956c49a32a2a04d0a9a632feee74aff92decff2b072471078ccd551103 (5FbbP2vs...), to: d8661378f4316e0738afbf332840c8aeda8e3b7efc9ae50f193048aa4450fe50 (5GxSXJEe...), amount: 195824851 }

Seems they were stress-testing and the sidechain could't keep up with syncing blocks

TargetA block import was struggling:

[clangenb]

Interesting, this seems to be the expected behavior:

• The author will import the parentchain block
• Will realize that we are no longer within the slot, so it will not execute any trusted calls
• Will still propose the block, which would lead to a fork in a multi-validateer setup, but the single one, was just fine:

  worker/sidechain/consensus/slots/src/lib.rs

  Line 364 in 8c76062

  if !timestamp_within_slot(&slot_info, &proposing.block) {

EDIT: I don't know why target a block import was struggling

[brenzi]

I think for now we can accept this behavior, because:

• such loads are (currently) not normal but artificial
• our UI would have paused all actions due to bad system health (shielding target sync lagging)