Warp sync crash (final_outcome.encoded_size() <= MAX_WARP_SYNC_PROOF_SIZE)

[RomarQ]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

A user reported the following crash on the moonbeam repository, it happened when using warp sync and the service was running for a month before crashing.

polkadot-sdk/substrate/client/consensus/grandpa/src/warp_proof.rs

Lines 183 to 185 in 53f6473

	let final_outcome = WarpSyncProof { proofs, is_finished };
	debug_assert!(final_outcome.encoded_size() <= MAX_WARP_SYNC_PROOF_SIZE);
	Ok(final_outcome)

Important to note that the panic didn't happen again since the user restarted the node.

Dec 09 22:31:48  moonbeam[1810]: 2024-12-09 22:31:48 [🌗] 🔮 Skipping candidate production because we are not eligible for slot 23775533
Dec 09 22:31:48  moonbeam[1810]: 2024-12-09 22:31:48 [🌗] Pinned block cache limit reached. Evicting value. hash = 0x8c73…9940
Dec 09 22:31:48  moonbeam[1810]: 2024-12-09 22:31:48 [🌗] 🆕 Imported #8684548 (0xd692…d374 → 0xf99e…cfd0)
Dec 09 22:31:49  moonbeam[1810]: ====================
Dec 09 22:31:49  moonbeam[1810]: Version: 0.41.1-eb89f949d5a
Dec 09 22:31:49  moonbeam[1810]:    0: sp_panic_handler::set::{{closure}}
Dec 09 22:31:49  moonbeam[1810]:    1: std::panicking::rust_panic_with_hook
Dec 09 22:31:49  moonbeam[1810]:    2: std::panicking::begin_panic_handler::{{closure}}
Dec 09 22:31:49  moonbeam[1810]:    3: std::sys_common::backtrace::__rust_end_short_backtrace
Dec 09 22:31:49  moonbeam[1810]:    4: rust_begin_unwind
Dec 09 22:31:49  moonbeam[1810]:    5: core::panicking::panic_fmt
Dec 09 22:31:49  moonbeam[1810]:    6: core::panicking::panic
Dec 09 22:31:49  moonbeam[1810]:    7: <sc_consensus_grandpa::warp_proof::NetworkProvider<Block,Backend> as sc_network_sync::strategy::warp::WarpSyncProvider<Block>>::generate
Dec 09 22:31:49  moonbeam[1810]:    8: sc_network_sync::warp_request_handler::RequestHandler<TBlock>::run::{{closure}}
Dec 09 22:31:49  moonbeam[1810]:    9: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
Dec 09 22:31:49  moonbeam[1810]:   10: tokio::runtime::task::raw::poll
Dec 09 22:31:49  moonbeam[1810]:   11: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
Dec 09 22:31:49  moonbeam[1810]:   12: tokio::runtime::scheduler::multi_thread::worker::run
Dec 09 22:31:49  moonbeam[1810]:   13: tokio::runtime::task::raw::poll
Dec 09 22:31:49  moonbeam[1810]:   14: std::sys_common::backtrace::__rust_begin_short_backtrace
Dec 09 22:31:49  moonbeam[1810]:   15: core::ops::function::FnOnce::call_once{{vtable.shim}}
Dec 09 22:31:49  moonbeam[1810]:   16: std::sys::pal::unix::thread::Thread::new::thread_start
Dec 09 22:31:49  moonbeam[1810]:   17: <unknown>
Dec 09 22:31:49  moonbeam[1810]:   18: <unknown>
Dec 09 22:31:49  moonbeam[1810]: Thread 'tokio-runtime-worker' panicked at 'assertion failed: final_outcome.encoded_size() <= MAX_WARP_SYNC_PROOF_SIZE', /root/.cargo/git/checkouts/polkadot-sdk-38b703c7469a7d1e/86b704d/substrate/client/consensus/grandpa/src/warp_proof.rs:184
...
Dec 09 22:31:53  systemd[1]: moonbeam.service: Main process exited, code=exited, status=1/FAILURE
Dec 09 22:31:53  systemd[1]: moonbeam.service: Failed with result 'exit-code'.
Dec 09 22:31:53  systemd[1]: moonbeam.service: Consumed 1month 4d 4h 24min 58.389s CPU time.
Dec 09 22:31:58  systemd[1]: moonbeam.service: Scheduled restart job, restart counter is at 1.
Dec 09 22:31:58  systemd[1]: Stopped moonbeam service.
Dec 09 22:31:58  systemd[1]: moonbeam.service: Consumed 1month 4d 4h 24min 58.389s CPU time.

Steps to reproduce

Does not seem easy to reproduce.

[bkchr]

@RomarQ ty for reporting, why is the operator running a debug build? :D

[RomarQ]

Thanks for having a look at it.

We currently have debug-assertions enabled in the production profile. I was going to disable it, but then noticed that build-linux-stable workflow in polkadot-sdk also enables it and decided to leave it enabled.

Will probably disable it for production.

polkadot-sdk/.github/workflows/build-publish-images.yml

Line 41 in 91bef33

RUSTFLAGS: "-Cdebug-assertions=y -Dwarnings"

[bkchr]

polkadot-sdk/.github/workflows/build-publish-images.yml

Line 39 in 91bef33

# Enable debug assertions since we are running optimized builds for testing

These images are only used for testing and not for production.