[Bug] Potential temporary sync stall from weak-link or malicious peers

[Meshiest]

🐛 Bug Report

A circumstance where block requests are forced to timeout can be reached when a peer with a pending request does not respond with a block response. This is similar to the temporary stall outlined in #3321.

Steps to Reproduce

1. prevent a block response from being created from the validator or the client's router
2. observe syncing nodes using the modified nodes as peers get stuck for ~10 minutes when reaching a block with an "incomplete request"

We encountered a 10 minute stall during a sync of a node on high performance hardware after merging the latest client sync fixes and deduced it may be related to a swarm of low-quality clients as connected peers (from another test). We were able to resume block sync immediately by forcibly disconnecting the weak links.

The 34.16.96.117 IP is one of 20 16core servers under extreme stress from 10 clients syncing blocks with dozens of transactions per block and as a result is unreliably able to respond to block requests, preventing a block request from being marked as complete.

Expected Behavior

Blocks to sync without stalls when a peer unable to respond.

Potential Fixes

• timeout for block requests on a peer-node basis
• disconnect from peers that are unreliable
• allow for a certain # of peers to be unresponsive when marking a request as complete in block_sync.rs
• shorter total block request timeouts

Your Environment

• snarkOS version: mainnet-staging on canary + rwlock (fix(sync): Fix client sync RwLock deadlock and client block request stall #3321) + node ip fixes (fix(tcp): fix outbound tcp requests not respecting --node bind port #3318)
• Rust version: 1.76
• OS: Ubuntu 24.04

[elderhammer]

#3320
As I observed a few days ago, I thought this issue can be ignored because the team did not comment on it. Now I reopened this issue

[raychu86]

Should be addressed with #3422. (This PR bans peers who frequently reach timeouts)