feat: outbound cctx scheduling with rate limiter

[ws4charlie]

Description

• Added CctxListPendingWithinRateLimit new gRPC query for zetaclient to use for rate limiting purpose.
• Adjusted zetaclient code to switch to the new gRPC query.

Closes: 2040

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Include instructions and any relevant details so others can reproduce.

• Tested CCTX in localnet
• Tested in development environment
• Go unit tests
• Go integration tests
• Tested via GitHub Actions

Checklist:

• I have added unit tests that prove my fix feature works

[github-actions]

!!!WARNING!!!
nosec detected in the following files: x/crosschain/keeper/grpc_query_cctx_rate_limit.go, x/crosschain/keeper/grpc_query_cctx_rate_limit_test.go, x/crosschain/client/cli/cli_cctx.go, x/crosschain/keeper/grpc_query_cctx.go, zetaclient/zetacore_observer.go

Be very careful about using #nosec in code. It can be a quick way to suppress security warnings and move forward with development, it should be employed with caution. Suppressing warnings with #nosec can hide potentially serious vulnerabilities. Only use #nosec when you're absolutely certain that the security issue is either a false positive or has been mitigated in another way.

Only suppress a single rule (or a specific set of rules) within a section of code, while continuing to scan for other problems. To do this, you can list the rule(s) to be suppressed within the #nosec annotation, e.g: /* #nosec G401 */ or //#nosec G201 G202 G203
Broad #nosec annotations should be avoided, as they can hide other vulnerabilities. The CI will block you from merging this PR until you remove #nosec annotations that do not target specific rules.

Pay extra attention to the way #nosec is being used in the files listed above.

[lumtis]

Just wondering, do we stop listing to pending cctxs outside of the window boundaries? What happen if a cctx is left behind the left boundary?

[codecov]

Codecov Report

Attention: Patch coverage is 31.85841% with 154 lines in your changes are missing coverage. Please review.

Project coverage is 70.70%. Comparing base (fb05f0d) to head (60e5dcc).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2045      +/-   ##
===========================================
- Coverage    71.32%   70.70%   -0.62%     
===========================================
  Files          251      252       +1     
  Lines        13868    14052     +184     
===========================================
+ Hits          9891     9936      +45     
- Misses        3517     3656     +139     
  Partials       460      460              

Files	Coverage Δ
x/crosschain/keeper/abci.go	80.76% <100.00%> (ø)
x/crosschain/keeper/cctx_utils.go	100.00% <100.00%> (ø)
x/crosschain/keeper/foreign_coins.go	100.00% <100.00%> (ø)
...osschain/keeper/msg_server_add_to_outtx_tracker.go	88.88% <100.00%> (ø)
zetaclient/evm/evm_signer.go	47.58% <0.00%> (ø)
x/crosschain/keeper/rate_limiter_flags.go	92.85% <88.23%> (-7.15%)	⬇️
x/crosschain/keeper/grpc_query_cctx.go	91.01% <84.61%> (+4.05%)	⬆️
x/observer/keeper/chain_params.go	84.78% <0.00%> (-15.22%)	⬇️
x/fungible/keeper/foreign_coins.go	86.15% <0.00%> (-13.85%)	⬇️
zetaclient/zetabridge/query.go	66.33% <31.25%> (-2.05%)	⬇️
... and 2 more

[brewmaster012]

Went over the main logic of the new RPC call that lists pending cctx with rate limiter, the logic looks good to me. Have not checked the arithmetic in the conversion and accumulation.

[brewmaster012]

Would be nice to be able to test against mainnet state in some way. Since this is read only RPC, maybe try a slightly mocked RPC and connect to mainnet node and do the query as sanity check.

[ws4charlie]

Just wondering, do we stop listing to pending cctxs outside of the window boundaries? What happen if a cctx is left behind the left boundary?

When looping backwards, the query picks up pending cctx in the range [nonceLow-1000, nonceLow]. If the cctx on nonceLow hasn't hit the left boundary (e.g,, current height - 2hours_of_blocks), the backwards loop will continue until it hits the left boundary in order to calculate the total withdrawn amount during the rate limiter window.