e2e: cctx status is not Reverted

[gartnera]

After #2483, the upgrade tests have been failing at a high rate. It seems the light upgrade tests fail a bit more, but the regular upgrade tests also fail sometimes:

erc20      | [ERROR]
	Error Trace:     /go/delivery/zeta-node/e2e/utils/require.go:22
                                 /go/delivery/zeta-node/e2e/e2etests/test_erc20_deposit_refund.go:85
	Error:      	Not equal:
	            	expected: 5
	            	actual  : 6
	Messages:   	cctx status is not "Reverted" cctx index 0x7850b6987192a8e7a287405cf374ee4a598d0d20d51c8214fa16c526bc89c35a

erc20      | [ERROR]Test failed

I was able to to run the light upgrade tests on the prior commit 6 times without failure.

Running into some other errors too:

bitcoin    | ⏳running - testing Bitcoin ERC20 cross-chain swap
erc20      | ✅ completed in 1m36.259911982s - withdraw ERC20 from ZEVM
erc20      | ⏳running - withdraw ERC20 from ZEVM in multiple withdrawals
zeta       | ✅ completed in 1m41.307115252s - withdraw ZETA from ZEVM to Ethereum
zeta       | ⏳running - evm->evm message passing (sending ZETA only)
ether      | ✅ completed in 1m56.343918893s - withdraw Ether from ZEVM
ether      | ⏳running - tests sending ETH on ZEVM and check context data using ContextApp
ether      | ✅ completed in 9.079634056s - tests sending ETH on ZEVM and check context data using ContextApp
ether      | ⏳running - deposit ZRC20 into ZEVM and call a contract
bitcoin    | [ERROR]
	Error Trace:	/go/delivery/zeta-node/e2e/utils/require.go:22
	            				/go/delivery/zeta-node/e2e/e2etests/test_crosschain_swap.go:123
	Error:      	Not equal:
	            	expected: 3
	            	actual  : 5
	Messages:   	cctx status is not "OutboundMined" cctx index 0xc8d986cf36fa336619b08f475a05a36eac8aad3b300dc4e532cdc2196648adb2

bitcoin    | [ERROR]Test failed

[lumtis]

The priority fee is not used yet so there shouldn't be any logic influencing the cctx workflow. Having some look into it.

[lumtis]

There are panic in runTx

12:17PM ERR panic recovered in runTx err="recovered: runtime error: index out of range [0] with length 0\nstack:\ngoroutine 386 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x64\ngithub.com/cosmos/cosmos-sdk/baseapp.NewBaseApp.newDefaultRecoveryMiddleware.func3({0x2f28340, 0x4003d38690})\n\t/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:71 +0x24\ngithub.com/cosmos/cosmos-sdk/baseapp.NewBaseApp.newDefaultRecoveryMiddleware.newRecoveryMiddleware.func5({0x2f28340?, 0x4003d38690?})\n\t/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:39 +0x38\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x2f28340, 0x4003d38690}, 0x400349a978?)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:28 +0x38\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x2f28340, 0x4003d38690}, 0x60?)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:33 +0x60\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx.func1()\n\t/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/baseapp.go:646 +0xe0\npanic({0x2f28340?, 0x4003d38690?})\n\t/usr/local/go/src/runtime/panic.go:770 +0x124\ngithub.com/zeta-chain/zetacore/x/crosschain/keeper.msgServer.VoteGasPrice({{{0x43e9620, 0x40013fb6b0}, {0x4390990, 0x4001cc7b30}, {0x4390990, 0x0}, {0x437f9a0, 0x4000edb860}, {0x437f9c0, 0x4001906af0}, ...}}, ...)\n\t/go/delivery/zeta-node/x/crosschain/keeper/msg_server_vote_gas_price.go:60 +0x65c\ngithub.com/zeta-chain/zetacore/x/crosschain/types._Msg_VoteGasPrice_Handler.func1({0x43b71c0?, 0x4003d0c060?}, {0x309a8c0?, 0x400381bc20?})\n\t/go/delivery/zeta-node/x/crosschain/types/tx.pb.go:1837 +0xd0\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2.1({0x43b6cb8, 0x40023ec008}, {0x400349be58?, 0x121fdf8?}, 0x2a0?, 0x4003732ea0)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/msg_service_router.go:118 +0x98\ngithub.com/zeta-chain/zetacore/x/crosschain/types._Msg_VoteGasPrice_Handler({0x3133620, 0x40019c0370}, {0x43b6cb8, 0x40023ec008}, 0x3c87118, 0x4003319420)\n\t/go/delivery/zeta-node/x/crosschain/types/tx.pb.go:1839 +0x148\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2({{0x43b6c10, 0x62901e0}, {0x43d30d0, 0x40023b27c0}, {{0xb, 0x0}, {0x4002897894, 0xc}, 0x10d, {0x3028ac58, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/msg_service_router.go:139 +0x238\ngithub.com/cosmos/cosmos-sdk/x/authz/keeper.Keeper.DispatchActions({{_, _}, {_, _}, _, {_, _}}, {{0x43b6c10, 0x62901e0}, {0x43d30d0, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/authz/keeper/keeper.go:140 +0x3f8\ngithub.com/cosmos/cosmos-sdk/x/authz/keeper.Keeper.Exec({{0x4390990, 0x4001cc7ae0}, {0x43d3300, 0x40013fb6b0}, 0x4000f94900, {0x439add0, 0x4001906960}}, {0x43b71c0?, 0x4003d0c000?}, 0x400381bbc0)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/authz/keeper/msg_server.go:84 +0x130\ngithub.com/cosmos/cosmos-sdk/x/authz._Msg_Exec_Handler.func1({0x43b71c0?, 0x4003d0c000?}, {0x3071860?, 0x400381bbc0?})\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/authz/tx.pb.go:449 +0xd0\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2.1({0x43b6cb8, 0x4002397608}, {0x400349db78?, 0x121fdf8?}, 0x2a0?, 0x4003732e28)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/msg_service_router.go:118 +0x98\ngithub.com/cosmos/cosmos-sdk/x/authz._Msg_Exec_Handler({0x30c2500, 0x4000b9de80}, {0x43b6cb8, 0x4002397608}, 0x3c87118, 0x40033193a0)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/authz/tx.pb.go:451 +0x148\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2({{0x43b6c10, 0x62901e0}, {0x43d30d0, 0x40023b27c0}, {{0xb, 0x0}, {0x4002897894, 0xc}, 0x10d, {0x3028ac58, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/msg_service_router.go:139 +0x238\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x43b6c10, 0x62901e0}, {0x43d30d0, 0x40023b27c0}, {{0xb, 0x0}, {0x4002897894, 0xc}, 0x10d, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/baseapp.go:806 +0x1e4\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0x40008a25a0, 0x3, {0x40017b0900, 0x175, 0x175})\n\t/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/baseapp.go:749 +0xa6c\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).DeliverTx(0x40008a25a0, {{0x40017b0900?, 0x40034a20a8?, 0xe7c544?}})\n\t/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:413 +0xfc\ngithub.com/cometbft/cometbft/abci/client.(*localClient).DeliverTxAsync(0x4001ec2240, {{0x40017b0900?, 0x0?, 0x0?}})\n\t/go/pkg/mod/github.com/cometbft/[email protected]/abci/client/local_client.go:82 +0xec\ngithub.com/cometbft/cometbft/proxy.(*appConnConsensus).DeliverTxAsync(0x4001d7bb18, {{0x40017b0900?, 0x20?, 0xb?}})\n\t/go/pkg/mod/github.com/cometbft/[email protected]/proxy/app_conn.go:106 +0x134\ngithub.com/cometbft/cometbft/state.execBlockOnProxyApp({0x43b6c80, 0x40011612e0}, {0x43cedc0, 0x4001d7bb18}, 0x400205e780, {0x43d3ee8, 0x400060e540}, 0x1)\n\t/go/pkg/mod/github.com/cometbft/[email protected]/state/execution.go:376 +0x4e0\ngithub.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0x4001301b10, 0x7}}, {0x4001301b20, 0xc}, 0x1, 0x10c, {{0x40035086c0, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cometbft/[email protected]/state/execution.go:197 +0x114\ngithub.com/cometbft/cometbft/consensus.(*State).finalizeCommit(0x4001626008, 0x10d)\n\t/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1710 +0x8b4\ngithub.com/cometbft/cometbft/consensus.(*State).tryFinalizeCommit(0x4001626008, 0x10d)\n\t/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1619 +0x26c\ngithub.com/cometbft/cometbft/consensus.(*State).enterCommit.func1()\n\t/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1554 +0x8c\ngithub.com/cometbft/cometbft/consensus.(*State).enterCommit(0x4001626008, 0x10d, 0x0)\n\t/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1592 +0xac4\ngithub.com/cometbft/cometbft/consensus.(*State).addVote(0x4001626008, 0x40040972c0, {0x0, 0x0})\n\t/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:2223 +0x14c8\ngithub.com/cometbft/cometbft/consensus.(*State).tryAddVote(0x4001626008, 0x40040972c0, {0x0?, 0x507640?})\n\t/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:2012 +0x28\ngithub.com/cometbft/cometbft/consensus.(*State).handleMsg(0x4001626008, {{0x43847a0, 0x4001e38508}, {0x0, 0x0}})\n\t/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:865 +0x300\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine(0x4001626008, 0x0)\n\t/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:792 +0x370\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart in goroutine 316\n\t/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:383 +0xf0\n: panic" module=server

\