Get transaction execution status over geyser

[godmodegalactus]

Problem

Currently, we do not have any information on why the transaction was not executed during the banking stage. As we crank the market and send many transactions per block, we would like feedback on why the transaction was not included. We only have feedback if the transaction was executed; the feedback is either Ok or Err. Usually, transactions are dropped because of errors in the scheduling stage, mainly when a writable account or the block hits the CU limit.

As a client, I would like to know if the cluster rejects my transaction and the reason. I would like to benchmark the cluster to understand bottlenecks and adapt my dapp design by following the lifecycle of a transaction. Solana logs are too verbose, and we need access to all the machines in the cluster. So, to get valid logs, we have to run a validator and get a few slots as a leader. Solana metrics can be only used to get an overview of statistics at a particular point in time but not a particular transaction.

I had proposed a solution to tackle this problem with the help of bidirectional quic service, which is a separate service:
#30161
The proposal was not accepted on basis of security (i.e could be attack vector) and too much development. But I was advised to make changes in RPC so that we can get this data over RPC from the leader.

Proposed Solution

We are going to migrate all the RPC services over geyser. I want to propose that this feature could be easily implemented over geyser. There is no need for any separate service; we will miss a few transactions that may be dropped during sigverify stage but would be a tiny portion after the updates to drop_excess packets in sigverify.

The Geyser plugin will have a new channel connected to the banking stage. A message will be sent over the channel if the scheduler cannot schedule the transaction properly.