- June 01, 2023: Initial Draft (@alexanderbez)
DRAFT
The Cosmos SDK simulator is a tool that allows developers to test the entirety of their application's state machine through the use of pseudo-randomized "operations", which represent transactions. The simulator also provides primitives that ensures there are no non-determinism issues and that the application's state machine can be successfully exported and imported using randomized state.
The simulator has played an absolutely critical role in the development and testing of the Cosmos Hub and all the releases of the Cosmos SDK after the launch of the Cosmos Hub. Since the Hub, the simulator has relatively not changed much, so it's overdue for a revamp.
The current simulator, x/simulation
, acts as a semi-fuzz testing suite that takes
in an integer that represents a seed into a PRNG. The PRNG is used to generate a
sequence of "operations" that are meant to reflect transactions that an application's
state machine can process. Through the use of the PRNG, all aspects of block production
and consumption are randomized. This includes a block's proposer, the validators
who both sign and miss the block, along with the transaction operations themselves.
Each Cosmos SDK module defines a set of simulation operations that attempt to
produce valid transactions, e.g. x/distribution/simulation/operations.go
. These
operations can sometimes fail depending on the accumulated state of the application
within that simulation run. The simulator will continue to generate operations
until it has reached a certain number of operations or until it has reached a
fatal state, reporting results. This gives the ability for application developers
to reliably execute full range application simulation and fuzz testing against
their application.
However, there are a few major drawbacks. Namely, with the advent of ABCI++, specifically
FinalizeBlock
, the internal workings of the simulator no longer comply with how
an application would actually perform. Specifically, operations are executed
after FinalizeBlock
, whereas they should be executed within FinalizeBlock
.
Additionally, the simulator is not very extensible. Developers should be able to easily define and extend the following:
- Consistency or validity predicates (what are known as invariants today)
- Property tests of state before and after a block is simulated
In addition, we also want to achieve the following:
- Consolidated weight management, i.e. define weights within the simulator itself via a config and not defined in each module
- Observability of the simulator's execution, i.e. have easy to understand output/logs with the ability to pipe those logs into some external sink
- Smart replay, i.e. the ability to not only rerun a simulation from a seed, but also the ability to replay from an arbitrary breakpoint
- Run a simulation based off of real network state
Instead of refactoring the existing simulator, x/simulation
, we propose to create
a new package in the root of the Cosmos SDK, simulator
, that will be the new
simulation framework. The simulator will more accurately reflect the complete
lifecycle of an ABCI application.
Specifically, we propose a similar implementation and use of a simulator.Manager
,
that exists today, that is responsible for managing the execution of a simulation.
The manager will wrap an ABCI application and will be responsible for the following:
- Populating the application's mempool with a set of pseudo-random transactions before each block, some of which may contain invalid messages.
- Selecting transactions and a random proposer to execute
PrepareProposal
. - Executing
ProcessProposal
,FinalizeBlock
andCommit
. - Executing a set of validity predicates before and after each block.
- Maintaining a CPU and RAM profile of the simulation execution.
- Allowing a simulation to stop and resume from a given block height.
- Simulation liveness of each validator per-block.
From an application developer's perspective, they will only need to provide the modules to be used in the simulator and the manager will take care of the rest. In addition, they will not need to write their own simulation test(s), e.g. non-determinism, multi-seed, etc..., as the manager will provide these as well.
type Manager struct {
app sdk.Application
mempool sdk.Mempool
rng rand.Rand
// ...
}
The simulator's testing input will be driven by a configuration file, as opposed to CLI arguments. This will allow for more extensibility and ease of use along with the ability to have shared configuration files across multiple simulations.
As alluded to previously, after the execution of each block, the manager will
generate a series of pseudo-random transactions and attempt to insert them into
the mempool via BaseApp#CheckTx
. During the ABCI lifecycle of a block, this
mempool will be used to seed the transactions into a block proposal as it would
in a real network. This allows us to not only test the state machine, but also
test the ABCI lifecycle of a block.
Statistics, such as total blocks and total failed proposals, will be collected, logged and written to output after the full or partial execution of a simulation. The output destination of these statistics will be configurable.
func (s *Simulator) SimulateBlock() {
rProposer := s.SelectRandomProposer()
rTxs := s.SelectTxs()
prepareResp, err := s.app.PrepareProposal(&abci.RequestPrepareProposal{Txs: rTxs})
// handle error
processResp, err := s.app.ProcessProposal(&abci.RequestProcessProposal{
Txs: prepareResp.Txs,
// ...
})
// handle error
// execute liveness matrix...
_, err = s.app.FinalizeBlock(...)
// handle error
_, err = s.app.Commit(...)
// handle error
}
Note, some applications do not define or need their own app-side mempool, so we
propose that SelectTxs
mimic CometBFT and just return FIFO-ordered transactions
from an ad-hoc simulator mempool. In the case where an application does define
its own mempool, it will simply ignore what is provided in RequestPrepareProposal
.
The manager will be responsible for collecting CPU and RAM profiles of the simulation execution. We propose to use Pyroscope to capture profiles and export them to a local file and via an HTTP endpoint. This can be disabled or enabled by configuration.
Via configuration, a caller can express a height-based breakpoint that will allow the simulation to stop and resume from a given height. This will allow for debugging of CPU, RAM, and state.
We propose to provide the ability for an application to provide the simulator a set of validity predicates, i.e. invariant checkers, that will be executed before and after each block. This will allow for the application to assert that certain state invariants are held before and after each block. Note, as a consequence of this, we propose to remove the existing notion of invariants from module production execution paths and deprecate their usage altogether.
type Manager struct {
// ...
preBlockValidator func(sdk.Context) error
postBlockValidator func(sdk.Context) error
}
The new simulator package will not naturally not be backwards compatible with the
existing x/simulation
module. However, modules will still be responsible for
providing pseudo-random transactions to the simulator.
- Providing more intuitive and cleaner APIs for application developers
- More closely resembling the true lifecycle of an ABCI application
- Breaking current Cosmos SDK module APIs for transaction generation