diff --git a/README.md b/README.md index 9d39c527d3..93a458c870 100644 --- a/README.md +++ b/README.md @@ -1,379 +1,46 @@ -# hive - Ethereum end-to-end test harness +# hive - Ethereum end-to-end test harness -[![API Reference]( -https://camo.githubusercontent.com/915b7be44ada53c290eb157634330494ebe3e30a/68747470733a2f2f676f646f632e6f72672f6769746875622e636f6d2f676f6c616e672f6764646f3f7374617475732e737667 -)](https://pkg.go.dev/github.com/ethereum/hive?tab=doc) +Hive is a system for running integration tests against Ethereum clients. -hive is a testing rig designed to make it easy to run specification conformance and behavior tests against any eth1 client implementation. +Ethereum Foundation maintains a public Hive instance to check for consensus, p2p and +blockchain compatibility. You can find the latest test results [here][hive-prod]. -# Public test results +**To read more about hive, please check [the documentation][doc].** -An Ethereum Foundation server often runs Hive to check for consensus, p2p and blockchain compatibility as part of our CI workflow. This is needed to ensure a baseline quality for eth1 client implementations. +### Trophies -Test results are made public [here](https://hivetests.ethdevops.io/). +If you find a bug in your client implementation due to this project, please be so kind as +to add it here to the trophy list. It could help prove that `hive` is indeed a useful tool +for validating Ethereum client implementations. -# Running hive +- go-ethereum: + - Genesis chain config couldn't handle present but empty settings: [#2790](https://github.com/ethereum/go-ethereum/pull/2790) + - Data race between remote block import and local block mining: [#2793](https://github.com/ethereum/go-ethereum/pull/2793) + - Downloader didn't penalize incompatible forks hashly enough: [#2801](https://github.com/ethereum/go-ethereum/pull/2801) +- Nethermind: + - Bug in p2p whith bonding nodes algorithm found by Hive: [#1894](https://github.com/NethermindEth/nethermind/pull/1894) + - Difference in return value for 'r' parameter in getTransactionByHash: [#2372](https://github.com/NethermindEth/nethermind/issues/2372) -The `hive` project is based on Go. You'll need a valid Go (1.6 and upwards) installation available. +### Contributions -After cloning the repository, build the hive executable by running `go build` inside the root directory. +This project takes a different approach to code contributions than your usual FOSS project +with well ingrained maintainers and relatively few external contributors. It is an +experiment. Whether it will work out or not is for the future to decide. -Then, to run a simulation, use the following command: -```bash -./hive --sim --client --loglevel -``` -For example, if you want to run the `discv4` test, here is how the command would look: - -```bash -./hive --sim devp2p/discv4 --client go-ethereum_latest --loglevel 6 -``` - -## Quickstart command lines - -This section is a quick start guide for command line options covering typical hive usage scenarios. - -### Consensus test -The following command will run consensus tests on the parity client -```text - --sim consensus - --client parity_latest - --results-root /mytests/tests -``` - -### Devp2p tests - -The following command will run devp2p tests on nethermind: - -```text - --sim devp2p - --client nethermind - --loglevel 6 - --results-root /mytests/tests - --sim.parallelism 1 -``` -The `--sim.parallelism` flag will set the maximum number of clients against which to run the simulation in parallel. - -### Sync simulation - -The following command will run a test verifying that a blockchain can be synced between differing implementations (in this case, parity and geth): - - --sim sync - --client go-ethereum_latest,parity_latest - --loglevel 6 - --results-root /mytests/tests - -### Iterating on bug fixes locally - -If you are testing locally and want to make changes to the simulation or client, run hive with the flag `--docker.nocache ` so that hive rebuilds the container from scratch. - -If you want to rebuild both, separate the names with a `,` as such: -```text ---docker.nocache devp2p,go-ethereum_latest -``` - -# Adding a Simulation - -A hive simulation is any spec conformance or behavior test that is designed to be executed against an ethereum client implementation using the simulation API provided by hive. - -A simulation can be written in any language as long as it is properly dockerized and makes use of the simulation API. - -## Simulation API - -The simulation API provided by hive is a simple gateway to communicating with the client container(s). - -There are a couple of components that are important to a hive simulation: -* test suites -* test cases -* clients -* networks - -A **test suite** is a single run of a simulator. It can contain several **test cases**, which are individual tests against one or more **clients**. - -**Networks** are also useful if your test(s) require a more complex network topology. - -### Test Suite Endpoints -**Create a test suite** -``` -POST /testsuite -``` -Response -``` -1 -``` -**Delete a test suite** -``` -DELETE /testsuite/{suite} -``` -### Test Case Endpoints -**Create a test case** -``` -POST /testsuite/{suite}/test -``` -Response -``` -2 -``` -**Delete a test case** - -_Note:_ we use POST as the HTTP method for this request as DELETE does not always support a message body. Hive expects a `summaryresult` in the body / URL form of this request. -``` -POST /testsuite/{suite}/test/{test} -``` - -### Client endpoints - -**Get client types** - -``` -GET /clients -``` - -Response -``` -["go-ethereum_latest"] -``` - -**Start client** - -This request requires several form values in the body, such as parameters and files for configuring the client. One parameter must be named `CLIENT` and should contain one of the client types from the `/clients` endpoint. The parameters are used as environment variables in the new container. - -``` -POST /testsuite/{suite}/test/{test}/node -``` -Response -``` -["@@"] -``` - -**Get client enode URL** -``` -GET /testsuite/{suite}/test/{test}/node/{node} -``` -Response -``` -enode://1ba850b467b3b96eacdcb6c133d2c7907878794dbdfc114269c7f240d278594439f79975f87e43c45152072c9bd68f9311eb15fd37f1fd438812240e82de9ef9@172.17.0.3:30303 -``` - -**Stop client** -``` -DELETE /testsuite/{suite}/test/{test}/node/{node} -``` - -### Network endpoints - -**Create a network** -This endpoint will create a docker network with the given name -``` -POST /testsuite/{suite}/network/{network} -``` -Response -``` -"success" -``` -**Remove a network** - -_Note: this request will fail if containers are still connected to the network._ -``` -DELETE /testsuite/{suite}/network/{network} -``` -Response -``` -"success" -``` -**Connect a container to a network** -``` -POST /testsuite/{suite}/network/{network}/{node} -``` - -**Get a container's IP address on a network** - -``` -GET /testsuite/{suite}/network/{network}/{node} -``` -Response -``` -172.22.0.2 -``` - -**Disconnect a container from a network** -``` -DELETE /testsuite/{suite}/network/{network}/{node} -``` - -## Placement -_(This section is relevant if you plan to merge your simulation upstream)_ - -If the theme of the test suite can be grouped in one of the directories located in `simulators/`, please place the new simulation in that directory. Otherwise, if the simulation cannot be categorized with the current groupings, create a new directory in `simulators/` and name it according to the theme of the test suite. - -## Dockerizing the Simulation -Create a Dockerfile and place it in the same directory as the simulation. The Dockerfile should: - -* build the simulation executable -* build the test tool / executable -* set the entrypoint as the simulation executable (this will ensure that the simulation handles the communication between the hive server and the test itself) - -## Using the `hivesim` simulation API wrapper -While it's possible to communicate with the hive simulation server via the API documented above, hive also provides an easy-to-use API wrapper _(written in Go)_. - -The documentation is provided [here](/hivesim/doc.go). - -# Adding a Client - -## Creating a client image - -Adding a new client implementation to `hive` entails creating a Dockerfile (and related resources), -based on which `hive` will assemble the docker image to use as the blueprint for testing. - -The client definition(s) should reside in the `clients` folder, inside a folder named `` where `` is the official name of the client (lowercase, no fancy characters). `hive` will automatically pick up all clients from this folder. - -There aren't many contraints on the image itself, though a few required caveats exist: - - * It should be as tiny as possible (play nice with others). Preferably use `alpine` Linux. - * It should expose the following ports: 8545 (HTTP RPC), 8546 (WS RPC), 30303 (devp2p). - * It should have a single entrypoint (or script) defined, which can initialize and run the client. - -For guidance, check out the reference [go-ethereum](/clients/go-ethereum/Dockerfile) client. - -## Initializing the client - -hive injects all the required configurations into the Linux containers prior to launching the client's `entrypoint` script. It is then left to this script to interpret all the environmental configs and initialize the client appropriately. - -The chain configurations files: - - * `/genesis.json` contains the JSON specification of the Ethereum genesis states - * `/chain.rlp` contains a batch of RLP encoded blocks to import before startup - * `/blocks/` folder with numbered singleton blocks to import before startup - * `/keys/` contains account keys that should be imported before startup - -Client startup scripts need to ensure that they load the genesis state first, then import a possibly longer blockchain and then import possibly numerous individual blocks. The reason for requiring two different block sources is that specifying a single chain is more optimal, but tests requiring forking chains cannot create a single chain. - -Besides the standardized chain configurations, clients can in general be modified behavior-wise in quite a few ways that are mostly supported by all clients, yet are implemented differently in each. As such, each possible behavioral change required by some simulator is characterized by an environment variable, which clients should interpret as best as they can. - -The behavioral configuration variables: - - * `HIVE_BOOTNODE` enode URL of the discovery-only node to bootstrap the client - * `HIVE_TESTNET` whether clients should run with modified starting nonces (`2^20`) - * `HIVE_NODETYPE` specifying the sync and pruning algos that should be used - * If unset, then uninteresting and run in the node's default mode - * If `archive`, assumes that all historical state is retained after sync - * If `full`, assumes fast sync and consecutive pruning of historical state - * If `light`, assumes header only sync and no state maintenance at all - * `HIVE_FORK_HOMESTEAD` the block number of the Ethereum Homestead transition - * `HIVE_FORK_DAO_BLOCK` the block number of the DAO hard-fork transition - * `HIVE_FORK_DAO_VOTE` whether the node supports or opposes the DAO hard-fork - * `HIVE_FORK_TANGERINE` the block number of the Ethereum TangerineWhistle transition - * The HF for repricing certain opcodes, EIP 150 - * `HIVE_FORK_SPURIOUS` the block number of the Ethereum Homestead transition - * The HF for replay protection, state cleaning etc. EIPs 155,160,161. - * `HIVE_FORK_METROPOLIS` the block number of the Metropolis hardfork - * `HIVE_MINER` address to credit with mining rewards (if set, start mining) - * `HIVE_MINER_EXTRA` extra-data field to set for newly minted blocks - -The client has the responsibility of mapping the hive environment variables to its own command line flags. To assist in this, Hive illustrates a technique in the `clients/go-ethereum` folder using `mapper.jq`, which is invoked in `geth.sh` This technique can be replicated for other clients. - -## Enode script - -For devp2p tests or other simulations that require to know the specific enode URL of the client instance, the client must provide an `enode.sh` that echoes the enode of the running instance. This is executed by the Hive host remotely in order to retrieve the enode URL. - -## Starting the client - -After initializing the client blockchain (genesis, chain, blocks), the last task of the entry script is to start up the client itself. The following defaults are required by `hive` to enable automatic network assembly and firewall enforcement: - - * Clients should open their HTTP-RPC endpoint on `0.0.0.0:8545` (mandatory) - * Clients should open their WS-RPC endpoint on `0.0.0.0:8546` (optional) - * Clients should open their IPC-RPC endpoints at `/rpc.ipc` (optional) - -There is no need to handle graceful client termination. Clients will be forcefully aborted upon test suite completion and all related data purged. A new instance will be started for every test. - -### Smoke testing new clients - -To quickly check if a client adheres to the requirements of `hive`, there is a suite of smoke test simulations that just initialize clients with some pre-configured states and queries it from the various RPC endpoints. - -``` -$ hive --client=go-ethereum_latest --sim smoke -... -Simulation results: -{ - "go-ethereum:latest": { - "smoke/lifecycle": { - "start": "2017-01-31T09:20:16.975219924Z", - "end": "2017-01-31T09:20:18.705302536Z", - "success": true - } - } -} -``` - -*Note: All smoke tests must pass for a client to be included into `hive`.* - -# Generating test blockchains with the `hivechain` tool - -hive's `hivechain` tool allows you to create RLP-encoded blockchains for inclusion into simulations. - -## Generating a blockchain - -Build the `hivechain` tool, located in the `cmd/` directory. - -Then, to generate a chain of a desired length, run the following command: - -```bash -hivechain generate -genesis -length -``` -The `hivechain` tool will generate blocks with transactions as well if the following accounts are present and have a balance in the genesis block: - -```text -"0x71562b71999873DB5b286dF957af199Ec94617F7" -"0x703c4b2bD70c169f5717101CaeE543299Fc946C7" -"0x0D3ab14BBaD3D99F4203bd7a11aCB94882050E7e" -``` - -### Additional options: - -```text - -blocktime int - The desired block time in seconds (default 30) - -genesis string - The path and filename to the source genesis.json - -length int - The length of the chain to generate (default 2) - -mine - Enables ethash mining - -output string - Chain destination folder (default ".") - -tx-interval int - Add transaction to chain every n blocks (default 10) -``` - -# Trophies - -If you find a bug in your client implementation due to this project, please be so -kind as to add it here to the trophy list. It could help prove that `hive` is indeed -a useful tool for validating Ethereum client implementations. - - * go-ethereum - * Genesis chain config couldn't handle present but empty settings: [#2790](https://github.com/ethereum/go-ethereum/pull/2790) - * Data race between remote block import and local block mining: [#2793](https://github.com/ethereum/go-ethereum/pull/2793) - * Downloader didn't penalize incompatible forks hashly enough: [#2801](https://github.com/ethereum/go-ethereum/pull/2801) - * Nethermind - * Bug in p2p whith bonding nodes algorithm found by Hive: [#1894](https://github.com/NethermindEth/nethermind/pull/1894) - -# Contributions - -This project takes a different approach to code contributions than your usual FOSS project with well ingrained maintainers and relatively few external contributors. It is an experiment. Whether it will work out or not is for the future to decide. - -We follow the [Collective Code Construction Contract (C4)](http://rfc.zeromq.org/spec:22/C4/), code -contribution model, as expanded and explained in [The ZeroMQ Process](https://hintjens.gitbooks.io/social-architecture/content/chapter4.html). -The core idea being that any patch that successfully solves an issue (bug/feature) and doesn't break -any existing code/contracts **must** be optimistically merged by maintainers. Followup patches may -be used to for additional polishes – and patches may even be outright reverted if they turn out to +We follow the [Collective Code Construction Contract (C4)][c4], code contribution model, +as expanded and explained in [The ZeroMQ Process][zmq-process]. The core idea being that +any patch that successfully solves an issue (bug/feature) and doesn't break any existing +code/contracts must be optimistically merged by maintainers. Followup patches may be used +to for additional polishes – and patches may even be outright reverted if they turn out to have a negative impact – but no change must be rejected based on personal values. -Please consult the two C4 documents for details: - - * [Collective Code Construction Contract (C4)](http://rfc.zeromq.org/spec:22/C4/) - * [The ZeroMQ Process](https://hintjens.gitbooks.io/social-architecture/content/chapter4.html) +### License -# License +The hive project is licensed under the [GNU General Public License v3.0][gpl]. You can +find it in the COPYING file. -The `hive` project is licensed under the [GNU General Public License v3.0](http://www.gnu.org/licenses/gpl-3.0.en.html), -also included in our repository in the COPYING file. +[hive-prod]: https://hivetests.ethdevops.io/ +[doc]: ./docs/overview.md +[c4]: http://rfc.zeromq.org/spec:22/C4/ +[zmq-process]: https://hintjens.gitbooks.io/social-architecture/content/chapter4.html +[gpl]: http://www.gnu.org/licenses/gpl-3.0.en.html diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000000..fbd7ba8137 --- /dev/null +++ b/docs/README.md @@ -0,0 +1 @@ +Start here: [overview.md](./overview.md). diff --git a/docs/clients.md b/docs/clients.md new file mode 100644 index 0000000000..2d92806ad0 --- /dev/null +++ b/docs/clients.md @@ -0,0 +1,126 @@ +[Overview] | [Hive Commands] | [Simulators] | [Clients] + +## Hive Clients + +This page explains how client containers work in Hive. + +Clients are docker images which can be instantiated by a simulation. A client definition +consist of a Dockerfile and associated resources. Client definitions live in +subdirectories of `clients/` in the hive repository. + +When hive runs a simulation, it first builds all client docker images using their +Dockerfile, i.e. it basically runs `docker build .` in the client directory. Since most +client definitions wrap an existing Ethereum client, and building the client from source +may take a long time, it is usually best to base the hive client wrapper on a pre-built +docker image from Docker Hub. + +Client Dockerfiles should support an optional argument named `branch`, which specifies the +requested client version. This argument can be set by users by appending it to the client +name like: + + ./hive --sim my-simulation --client go-ethereum_v1.9.23,go_ethereum_v1.9.22 + +See the [go-ethereum client definition][geth-docker] for an example of a client +Dockerfile. + +## Client Lifecycle + +When the simulation requests a client instance, hive creates a docker container from the +client image. The simulator can customize the container by passing environment variables +with prefix `HIVE_`. It may also upload files into the container before it starts. Once +the container is created, hive simply runs the entry point defined in the `Dockerfile`. + +For all client containers, hive waits for TCP port 8545 to open before considering the +client ready for use by the simulator. If the client container does not open this port +within a certain timeout, hive assumes the client has failed to start. + +Environment variables and files interpreted by the entry point define a 'protocol' +between the simulator and client. While hive itself does not require support for any +specific variables or files, simulators usually expect client containers to be +configurable in certain ways. In order to run tests against multiple Ethereum clients, for +example, the simulator needs to be able to configure all clients for a specific blockchain +and make them join the peer-to-peer network used for testing. + +## Eth1 Client Requirements + +This section describes the requirements for Ethereum 1.x client wrappers in hive. Client +entry point scripts must support this interface in order to be tested by existing Ethereum +1.x-specific simulators. + +Clients must provide JSON-RPC over HTTP on TCP port 8545. They may also support JSON-RPC +over WebSocket on port 8546, but this is not strictly required. + +### Files + +The simulator may customize client startup by placing these files into the client container: + +- `/genesis.json` contains Ethereum genesis state in the JSON format used by Geth. This + file is mandatory. +- `/chain.rlp` contains RLP-encoded blocks to import before startup. +- `/blocks/` directory containing `.rlp` files. + +On startup, client entry point scripts must first load the genesis block and state into +the client implementation from `/genesis.json`. To do this, the script needs to translate +from Geth genesis format into a format appropriate for the specific client implementation. +The translation is usually done using a jq script. See the [openethereum genesis +translator][oe-genesis-jq], for example. + +After the genesis state, the client should import the blocks from `/chain.rlp` if it is +present, and finally import the individual blocks from `/blocks` in file name order. The +reason for requiring two different block sources is that specifying a single chain is more +optimal, but tests requiring forking chains cannot create a single chain. The client +should start even if the blocks are invalid, i.e. after the import, the client's 'best +block' should be the last valid, imported block. + +### Environment + +Clients must support the following environment variables. The client's entry point script +may map these to command line flags or use them generate a config file, for example. + +| Variable | Value | | +|----------------------------|----------------------|------------------------------------------------| +| `HIVE_LOGLEVEL` | 0 - 5 | configures log level of client | +| `HIVE_NODETYPE` | archive, full, light | sets sync algorithm | +| `HIVE_BOOTNODE` | enode URL | makes client connect to another node | +| `HIVE_GRAPHQL_ENABLED` | 0 - 1 | if set, GraphQL is enabled on port 8545 | +| `HIVE_MINER` | address | if set, mining is enabled. value is coinbase | +| `HIVE_MINER_EXTRA` | hex | extradata for mined blocks | +| `HIVE_CLIQUE_PERIOD` | decimal | enables clique PoA. value is target block time | +| `HIVE_CLIQUE_PRIVATEKEY` | hex | private key for signing of clique blocks | +| `HIVE_SKIP_POW` | 0 - 1 | disables PoW check during block import | +| `HIVE_NETWORK_ID` | decimal | p2p network ID | +| `HIVE_CHAIN_ID` | decimal | [EIP-155] chain ID | +| `HIVE_FORK_HOMESTEAD` | decimal | [Homestead][EIP-606] transition block | +| `HIVE_FORK_DAO_BLOCK` | decimal | [DAO fork][EIP-779] transition block | +| `HIVE_FORK_TANGERINE` | decimal | [Tangerine Whistle][EIP-608] transition block | +| `HIVE_FORK_SPURIOUS` | decimal | [Spurious Dragon][EIP-607] transition block | +| `HIVE_FORK_BYZANTIUM` | decimal | [Byzantium][EIP-609] transition block | +| `HIVE_FORK_CONSTANTINOPLE` | decimal | [Constantinople][EIP-1013] transition block | +| `HIVE_FORK_PETERSBURG` | decimal | [Petersburg][EIP-1716] transition block | +| `HIVE_FORK_ISTANBUL` | decimal | [Istanbul][EIP-1679] transition block | +| `HIVE_FORK_MUIRGLACIER` | decimal | [Muir Glacier][EIP-2387] transition block | +| `HIVE_FORK_BERLIN` | decimal | [Berlin][EIP-2070] transition block | + +### Enode script + +Some tests require peer-to-peer node information of the client instance. The client +container must contain an `/enode.sh` script that echoes the enode of the running +instance. This script is executed by the Hive host in order to retrieve the enode URL. + +[geth-docker]: ../clients/go-ethereum/Dockerfile +[oe-genesis-jq]: ../clients/openethereum/mapper.jq +[EIP-155]: https://eips.ethereum.org/EIPS/eip-155 +[EIP-606]: https://eips.ethereum.org/EIPS/eip-606 +[EIP-607]: https://eips.ethereum.org/EIPS/eip-607 +[EIP-608]: https://eips.ethereum.org/EIPS/eip-608 +[EIP-609]: https://eips.ethereum.org/EIPS/eip-609 +[EIP-779]: https://eips.ethereum.org/EIPS/eip-779 +[EIP-1013]: https://eips.ethereum.org/EIPS/eip-1013 +[EIP-1679]: https://eips.ethereum.org/EIPS/eip-1679 +[EIP-1716]: https://eips.ethereum.org/EIPS/eip-1716 +[EIP-2387]: https://eips.ethereum.org/EIPS/eip-2387 +[EIP-2070]: https://eips.ethereum.org/EIPS/eip-2070 +[Overview]: ./overview.md +[Hive Commands]: ./commandline.md +[Simulators]: ./simulators.md +[Clients]: ./clients.md diff --git a/docs/commandline.md b/docs/commandline.md new file mode 100644 index 0000000000..a9564f1048 --- /dev/null +++ b/docs/commandline.md @@ -0,0 +1,101 @@ +[Overview] | [Hive Commands] | [Simulators] | [Clients] + +## Running Hive + +The hive project is implemented in Go. You need to install Go version 1.13 or later to use +hive. To run simulations, you also need a working Docker setup, and hive needs to be run on the +same machine as dockerd. Using docker remotely is not supported at this time. We have also +not tested hive extensively on any OS but Linux, so you must run Linux to use hive. + +To get hive, you first need to clone the repository to any location, then build the hive +executable. + + git clone https://github.com/ethereum/hive + cd ./hive + go build . + +All hive commands should be run from within the root of the repository. To run a +simulation, use the following command: + + ./hive --sim --client + +For example, if you want to run the `discv4` test against geth and openethereum, here is +how the command would look: + + ./hive --sim devp2p/discv4 --client go-ethereum,openethereum + +The client list may contain any number of clients. You can select a specific client +version by appending it to the client name with `_`, for example: + + ./hive --sim devp2p/discv4 --client go-ethereum_v1.9.22,go-ethereum_v1.9.23 + +Simulation runs can be customized in many ways. Here's an overview of the available +command-line options. + +`--client.checktimelimit `: The timeout of waiting for clients to open up TCP +port 8545. If a very long chain is imported, this timeout may need to be quite long. A +lower value means that hive won't wait as long in case the node crashes and never opens +the RPC port. Defaults to 3 minutes. + +`--docker.pull`: Setting this option makes hive re-pull the base images of all built +docker containers. + +`--docker.output`: This enables printing of all docker container output to stderr. + +`--docker.nocache `: Regular expression selecting docker images to forcibly +rebuild. You can use this option during simulator development to ensure a new image is +built even when there are no changes to the simulator code. + +`--sim.timelimit `: Simulation timeout. Hive aborts the simulator if it exceeds +this time. There is no default timeout. + +`--sim.loglevel `: Selects log level of client instances. Supports values 0-5, +defaults to 3. Note that this value may be overridden by simulators for specific clients. +This sets the default value of `HIVE_LOGLEVEL` in client containers. + +`--sim.parallelism `: Sets max number of parallel clients/containers. This is +interpreted by simulators. It sets the `HIVE_PARALLELISM` environment variable. Defaults +to 1. + +`--sim.testlimit `: Max number of tests to execute per client. This is interpreted +by simulators. It sets the `HIVE_SIMLIMIT` environment variable. + +## Viewing simulation results (hiveview) + +The results of hive simulation runs are stored in JSON files containing test results, and +hive also creates several log files containing the output of the simulator and clients. To +view test results and logs in a web browser, you can use the `hiveview` tool. Build it +with: + + go build ./cmd/hiveview + +Run it like this to start the HTTP server: + + ./hiveview --serve --logdir ./workspace/logs + +This command runs a web interface on . The interface shows +information about all simulation runs for which information was collected. + +## Generating Ethereum 1.x test chains (hivechain) + +The `hivechain` tool allows you to create RLP-encoded blockchains for inclusion into +simulations. Build it with: + + go build ./cmd/hivechain + +To generate a chain of a desired length, run the following command: + + hivechain generate -genesis ./genesis.json -length 200 + +hivechain generates empty blocks by default. The chain will contain non-empty blocks if +the following accounts have balance in genesis state. You can find the corresponding +private keys in the hivechain source code. + +- `0x71562b71999873DB5b286dF957af199Ec94617F7` +- `0x703c4b2bD70c169f5717101CaeE543299Fc946C7` +- `0x0D3ab14BBaD3D99F4203bd7a11aCB94882050E7e` + +[Overview]: ./overview.md +[Hive Commands]: ./commandline.md +[Simulators]: ./simulators.md +[Clients]: ./clients.md diff --git a/docs/img/sim-overview.svg b/docs/img/sim-overview.svg new file mode 100644 index 0000000000..391f2d46c5 --- /dev/null +++ b/docs/img/sim-overview.svg @@ -0,0 +1,375 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 0000000000..a255e607a9 --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,142 @@ +[Overview] | [Hive Commands] | [Simulators] | [Clients] + +## What is Hive? + +Hive is a system for running integration tests against Ethereum clients. + +In hive, integration tests are called 'simulations'. A simulation is controlled by a +program (the 'simulator') written in any language. The simulator launches clients and +contains test logic. It reports test results back to hive, where they are aggregated for +display in a web browser. + +What makes hive different from other, generic CI infrastructure is the tight integration +of Ethereum clients and their features. Simulator programs usually don't need to care +about the differences between client implementations because hive provides a common +interface to launch and configure them all. At this time, clients can be configured for +any Ethereum 1 network definition, i.e. genesis block and hard fork activation block +numbers. Simulations can also instruct clients to load a pre-defined test chain and enable +block mining. You can find more information about client configuration in the [client +documentation]. + +Ethereum Foundation operates a public instance of Hive to check for consensus +compatibility, peer-to-peer networking spec compliance, and user API support for most +Ethereum client implementations. You can find the latest test results at +. + +## Overview of available simulators + +This is an overview of some of the simulators which are currently implemented and running +continuously on the production hive instance: + +- `devp2p/eth`, `devp2p/discv4`: These simulators run the 'eth' peer-to-peer protocol + tests and Discovery v4 protocol tests. The test suites themselves are maintained in the + go-ethereum repository. In their hive adaptation, the simulator launches the client with + a known test chain, obtains its peer-to-peer endpoint (the `enode://` URL) and sends + protocol messages to it. The client's responses are analyzed by the test suite to ensure + that they conform to the respective protocol specification. + +- `ethereum/sync`: This simulator attempts to synchronize the blockchain among all + clients. For each enabled client implementation, it creates one instance of the client + as the 'source'. The 'source' client is initialized with a known test chain. The + simulator then launches a 'sink' instance of every known client against the source and + checks whether the sink can synchronize the chain from the source client. + +- `ethereum/consensus`: This simulator runs the Ethereum 1 consensus tests against all + clients. While client implementers are generally expected to run these tests themselves, + they might not always run the latest tests, and may skip some of them if they take too + long. Running these tests in a hive simulation ensures that none are skipped. + +- `ethereum/rpc`: The RPC simulator configures a client for clique PoA mining and runs + various tests against the web3 JSON-RPC interface. These tests ensure that the client is + able to receive transactions via RPC, incorporate them into its chain, and report + transaction results via the standard APIs. + +- `ethereum/graphql`: This simulator initializes a client with a known test chain and + enables the GraphQL API endpoint. It then performs certain queries and compares their + output to known good outputs. + +## How it works + +This section explains how a single simulation run works. + +For a single run, the user provides the name of the simulator to run, and a set of client +names to run against. For example: + + ./hive --sim ethereum/sync --client go-ethereum,besu,openethereum + +Hive first builds simulator and client images using docker. It expects a Dockerfile in the +`./simulators/ethereum/sync` directory as well as a Dockerfile for each client (in +`./clients/*/Dockerfile`). + +While the simulator build must always work without error, it's OK for some client builds +to fail as long as one of them succeeds. This is because client code pulled from the +respective upstream repositories may occasionally fail to build. + +![hive simulation docker containers](./img/sim-overview.svg) + +Once all images are built, the simulator program is launched in a docker container. The +`HIVE_SIMULATOR` environment variable contains the HTTP server URL of the hive controller. +The [hive simulation API] can be accessed at this URL. The simulator launches clients and +reports test results through the API. + +When the simulator requests a client instance, the hive controller launches a new docker +container using the built client image. The client container entry point receives +configuration through environment variables and files provided by the simulator. Depending +on this configuration data, the client entry point configures the client's genesis state +and imports the test chain (if provided). The client is now expected to launch its network +endpoints for RPC and p2p communication. + +When the client has finished starting, the simulator program communicates with it on the +RPC and p2p endpoints. More than one client may be launched, and the clients can also +communicate with each other. + +During the simulation run, information about 'test suites' and their test cases must be +provided by the simulator via the simulation API. The hive controller collects this +information in a JSON file. It also collects client logs as well as the output of the +simulator program. All files are written to the results directory (`./workspace/logs`). + +When the simulator program exits, the simulator container and all client containers are +stopped and removed. The `hive` command then exits as well. + +## Simulation Output Files + +The results of simulation runs are stored in the 'result directory'. For every test suite +executed by a simulator, a JSON file like the following is created: + + { + "id": 0, + "name": "sync", + "description": "This test suite verifies that...", + "clientVersions": { + "besu": "", + "go-ethereum": "" + }, + "simLog": "1612356621-simulator-a9a2e71a6aabe509bbde35c79e7f0ed9c259a642c19ba0da6167fa9efd0ea5a1.log" + "testCases": { + "1": { + "name": "besu as sync source", + "description": "This loads the test chain...", + "start": "2021-02-03T12:50:21.77396767Z", + "end": "2021-02-03T12:51:56.080650164Z", + "summaryResult": { + "pass": true, + "details": "" + }, + "clientInfo": { + "893a6ea2": { + "ip": "172.17.0.4", + "name": "besu", + "instantiatedAt": "2021-02-03T12:51:04.371913809Z", + "logFile": "besu/client-893a6ea2.log" + } + } + } + } + } + +The result directory also contains log files of simulator and client output. + +[Overview]: ./overview.md +[Hive Commands]: ./commandline.md +[Simulators]: ./simulators.md +[Clients]: ./clients.md diff --git a/docs/simulators.md b/docs/simulators.md new file mode 100644 index 0000000000..307a024f2a --- /dev/null +++ b/docs/simulators.md @@ -0,0 +1,293 @@ +[Overview] | [Hive Commands] | [Simulators] | [Clients] + +## Hive Simulator Programming + +This guide explains how to write hive simulators. + +A simulator is a program written against the HTTP-based simulation API provided by hive. +Simulators can be written in any programming language as long as they are packaged using +docker. + +Simulators live in the `simulators/` directory of the hive repository. There is a +dedicated sub-directory for every simulator. When hive runs a simulation, it first builds +an image using `docker build` in the simulator directory, using the Dockerfile. The image +must contain all resources needed for testing. + +When the simulator container entry point runs, the `HIVE_SIMULATOR` environment variable +is set to the URL of the API server. + +The simulation API assumes a certain data model, and this model dictates how the API can +be used. In order to do anything with the API, the simulator must first request the start +of a *test suite* and remembers its ID. Test suites have a name and description assigned +by the simulator. All other resources provided by the API are scoped to the test suite and +are kept until the simulator ends the suite. + +Next, the simulator can start *test cases* on the suite. Test cases are named and also +have an ID assigned by the API server. Multiple test cases may be running at any time +within a single suite. Note that test suites do not have an overall pass / fail status, +the only way to signal failure is with a test. At least one test case must be started for +a suite, otherwise no results can be reported. + +Within the context of a test case, client containers may be started. Clients are +associated with the test and are shut down automatically when the test that started them +ends. If many tests are to be executed against a single client, it is good practice to +create a dedicated 'client launch' test just for starting the client, and then signal the +results of the other tests as individual test cases. + +The simulator must report the results of all running test cases before ending the test +suite. + +## Writing Simulators in Go + +While simulators may be written in any language (they're just docker containers after +all), hive provides a Go library that wraps the simulation API in a way that resembles the +standard library "testing" package. Be sure to check the [Go API reference] of package +hivesim for more information about writing simulators in Go. + +Simulators are contained in the hive repository as independent Go modules. To create one, +first create a new subdirectory in `./simulators` and initialize a Go module there: + + mkdir ./simulators/ethereum/my-test + cd ./simulators/ethereum/my-test + go mod init github.com/ethereum/hive/simulators/ethereum/my-test + go get github.com/ethereum/hive/hivesim@latest + +Now create the simulator program: + + package main + + import "github.com/ethereum/hive/hivesim" + + func main() { + suite := hivesim.Suite{ + Name: "my-suite", + Description: "This test suite performs some tests.", + } + suite.Add(hivesim.ClientTestSpec{ + Name: "the-test", + Description: "This is the example test case.", + Files: map[string]string{"/genesis.json": "genesis.json"}, + Run: runMyTest, + }) + hivesim.MustRunSuite(hivesim.New(), suite) + } + +Your simulator directory must also contain a Dockerfile so that hive can build it: + + FROM golang:1-alpine AS builder + RUN apk --no-cache add gcc musl-dev linux-headers + ADD . /source + WORKDIR /source + RUN go build -o ./sim . + + # Build the runner container. + FROM alpine:latest + ADD . / + COPY --from=builder /source/sim / + ENTRYPOINT ["./sim"] + +## Simulation API Reference + +This section lists all HTTP endpoints provided by the simulation API. + +### Suite and Test Case Endpoints + +#### Creating a test suite + + POST /testsuite + content-type: application/x-www-form-urlencoded + + name=test-suite-name&description=this%20suite%20does%20... + +This request signals the start of a test suite. The API responds with a test suite ID. + + 200 OK + content-type: text/plain + + 1 + +#### Ending a test suite + + DELETE /testsuite/{suite} + +This request ends a test suite. The simulator must end all running test cases before +ending the test suite. + +Response: + + 200 OK + +#### Creating a test case + + POST /testsuite/{suite}/test + content-type: application/x-www-form-urlencoded + +The API responds with a test case ID. + + 200 OK + content-type: text/plain + + 2 + +#### Ending a test case + + POST /testsuite/{suite}/test/{test} + content-type: application/x-www-form-urlencoded + + summaryresult=%7B%22pass%22%3Atrue%2C%22details%22%3A%22this%20is%20the%20test%20output%22%7D + +This request reports the result of a test case. The request body is a form submission +containing a single field `summaryresult`. The test result is a JSON object of the form: + + {"pass": true/false, "details": "text..."} + +Response: + + 200 OK + +### Working with clients + +#### Getting available client types + + GET /clients + +This returns a JSON array of client names available to the simulation run. + +Response + + 200 OK + content-type: application/json + + ["go-ethereum","besu"] + +#### Starting a client container + + POST /testsuite/{suite}/test/{test}/node + content-type: multipart/form-data; boundary=--boundary-- + + --boundary-- + content-disposition: form-data; name=CLIENT + + go-ethereum + --boundary-- + content-disposition: form-data; name=HIVE_CHAIN_ID + + 8 + --boundary-- + content-disposition: form-data; name=file; filename="/genesis.json" + + { + "difficulty": "0x20000", + "gasLimit": "0xFFFFFFFF", + ... + } + --boundary---- + +This request starts a client container. The request body must be encoded as multipart form data. + +The `CLIENT` form field is required and specifies the client type that should be started. +It must match one of the client names returned by the `/clients` endpoint. + +Other form fields, specifically those with a prefix of `HIVE_`, are passed to the client +entry point as environment variables. Please see the [client interface documentation] for +environment variables supported by Ethereum clients. + +Form fields with a filename are copied into the client container as files. + +Response: + + 200 OK + content-type: text/plain + + @@ + +#### Geting the enode URL of a running client + + GET /testsuite/{suite}/test/{test}/node/{container} + +This request returns the enode URL of a running client. + +Response: + + 200 OK + content-type: text/plain + + enode://1ba850b467b3b96eacdcb6c133d2c7907878794dbdfc114269c7f240d278594439f79975f87e43c45152072c9bd68f9311eb15fd37f1fd438812240e82de9ef9@172.17.0.3:30303 + +#### Stopping a client + + DELETE /testsuite/{suite}/test/{test}/node/{container} + +This terminates the given client container immediately. Using this endpoint is usually not +required because all clients associated with a test will be shut down when the test ends. + +Response: + + 200 OK + +### Networks + +#### Creating a network + + POST /testsuite/{suite}/network/{network} + +This request creates a network. Unlike with other APIs, networks do not have IDs. Instead, +the network name is assigned the API client. + +Response: + + 200 OK + +#### Removing a network + + DELETE /testsuite/{suite}/network/{network} + +This request removes a network. Note: the request will fail if containers are still +connected to the network. + +Response: + + 200 OK + +#### Connecting containers to a network + + POST /testsuite/{suite}/network/{network}/{container} + +This request connects a client container to a network. You can use any client container ID +as the `container`. You can also use `"simulation"` as the container ID, in which case the +simulator container will be connected. + +Response: + + 200 OK + +#### Disconnecting a container from a network + + DELETE /testsuite/{suite}/network/{network}/{container} + +This request disconnects a container from a network. As with the connect request, use any +client container ID or `"simulation"` as the `container` value. + +Response: + + 200 OK + +#### Getting the client IP + + GET /testsuite/{suite}/network/{network}/{container} + +This returns the IP of a client container on the given network. + +Response: + + 200 OK + content-type: text/plain + + 172.22.0.2 + +[client interface documentation]: ./clients.md +[Go API reference]: https://pkg.go.dev/github.com/ethereum/hive/hivesim +[Overview]: ./overview.md +[Hive Commands]: ./commandline.md +[Simulators]: ./simulators.md +[Clients]: ./clients.md