Update contributing.md (#1632)

.github/workflows/build.yaml

@@ -39,6 +39,9 @@ jobs:
         # so we should leave the cache alone so build_and_test can make more use of it.
       - name: Install ubuntu packages
         run: shotover-proxy/build/install_ubuntu_packages.sh
+      # This can be removed when we swap to aarch64 runners instead of cross compiling
+      - name: Install aarch64 gcc
+        run: sudo apt-get install -y gcc-aarch64-linux-gnu
       - name: Install cargo-hack
         uses: taiki-e/install-action@v2
         with:

docs/src/SUMMARY.md

@@ -16,5 +16,5 @@
     - [Aware client](./examples/redis-clustering-aware.md)
   - [Cassandra Cluster]()
     - [Shotover sidecars](./examples/cassandra-cluster-shotover-sidecar.md)
-- [Contributing](./contributing.md)
-  - [Debugging](debugging.md)
+- [Contributing](./dev-docs/contributing.md)
+  - [Debugging](./dev-docs/debugging.md)

docs/src/dev-docs/contributing.md

@@ -0,0 +1,81 @@
+# Contributing to Shotover
+
+This guide describes how to setup and use your environment for contributing to shotover itself.
+
+## Setup
+
+Shotover requires [rustup](https://rustup.rs/) to be installed so that the project specific rust version will be used.
+
+The rest of the setup is specific to your operating system.
+Shotover supports development on linux and partiallly supports macOS.
+
+### Setting up on Linux
+
+On linux, all tests will pass as long as the required dependencies are installed.
+
+See the [Linux specific setup instructions](setting-up-linux.md)
+
+### Setting up on macOS
+
+* All tests that use a single docker instance will pass. But anything with more than one docker instance will fail.
+* Tests that rely on external C++ dependencies cannot be built.
+  * They are hidden behind the `cassandra-cpp-driver-tests` and `kafka-cpp-driver-tests` feature flags to allow the rest of the tests to build on macOS
+
+Everything else should be buildable and pass.
+
+See the [macOS specific setup instructions](setting-up-macos.md)
+
+### Setting up via ec2-cargo
+
+The shotover repo contains a tool `ec2-cargo` which makes it easy to setup and a linux EC2 instance for developing shotover.
+This can be used by:
+
+* Linux users to test on a fresh machine or to test on a different cpu architecture
+* macOS users to run tests that do not run on macOS
+* To enable development from any other OS
+
+TODO: document setup and usage
+
+## Run Shotover tests
+
+Shotover's test suite must be run via [nextest](https://nexte.st) as we rely on its configuration to avoid running incompatible integration tests concurrently.
+
+To use nextest:
+
+1. Install nextest: `cargo install cargo-nextest --locked`
+2. Then run the tests: `cargo nextest run`
+
+The tests rely on configuration in `tests/test-configs/`, so if for example, you wanted to manually setup the services for the redis-passthrough test, you could run these commands in the `shotover-proxy` directory:
+
+* `docker-compose -f shotover-proxy/tests/test-configs/redis-passthrough/docker-compose.yaml up`
+* `cargo run -- --topology-file tests/test-configs/redis-passthrough/topology.yaml`
+
+## Submitting a PR
+
+Before submitting a PR you can run the following in preparation to make your PR more likely to pass CI:
+
+* `cargo fmt --all` - Ensure your code follows standard rust formatting.
+* `cargo clippy --all-features --all-targets` - Ensure you haven't introduced any warnings.
+* `cargo nextest run --all-features --all-targets` - Ensure all tests pass.
+
+For formatting you should configure your IDE to auto-format on save so that you never hit this issue at CI time.
+For clippy you can setup your IDE or a tool like [bacon](https://github.com/Canop/bacon) to run clippy while you work.
+If you find clippy too noisy, you should setup `cargo check` to run during development instead.
+But you will have to eventually resolve the clippy warnings before the PR can be merged.
+
+For the integration tests, CI will complete quicker than your local machine.
+So its much more realistic to just:
+
+1. Run some tests related to your changes
+2. Submit your PR as a draft
+3. See what fails
+4. Fix the failures before marking as ready to review.
+
+Also note that CI will run clippy against every permutation of features.
+So check what its doing in `.github/workflows/lint.yaml` if you have a failure in CI that is not reproducing locally.
+
+### Building Shotover (release)
+
+To build a release binary of shotover run `cargo build --release`.
+The built binary is located at `target/release/shotover-proxy`.
+The `--release` is very important, never deploy a non-release binary as it will be far too slow.

docs/src/dev-docs/setting-up-linux.md

@@ -0,0 +1,18 @@
+# Linux Specific setup
+
+## Building shotover
+
+Shotover requires a single external dependencies to build on linux.
+`gcc` must be installed due to the `ring` crate containing some C code.
+On Ubuntu you can install it via:
+
+```shell
+sudo apt-get install gcc
+```
+
+## Integration test dependencies
+
+Building and running integration tests and benchmarks requires many more external dependencies.
+To set them up on ubuntu run the script at `shotover-proxy/build/install_ubuntu_deps.sh`.
+Inspect the contents of the script to learn what it installs and why, some of the dependencies are optional, so feel free to skip those by installing the rest manually.
+If you already have docker installed make sure it is not a rootless install, see `install_ubuntu_deps.sh` for more information.

docs/src/dev-docs/setting-up-macos.md

@@ -0,0 +1,16 @@
+# macOS Specific Setup
+
+## Building shotover
+
+There are no external dependencies required for building shotover on macOS.
+
+## Integration test dependencies
+
+To run the tests capable of running on macOS, install the following dependencies:
+
+```shell
+brew install --cask docker
+brew install openssl@3
+```
+
+Make sure that docker desktop is running when you run the tests.

ec2-cargo/src/main.rs

@@ -66,7 +66,7 @@ until sudo apt-get update -qq
 do
   sleep 1
 done
-sudo apt-get install -y cmake pkg-config g++ libssl-dev librdkafka-dev uidmap unzip
+sudo apt-get install -y cmake pkg-config g++ libssl-dev librdkafka-dev unzip
 curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
 
 curl -sSL https://get.docker.com/ | sudo sh

shotover-proxy/build/install_ubuntu_deps.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+
+set -e
+
+if [ "$EUID" -e 0 ]
+  then echo "This script should not be run as root. It will run sudo itself as required."
+  exit
+fi
+cd "$(dirname "$0")"
+
+sudo apt-get update
+
+# Install dependencies for openssl, needed by `redis` crate
+sudo apt-get install -y pkg-config
+
+# Install docker
+curl -sSL https://get.docker.com/ | sudo sh
+# Do not use the rootless install of docker as many of our tests rely on the user created bridge networks
+# having their interface exposed to the host, which rootless install does not support.
+# Instead add your user to the docker group:
+usermod -aG docker $USER`
+
+# Install dependencies for kafka java driver tests
+sudo apt-get install -y default-jre-headless`
+
+# The remaining dependencies are for tests behind optional features.
+# So feel free to skip them.
+
+# Install dependencies needed for `--features kafka-cpp-driver-tests`
+sudo apt-get install -y cmake g++
+
+## Install dependencies for `--features cassandra-cpp-driver-tests`
+./install_ubuntu_packages.sh

shotover-proxy/build/install_ubuntu_packages.sh

@@ -1,13 +1,19 @@
 #!/bin/bash
 
+# The cassandra integration tests also support other drivers that don't rely on external dependencies.
+# But if you want to test the cassandra-cpp driver, you will need to install it.
+#
+# Upstream installation information and dependencies for the Cassandra CPP driver can be found [here](https://docs.datastax.com/en/developer/cpp-driver/2.16/).
+#
+# However that is likely unusable because datastax do not ship packages for modern ubuntu so we have our own script which will compile, package and install the driver on a modern ubuntu.
+# So to install the driver on ubuntu run this script.
+
 set -e
 
 cd "$(dirname "$0")"
 
-sudo apt-get update
-sudo apt-get install -y gcc-aarch64-linux-gnu
-
 # Install dependencies of the cpp-driver even if they are already on CI so that we can run this locally
+sudo apt-get update
 sudo apt-get install -y libuv1 libuv1-dev cmake g++ libssl-dev zlib1g-dev
 
 # set VERSION to one of the tags here: https://github.com/datastax/cpp-driver/tags

windsock-cloud-docker/src/container.rs

@@ -28,7 +28,7 @@ impl Container {
             .await;
             container_bash("apt-get update").await;
             container_bash(
-            "DEBIAN_FRONTEND=noninteractive apt-get install -y curl git cmake pkg-config g++ libssl-dev librdkafka-dev uidmap psmisc unzip",
+            "DEBIAN_FRONTEND=noninteractive apt-get install -y curl git cmake pkg-config g++ libssl-dev librdkafka-dev psmisc unzip",
         ).await;
             container_bash(
                 "curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y",