From 5ed7a70a14e6a7f26de5d54208ae67235c16b346 Mon Sep 17 00:00:00 2001 From: Lucas Kent Date: Tue, 26 Nov 2024 16:43:24 +1100 Subject: [PATCH] rewrite readme (#1839) --- README.md | 43 ++++++++++++++++--------------------------- 1 file changed, 16 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index bd3c75c1e..b6a3e4f23 100644 --- a/README.md +++ b/README.md @@ -19,38 +19,40 @@ To build Shotover from source please refer to [the contributing documentation](h Shotover is a high performance, configurable and extensible L7 data-layer proxy for controlling, managing and modifying the flow of database requests in transit. It can be used to solve many different operational and interoperability challenges by transparently intercepting and transforming queries. It is transparent in the sense that it can be plugged into your architecture without requiring application change. -Shotover currently supports intercepting requests for the following technologies (sources): +Shotover currently supports the following technologies: -* Cassandra (CQL4) -* Redis (RESP2) +* Cassandra (CQL4 + CQL5) +* Valkey/Redis (RESP2) * Kafka (Kafka Wire Protocol) -It currently supports writing output to the following technologies (sinks): - -* Cassandra -* Redis -* Kafka - ## What problems does Shotover solve? Concrete examples where Shotover has been applied include: -* [Multi-region, active-active redis](https://github.com/shotover/shotover-examples/tree/main/redis-backup-cluster) - +* [Multi-region, active-active valkey](https://github.com/shotover/shotover-examples/tree/main/valkey-backup-cluster) +* [Cassandra AWS PrivateLink](https://github.com/shotover/shotover-examples/blob/main/cassandra-1-many/topology.yaml) +* [Kafka GCP Private Service Connect](https://github.com/shotover/shotover-examples/blob/main/kafka-1-many/topology.yaml) -More broadly, Shotover is designed to be used for a very wide ranging class of problems where it is useful to transparently intercept a database call and redirect it. This allows you to change the behaviour of running applications at the infrastructure level without change to the application code itself. +More broadly, Shotover is designed to be used for a very wide ranging class of problems where it is useful to transparently intercept a database call and redirect/rewrite it. This allows you to change the behaviour of running applications at the infrastructure level without change to the application code itself. Some examples where we envisage Shotover could be deployed include: +* Enabling use of AWS PrivateLink or GCP Private Service Connect for databases that do not support distinct load balancer nodes. * Moving very large or very hot tenants/customers/keys (that can cause unbalanced partition problems in some systems) to a separate data store by intercepting and redirecting queries for those particular keys * Dual writing and/or query translation to allow the underlying storage technology to be changed (for example, from DynamoDB to Apache Cassandra) * As an alternative to Change Data Capture technology to send writes to a message stream such as Apache Kafka in addition to the primary database * Adding auditing, encryption or other security measures -Of course, Shotover is designed to be configurable and extensible so use your imagination and let us know what uses you find! +Shotover does not provide transforms for all of these yet, but with a custom transform these can be achieved today. + +## Custom Transforms + +Shotover comes with some predefined Transforms for common operations. +However Shotover also allows for user defined [Custom Transforms](https://github.com/shotover/shotover-custom-transforms-template) to be written in rust. +Custom Transforms are given access to the AST of all messages passing through, allowing for any kind of transformation to be performed. ## Deploying Shotover -Shotover can be deployed in several different ways based on the problem you are trying to solve, but they all fall into three categories: +Shotover can be deployed in several different ways based on the problem you are trying to solve: * As an application sidecar - Shotover is pretty lightweight, so feel free to deploy it as a sidecar to each of your application instances. @@ -59,19 +61,6 @@ and really let it fly. * As a sidecar to your database - You can also stick Shotover on the same instance/server as your database is running on; we do it, so we won't judge you. -## Roadmap - -* Support relevant xDS APIs (so Shotover can play nicely with service mesh implementations) -* Support hot-reloads and a dynamic configuration API. -* Performance metrics. -* Additional sources (DynamoDB and PostgreSQL are good first candidates). -* Add support for rate limiting, explicit back-pressure mechanisms, etc. -* Additional distributed algorithm transform primitives (e.g RAFT, 2PC, etc) -* Additional sink transforms (these generally get implemented alongside sources). -* Support user-defined / generated sources (e.g. thrift or a gRPC service from a proto definition). -* Simulation testing once tokio-rs/simulation reaches compatibility with tokio-2.0. -* Zero-copy pass-through transforms and in-place query editing (performance). - ## Name Shotover refers to the Shotover (Kimi-ākau) river in Otago, New Zealand - close to Queenstown and eventually flowing into Lake Wakatipu