Skip to content

Commit

Permalink
rewrite readme
Browse files Browse the repository at this point in the history
  • Loading branch information
rukai committed Nov 25, 2024
1 parent 63a8dcd commit 060dca0
Showing 1 changed file with 16 additions and 27 deletions.
43 changes: 16 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,38 +19,40 @@ To build Shotover from source please refer to [the contributing documentation](h

Shotover is a high performance, configurable and extensible L7 data-layer proxy for controlling, managing and modifying the flow of database requests in transit. It can be used to solve many different operational and interoperability challenges by transparently intercepting and transforming queries. It is transparent in the sense that it can be plugged into your architecture without requiring application change.

Shotover currently supports intercepting requests for the following technologies (sources):
Shotover currently supports the following databases:

* Cassandra (CQL4)
* Redis (RESP2)
* Cassandra (CQL4 + CQL5)
* Valkey/Redis (RESP2)
* Kafka (Kafka Wire Protocol)

It currently supports writing output to the following technologies (sinks):

* Cassandra
* Redis
* Kafka

## What problems does Shotover solve?

Concrete examples where Shotover has been applied include:

* [Multi-region, active-active redis](https://github.com/shotover/shotover-examples/tree/main/redis-backup-cluster)
<!--* [Cassandra query caching in redis, with a query audit trail sent to kafka](shotover-proxy/tests/test-configs/cass-redis-kafka/)-->
* [Multi-region, active-active valkey](https://github.com/shotover/shotover-examples/tree/main/valkey-backup-cluster)
* [Cassandra AWS PrivateLink](https://github.com/shotover/shotover-examples/blob/main/cassandra-1-many/topology.yaml)
* [Kafka GCP Private Service Connect](https://github.com/shotover/shotover-examples/blob/main/kafka-1-many/topology.yaml)

More broadly, Shotover is designed to be used for a very wide ranging class of problems where it is useful to transparently intercept a database call and redirect it. This allows you to change the behaviour of running applications at the infrastructure level without change to the application code itself.
More broadly, Shotover is designed to be used for a very wide ranging class of problems where it is useful to transparently intercept a database call and redirect/rewrite it. This allows you to change the behaviour of running applications at the infrastructure level without change to the application code itself.
Some examples where we envisage Shotover could be deployed include:

* Enabling use of AWS private link or GCP private service connect for database's that do not support distinct load balancer nodes.
* Moving very large or very hot tenants/customers/keys (that can cause unbalanced partition problems in some systems) to a separate data store by intercepting and redirecting queries for those particular keys
* Dual writing and/or query translation to allow the underlying storage technology to be changed (for example, from DynamoDB to Apache Cassandra)
* As an alternative to Change Data Capture technology to send writes to a message stream such as Apache Kafka in addition to the primary database
* Adding auditing, encryption or other security measures

Of course, Shotover is designed to be configurable and extensible so use your imagination and let us know what uses you find!
Shotover does not provide transforms for all of these yet, but with a custom transform these can be achieved today.

## Custom Transforms

Shotover comes with some predefined Transforms for common operations.
However shotover also allows for user defined [Custom Transforms](https://github.com/shotover/shotover-custom-transforms-template) to be written in rust.
Custom Transforms are given access to the AST of all messages passing through, allowing for any kind of transformation to be performed.

## Deploying Shotover

Shotover can be deployed in several different ways based on the problem you are trying to solve, but they all fall into three categories:
Shotover can be deployed in several different ways based on the problem you are trying to solve:

* As an application sidecar - Shotover is pretty lightweight, so feel free to deploy it as a sidecar to each of your application
instances.
Expand All @@ -59,19 +61,6 @@ and really let it fly.
* As a sidecar to your database - You can also stick Shotover on the same instance/server as your database is running on; we do it, so
we won't judge you.

## Roadmap

* Support relevant xDS APIs (so Shotover can play nicely with service mesh implementations)
* Support hot-reloads and a dynamic configuration API.
* Performance metrics.
* Additional sources (DynamoDB and PostgreSQL are good first candidates).
* Add support for rate limiting, explicit back-pressure mechanisms, etc.
* Additional distributed algorithm transform primitives (e.g RAFT, 2PC, etc)
* Additional sink transforms (these generally get implemented alongside sources).
* Support user-defined / generated sources (e.g. thrift or a gRPC service from a proto definition).
* Simulation testing once tokio-rs/simulation reaches compatibility with tokio-2.0.
* Zero-copy pass-through transforms and in-place query editing (performance).

## Name

Shotover refers to the Shotover (Kimi-ākau) river in Otago, New Zealand - close to Queenstown and eventually flowing into Lake Wakatipu
Expand Down

0 comments on commit 060dca0

Please sign in to comment.