Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEATURE: Adding keeper #71

Merged
merged 135 commits into from
Aug 5, 2024
Merged
Show file tree
Hide file tree
Changes from 134 commits
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
6d34aa6
Admin methods for testing
ebatsell Jun 4, 2024
f78ef06
new program id
ebatsell Jun 5, 2024
57081b8
FIX: Skip reordering scored indexes if index DNE (#45)
coachchucksol Jun 10, 2024
7775fef
Christian/epoch update (#47)
coachchucksol Jun 13, 2024
f861b5f
CLI + FIXES: Steward CLI (#49)
coachchucksol Jun 17, 2024
4cf6c8b
starting to build monkey
coachchucksol Jun 17, 2024
41ed012
working
coachchucksol Jun 18, 2024
9cf1a08
adding in the flow
coachchucksol Jun 18, 2024
69ec8dc
have flow
coachchucksol Jun 19, 2024
ea4d8b5
setting up for keeper
coachchucksol Jun 19, 2024
4b84286
closter
coachchucksol Jun 19, 2024
6af54e0
closter
coachchucksol Jun 19, 2024
8d7826b
working
coachchucksol Jun 19, 2024
a1fadec
handling errors
coachchucksol Jun 20, 2024
5748852
testing removing and adding validators
coachchucksol Jun 20, 2024
16e5be6
startup flag added
coachchucksol Jun 21, 2024
6b0abce
adding to keeper
coachchucksol Jun 21, 2024
197b287
no more epoch forever loop
coachchucksol Jun 24, 2024
6b3ba23
Admin methods for testing
ebatsell Jun 4, 2024
0d8debe
new program id
ebatsell Jun 5, 2024
98e44ff
FIX: Skip reordering scored indexes if index DNE (#45)
coachchucksol Jun 10, 2024
d3d3d87
Christian/epoch update (#47)
coachchucksol Jun 13, 2024
2124bbe
CLI + FIXES: Steward CLI (#49)
coachchucksol Jun 17, 2024
c688bbd
loop fixed
CoachChuckFF Jun 24, 2024
235c003
Fixes after rebase
ebatsell Jun 24, 2024
52b535e
adding in last actions
CoachChuckFF Jun 24, 2024
a05f620
Merge branch 'steward-test-branch' of https://github.com/jito-foundat…
CoachChuckFF Jun 24, 2024
d0dd9de
merged
CoachChuckFF Jun 24, 2024
47d69d9
about to call auto remove validator
CoachChuckFF Jun 24, 2024
6e69fec
now removing bad validators
CoachChuckFF Jun 25, 2024
96b5989
removing bad validators
CoachChuckFF Jun 25, 2024
9ba4f70
removing validators work
CoachChuckFF Jun 25, 2024
4facad0
PROGRAM TWEAKS (#52)
coachchucksol Jun 28, 2024
b1b8516
deleted old stakenet
coachchucksol Jun 28, 2024
dad21f4
merged in new steward
coachchucksol Jun 28, 2024
28e9f65
adding to monkey
coachchucksol Jun 28, 2024
9598484
running monkey
coachchucksol Jun 28, 2024
e2d81e4
cranks now fail better
coachchucksol Jul 1, 2024
104f1b7
More events emitted (#54)
ebatsell Jul 1, 2024
3402078
cleaning up
coachchucksol Jul 1, 2024
514e76b
made logging a bit nicer
Jul 2, 2024
6f72f80
run keeper with monkey
Jul 3, 2024
9fcf03b
run keeper with monkey
Jul 3, 2024
be6a62c
little optimizations
Jul 3, 2024
92cd648
working
Jul 5, 2024
5bdcf4b
added keeper params
Jul 5, 2024
a1d3ce9
cursory logging added
Jul 5, 2024
1ec5880
metrics added
Jul 5, 2024
c455dc8
filtering out validators that do not meet criteria
Jul 8, 2024
12a161a
cleaning
Jul 10, 2024
30c1b20
added some metrics
Jul 11, 2024
69997fa
more metrics
Jul 11, 2024
35dc525
need to update program
Jul 16, 2024
f09da1c
working
Jul 17, 2024
be2fa23
REFACTOR: Added status flags (#55)
coachchucksol Jul 17, 2024
1c065f9
Merge branch 'steward-test-branch' of https://github.com/jito-foundat…
Jul 17, 2024
f8df08a
added state emits
Jul 17, 2024
ee537aa
new metrics
Jul 17, 2024
86a81c7
error code added for steward
Jul 18, 2024
f6cbfb2
catching invalid state
Jul 18, 2024
0425db0
FEATURE: Mark delinquent validators for instant removal (#57)
coachchucksol Jul 23, 2024
5965e01
making it run
Jul 23, 2024
95eb34b
check gossip
Jul 23, 2024
8691d1e
check gossip
Jul 23, 2024
3b33fc3
check gossip
Jul 23, 2024
a7dccae
FEATURE: Adding re-usable crank_checks function (#58)
coachchucksol Jul 23, 2024
aa93524
pulling in new program
Jul 23, 2024
004ad16
Evan/fix gossip (#61)
coachchucksol Jul 24, 2024
f4ea93a
TWEAKS: Last cleanup (#62)
coachchucksol Jul 24, 2024
7124d40
Merge branch 'steward-test-branch' of https://github.com/jito-foundat…
Jul 24, 2024
5f195da
working
Jul 24, 2024
7b7b6fe
getting ready for launch
Jul 25, 2024
34357ba
working
Jul 25, 2024
a924aea
cleanup
Jul 25, 2024
c31cabc
adding error codes
Jul 25, 2024
b2d4656
can now skip
Jul 25, 2024
891587b
working
Jul 25, 2024
77338b7
working
Jul 25, 2024
dbea67a
added stake_deposit_unstake_total
Jul 25, 2024
b1167d2
added stake_deposit_unstake_total
Jul 25, 2024
d874c57
increased retry count
Jul 25, 2024
f4d0b2c
working
Jul 25, 2024
16e155e
working
Jul 25, 2024
cf2eb09
showing more stats
Jul 25, 2024
3b74c3d
more metrics
Jul 25, 2024
a93b130
remove test
Jul 25, 2024
79bb6e2
added state code
Jul 26, 2024
0d1db64
emitting configurations
Jul 26, 2024
7878b44
more metrics
Jul 26, 2024
fbe5e68
working
Jul 26, 2024
2b971a6
added offsets
Jul 26, 2024
82333c6
cleaning
Jul 26, 2024
dea6f82
getting ready for PR
Jul 26, 2024
e9f9d68
ready for PR
Jul 26, 2024
10fe296
added unstake scoring amount
Jul 26, 2024
c63a56e
changed some formatting
Jul 27, 2024
7bd2bf1
fixed adding validators
Jul 27, 2024
0af7f34
adding cooldown to fires
Jul 27, 2024
c9295ad
started docs
Jul 28, 2024
f83602d
Missing emits and final program address (#65)
ebatsell Jul 29, 2024
ad711ce
PATCH: Carry over epoch maintenance flag (#64)
coachchucksol Jul 29, 2024
2021ba5
Fix build and steward verified build
ebatsell Jul 29, 2024
d5707ce
Merge branch 'steward-test-branch' of https://github.com/jito-foundat…
Jul 29, 2024
9be6f91
added ability to print out base58 txs
Jul 29, 2024
d4d1c1d
made change to readme
Jul 29, 2024
25c1474
working
Jul 29, 2024
6843535
working
Jul 29, 2024
5dee46b
added notes
Jul 29, 2024
c93c07a
working
Jul 29, 2024
3faa57e
copy vote account
Jul 29, 2024
30ae14b
Fix indices and checks for deactivating validators
ebatsell Jul 29, 2024
6365597
removed delinquent
Jul 29, 2024
80590f1
remove delinquient
Jul 29, 2024
51407e6
Merge branch 'steward-test-branch' of https://github.com/jito-foundat…
Jul 29, 2024
bd57a19
need to fix dd
Jul 29, 2024
9f20d85
REFACTOR: Condensed functions to an SDK package (#67)
coachchucksol Jul 30, 2024
d00b657
fixed env
Jul 31, 2024
bf5f5a0
fixed env
Jul 31, 2024
b83aecd
better docs
Jul 31, 2024
cae8cc2
remove history fetch
Jul 31, 2024
aef9708
working
Aug 1, 2024
992d941
updated rust version
Aug 1, 2024
4a04d86
can skip creating missing accounts
Aug 1, 2024
6c3f4aa
can now skip creating accounts
Aug 1, 2024
3061dc9
fixed deactivate delinquent
Aug 1, 2024
bbb4171
Merging in origin (#70)
coachchucksol Aug 1, 2024
4aa37ed
good to merge
Aug 1, 2024
27d4177
removed notes
Aug 1, 2024
f07ce24
format workspace
Aug 1, 2024
8a95460
fixed docker
Aug 1, 2024
b60d4f1
run metrics
Aug 2, 2024
fd3aa58
added in yaml
Aug 2, 2024
3db6bb1
comments addressed
Aug 5, 2024
bc5d761
Made linter happy
Aug 5, 2024
3b9cedd
Apply suggestions from code review
ebatsell Aug 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 31 additions & 18 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
[workspace]
members = [
"keepers/*",
"programs/*",
"programs/*",
"sdk",
"tests",
"utils/*",
]
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM rust:1.71-slim-buster as builder
FROM rust:1.75-slim-buster as builder

RUN apt-get update && apt-get install -y libudev-dev clang pkg-config libssl-dev build-essential cmake protobuf-compiler

Expand Down
90 changes: 71 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
# Validator History Program
# Stakenet

## About

Jito StakeNet is a decentralized Solana stake pool manager, blending Validator History and Steward Programs for secure, transparent validator management and autonomous stake operations.

## Validator History Program

The Validator History Program, a component of Jito StakeNet, is an on-chain record of verified Solana validator data, storing up to 512 epochs of history per validator. It takes fields accessible to the solana runtime like validator performance history, validator commission, MEV commission, as well as Gossip data like validator IP, version, and client type, and stores them all in a single account. It also contains some fields that currently require permissioned upload but are easily verifiable with a getVoteAccounts call, like total active stake per validator, stake rank, and superminority status. All these fields are stored in a single account per validator, the ValidatorHistory account. This enables all these disparate fields to be easily composed with in on chain programs, with a long lookback period and ease of access through the single account.

## Structure
### Structure

The main Anchor program is in `programs/validator-history`.

Expand All @@ -22,43 +26,51 @@ Note that this is a `zero_copy` account, which allows us to initialize a lot of

`Config`: Tracks admin authorities as well as global program metadata.

## Test
## Steward Program

Tests are in `tests/` written with solana-program-test.
Harnessing on-chain validator metrics and network data, the Steward Program employs advanced algorithms to evaluate and rank validators. Automated keepers then execute a state machine to optimally allocate stake, maximizing network security and efficiency.

All tests can be run by running:
```shell
./run_tests.sh
```
On-chain Steward accounts for JitoSOL:

| Account | Address |
|-----------------|---------------------------------------------|
| Program | Stewardf95sJbmtcZsyagb2dg4Mo8eVQho8gpECvLx8 |
| Steward Config | jitoVjT9jRUyeXHzvCwzPgHj7yWNRhLcUoXtes4wtjv |
| Steward State | 9BAmGVLGxzqct6bkgjWmKSv3BFB6iKYXNBQp8GWG1LDY|
| Authority | 9eZbWiHsPRsxLSiHxzg2pkXsAuQMwAjQrda7C7e21Fw6|

## Build
## Build and Test

### Build

`anchor build --program-name validator_history` (regular anchor build)
`solana-verify build --library-name validator_history` (solana verified build)

## Verify
### Verify

Verify with [solana-verifiable-build](https://github.com/Ellipsis-Labs/solana-verifiable-build):

`solana-verify verify-from-repo -um --program-id HistoryJTGbKQD2mRgLZ3XhqHnN811Qpez8X9kCcGHoa https://github.com/jito-foundation/stakenet`

## Running Keeper
### Test

Run as binary:
Tests are in `tests/` written with solana-program-test.

Build: `cargo b -r --package validator-keeper`
All tests can be run by running ( root directory ):

Run: `./target/release/validator-keeper --json-rpc-url <YOUR RPC> --cluster mainnet --tip-distribution-program-id F2Zu7QZiTYUhPd7u9ukRVwxh7B71oA3NMJcHuCHc29P2 --program-id HistoryJTGbKQD2mRgLZ3XhqHnN811Qpez8X9kCcGHoa --interval 600 --keypair <YOUR KEYPAIR>`
```shell
./run_tests.sh
```

Run as docker container (need to set environment variables in config/.env file):
## Running Keeper

`docker compose --env-file config/.env up -d --build validator-keeper`
Check out the [Keeper Bot Quick Start](./keeper-bot-quick-start.md)

Metrics for running can be sent to your influx server if you set the SOLANA_METRICS_CONFIG env var.
## CLIs

## CLI
### Validator History

The CLI can be used to see the status of on-chain validator history data.
This CLI can be used to see the status of on-chain validator history data.

Build: `cargo b -r --package validator-history-cli`

Expand All @@ -69,3 +81,43 @@ To see the current epoch state of all validator history accounts:
To see the historical state of a single validator history account:

`./target/release/validator-history-cli --json-rpc-url <YOUR RPC URL> history <VOTE ACCOUNT>`

### Steward

This CLI can be used to see the status of on-chain steward data.

Build:

```bash
cargo b -r --package steward-cli
```

To view the config:

```bash
./target/release/steward-cli --program-id Stewardf95sJbmtcZsyagb2dg4Mo8eVQho8gpECvLx8 view-config --steward-config jitoVjT9jRUyeXHzvCwzPgHj7yWNRhLcUoXtes4wtjv
```

To view the state:
(Note: This fetches a lot of accounts, you may want to use your own RPC)

```bash
./target/release/steward-cli --json-rpc-url YOUR_RPC view-state --steward-config jitoVjT9jRUyeXHzvCwzPgHj7yWNRhLcUoXtes4wtjv
```

To see the state of each validator in the context of the steward add `--verbose`

```bash
./target/release/steward-cli --json-rpc-url YOUR_RPC view-state --steward-config jitoVjT9jRUyeXHzvCwzPgHj7yWNRhLcUoXtes4wtjv --verbose
```

> TIP: To use your own RPC configured in your solana config, use the following:
> `--json-rpc-url $(solana config get | grep "RPC URL" | awk '{print $3}')`

To see all of the available commands:

```bash
./target/release/steward-cli -h
```

To see more info on the Steward CLI check out the [CLI notes](./utils/steward-cli/steward_cli_notes.md)
38 changes: 38 additions & 0 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
version: "3"
services:
validator-keeper:
build:
context: .
target: validator-history
container_name: validator-keeper
environment:
- RUST_LOG=${RUST_LOG:-info}
- SOLANA_METRICS_CONFIG=${SOLANA_METRICS_CONFIG}
- JSON_RPC_URL=${JSON_RPC_URL}
- KEYPAIR=${KEYPAIR}
- ORACLE_AUTHORITY_KEYPAIR=${ORACLE_AUTHORITY_KEYPAIR}
- CLUSTER=${CLUSTER}
- VALIDATOR_HISTORY_PROGRAM_ID=${VALIDATOR_HISTORY_PROGRAM_ID}
- TIP_DISTRIBUTION_PROGRAM_ID=${TIP_DISTRIBUTION_PROGRAM_ID}
- STEWARD_PROGRAM_ID=${STEWARD_PROGRAM_ID}
- STEWARD_CONFIG=${STEWARD_CONFIG}
- PRIORITY_FEES=${PRIORITY_FEES}
- RUN_CLUSTER_HISTORY=${RUN_CLUSTER_HISTORY}
- RUN_COPY_VOTE_ACCOUNTS=${RUN_COPY_VOTE_ACCOUNTS}
- RUN_MEV_COMMISSION=${RUN_MEV_COMMISSION}
- RUN_MEV_EARNED=${RUN_MEV_EARNED}
- RUN_STEWARD=${RUN_STEWARD}
- VALIDATOR_HISTORY_INTERVAL=${VALIDATOR_HISTORY_INTERVAL}
- STEWARD_INTERVAL=${STEWARD_INTERVAL}
- METRICS_INTERVAL=${METRICS_INTERVAL}
- RUN_STAKE_UPLOAD=${RUN_STAKE_UPLOAD}
- RUN_GOSSIP_UPLOAD=${RUN_GOSSIP_UPLOAD}
- RUN_EMIT_METRICS=${RUN_EMIT_METRICS}
- FULL_STARTUP=${FULL_STARTUP}
- NO_PACK=${NO_PACK}
- PAY_FOR_NEW_ACCOUNTS=${PAY_FOR_NEW_ACCOUNTS}
- COOL_DOWN_RANGE=${COOL_DOWN_RANGE}
- GOSSIP_ENTRYPOINT=${GOSSIP_ENTRYPOINT}
volumes:
- ./credentials:/credentials
restart: on-failure:5
Loading
Loading