jito patch

only reroute if relayer connected (#123) feat: add client tls config (#121) remove extra val (#129) fix clippy (#130) copy all binaries to docker-output (#131) Ledger tool halts at slot passed to create-snapshot (#118) update program submodule (#133) quick fix for tips and clearing old bundles (#135) update submodule to new program (#136) Improve stake-meta-generator usability (#134) pinning submodule head (#140) Use BundleAccountLocker when handling tip txs (#147) Add metrics for relayer + block engine proxy (#149) Build claim-mev in docker (#141) Rework bundle receiving and add metrics (#152) (#154) update submodule + dev files (#158) Deterministically find tip amounts, add meta to stake info, and cleanup pubkey/strings in MEV tips (#159) update jito-programs submodule (#160) Separate MEV tip related workflow (#161) Add block builder fee protos (#162) fix jito programs (#163) update submodule so autosnapshot exits out of ledger tool early (#164) Pipe through block builder fee (#167) pull in new snapshot code (#171) block builder bug (#172) Pull in new slack autosnapshot submodule (#174) sort stake meta json and use int math (#176) add accountsdb conn submod (#169) Update tip distribution parameters (#177) new submodules (#180) Add buildkite link for jito CI (#183) Fixed broken links to repositories (#184) Changed from ssh to https transfer for clone Seg/update submods (#187) fix tests (#190) rm geyser submod (#192) rm dangling geyser references (#193) fix syntax err (#195) use deterministic req ids in batch calls (#199) update jito-programs revert cargo update Cargo lock update with path fix fix cargo update autosnapshot with block lookback (#201) [JIT-460] When claiming mev tips, skip accounts that won't have min rent exempt amount after claiming (#203) Add logging for sol balance desired (#205) * add logging * add logging * update msg * tweak vars update submodule (#204) use efficient data structures when calling batch_simulate_bundles (#206) [JIT-504] Add low balance check in uploading merkle roots (#209) add config to simulate on top of working bank (#211) rm frozen bank check simulate_bundle rpc bugfixes (#214) rm frozen bank check in simulate_bundle rpc method [JIT-519] Store ClaimStatus address in merkle-root-json (#210) * add files * switch to include bump update submodule (#217) add amount filter (#218) update autosnapshot (#222) Print TX error in Bundles (#223) add new args to support single relayer and block-engine endpoints (#224) point to new jito-programs submod and invoke updated init tda instruction (#228) fix clippy errors (#230) fix validator start scripts (#232) Point README to gitbook (#237) use packaged cargo bin to build (#239) Add validator identity pubkey to StakeMeta (#226) The vote account associated with a validator is not a permanent link, so log the validator identity as well. bugfix: conditionally compile with debug flags (#240) Seg/tip distributor master (#242) * validate tree nodes * fix unit tests * pr feedback * bump jito-programs submod Simplify bootstrapping (#241) * startup without precompile * update spacing * use release mode * spacing fix validation rm validation skip Account for block builder fee when generating excess tip balance (#247) Improve docker caching delay constructing claim mev txs (#253) fix stake meta tests from bb fee (#254) fix tests Buffer bundles that exceed cost model (#225) * buffer bundles that exceed cost model clear qos failed bundles buffer if not leader soon (#260) update Cargo.lock to correct solana versions in jito-programs submodule (#265) fix simulate_bundle client and better error handling (#267) update submod (#272) Preallocate Bundle Cost (#238) fix Dockerfile (#278) Fix Tests (#279) Fix Tests (#281) * fix tests update jito-programs submod (#282) add reclaim rent workflow (#283) update jito-programs submod fix clippy errs rm wrong assertion and swap out file write fn call (#292) Remove security.md (#293) demote frequent relayer_stage-stream_error to warn (#275) account for case where TDA exists but not allocated (#295) implement better retries for tip-distributor workflows (#297) limit number of concurrent rpc calls (#298) Discard Empty Packet Batches (#299) Identity Hotswap (#290) small fixes (#305) Set backend config from admin rpc (#304) Admin Shred Receiver Change (#306) Seg/rm bundle UUID (#309) Fix github workflow to recursively clone (#327) Add recursive checkout for downstream-project-spl.yaml (#341) Use cluster info functions for tpu (#345) Use git rev-parse for git sha Remove blacklisted tx from message_hash_to_transaction (#374) Updates bootstrap and start scripts needed for local dev. (#384) Remove Deprecated Cli Args (#387) Master Rebase improve simulate_bundle errors and response (#404) derive Clone on accountoverrides (#416) Add upsert to AccountOverrides (#419) update jito-programs (#430) [JIT-1661] Faster Autosnapshot (#436) Reverts simulate_transaction result calls to upstream (#446) Don't unlock accounts in TransactionBatches used during simulation (#449) first pass at wiring up jito-plugin (#428) [JIT-1713] Fix bundle's blockspace preallocation (#489) [JIT-1708] Fix TOC TOU condition for relayer and block engine config (#491) [JIT-1710] - Optimize Bundle Consumer Checks (#490) Add Blockhash Metrics to Bundle Committer (#500) add priority fee ix to mev-claim (#520) Update Autosnapshot (#548) Run MEV claims + reclaiming rent-exempt amounts in parallel. (#582) Update CI (#584) - Add recursive submodule checkouts. - Re-add solana-secondary step Add more release fixes (#585) Fix more release urls (#588) [JIT-1812] Fix blocking mutexs (#495) [JIT-1711] Compare the unprocessed transaction storage BundleStorage against a constant instead of VecDeque::capacity() (#587) Automatically rebase Jito-Solana on a periodic basis. Send message on slack during any failures or success. Fix periodic rebase #594 Fixes the following bugs in the periodic rebase: Sends multiple messages on failure instead of one Cancels entire job if one branch fails Ignore buildkite curl errors for rebasing and try to keep curling until job times out (#597) Sleep longer waiting for buildkite to start (#598) correctly initialize account overrides (#595) Fix: Ensure set contact info to UDP port instead of QUIC (#603) Add fast replay branch to daily rebase (#607) take a snapshot of all bundle accounts before sim (#13) (#615) update jito-programs submodule Export agave binaries during docker build (BP #627) (#628) Backport #611 (#631) Publish releases to S3 and GCS (#633) (#634) Add packet flag for from staked sender (#655) Co-authored-by: Jed <[email protected]> Add bundle storage to new unprocessed transaction storage method Loosen tip requirement [v2.0] (#685) Add comments around ignoring the slot returned from ImmutableDeserializedPacket::build_sanitized_transaction Backport: Separate out broadcast + retransmit shredstream (#703) (#704)
jito-foundation · Dec 12, 2024 · 83df46f · 83df46f
1 parent 10b95d1
commit 83df46f
Show file tree

Hide file tree

Showing 180 changed files with 17,830 additions and 876 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -0,0 +1,9 @@
+.dockerignore
+.git/
+.github/
+.gitignore
+.idea/
+README.md
+Dockerfile
+f
+target/
diff --git a/.github/workflows/cargo.yml b/.github/workflows/cargo.yml
@@ -35,6 +35,8 @@ jobs:
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - uses: mozilla-actions/[email protected]
         with:

diff --git a/.github/workflows/changelog-label.yml b/.github/workflows/changelog-label.yml
@@ -13,6 +13,7 @@ jobs:
     - uses: actions/checkout@v4
       with:
         fetch-depth: 0
+        submodules: 'recursive'
     - name: Check if changes to CHANGELOG.md
       shell: bash
       env:

diff --git a/.github/workflows/client-targets.yml b/.github/workflows/client-targets.yml
@@ -32,6 +32,8 @@ jobs:
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - run: cargo install [email protected]
 
@@ -56,6 +58,8 @@ jobs:
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - name: Setup Rust
         run: |

diff --git a/.github/workflows/crate-check.yml b/.github/workflows/crate-check.yml
@@ -19,6 +19,7 @@ jobs:
       - uses: actions/checkout@v4
         with:
           fetch-depth: 0
+          submodules: 'recursive'
 
       - name: Get commit range (push)
         if: ${{ github.event_name == 'push' }}

diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
@@ -22,6 +22,7 @@ jobs:
         uses: actions/checkout@v4
         with:
           fetch-depth: 0
+          submodules: 'recursive'
 
       - name: Get commit range (push)
         if: ${{ github.event_name == 'push' }}
@@ -77,6 +78,8 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - name: Setup Node
         uses: actions/setup-node@v4

diff --git a/.github/workflows/downstream-project-anchor.yml b/.github/workflows/downstream-project-anchor.yml
@@ -43,10 +43,12 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        version: ["v0.29.0", "v0.30.0"]
+        version: [ "v0.29.0", "v0.30.0" ]
     if: false # Re-enable once new major versions for spl-token-2022 and spl-pod are out
     steps:
       - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - shell: bash
         run: |

diff --git a/.github/workflows/downstream-project-spl.yml b/.github/workflows/downstream-project-spl.yml
@@ -42,6 +42,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - shell: bash
         run: |
@@ -68,7 +70,7 @@ jobs:
         arrays:
           [
             {
-              test_paths: ["token/cli"],
+              test_paths: [ "token/cli" ],
               required_programs:
                 [
                   "token/program",
@@ -78,14 +80,14 @@ jobs:
                 ],
             },
             {
-              test_paths: ["single-pool/cli"],
+              test_paths: [ "single-pool/cli" ],
               required_programs:
                 [
                   "single-pool/program",
                 ],
             },
             {
-              test_paths: ["token-upgrade/cli"],
+              test_paths: [ "token-upgrade/cli" ],
               required_programs:
                 [
                   "token-upgrade/program",
@@ -94,6 +96,8 @@ jobs:
           ]
     steps:
       - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - shell: bash
         run: |
@@ -128,26 +132,28 @@ jobs:
     strategy:
       matrix:
         programs:
-          - [token/program]
+          - [ token/program ]
           - [
-              instruction-padding/program,
-              token/program-2022,
-              token/program-2022-test,
-            ]
+            instruction-padding/program,
+            token/program-2022,
+            token/program-2022-test,
+          ]
           - [
-              associated-token-account/program,
-              associated-token-account/program-test,
-            ]
-          - [token-upgrade/program]
-          - [feature-proposal/program]
-          - [governance/addin-mock/program, governance/program]
-          - [memo/program]
-          - [name-service/program]
-          - [stake-pool/program]
-          - [single-pool/program]
+            associated-token-account/program,
+            associated-token-account/program-test,
+          ]
+          - [ token-upgrade/program ]
+          - [ feature-proposal/program ]
+          - [ governance/addin-mock/program, governance/program ]
+          - [ memo/program ]
+          - [ name-service/program ]
+          - [ stake-pool/program ]
+          - [ single-pool/program ]
 
     steps:
       - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - shell: bash
         run: |

diff --git a/.github/workflows/rebase.yaml b/.github/workflows/rebase.yaml
@@ -0,0 +1,181 @@
+# This workflow runs a periodic rebase process, pulling in updates from an upstream repository
+# The workflow for rebasing a jito-solana branch to a solana labs branch locally is typically:
+# $ git checkout v1.17
+# $ git pull --rebase # --rebase needed locally
+# $ git branch -D lb/v1.17_rebase # deletes branch from last v1.17 rebase
+# $ git checkout -b lb/v1.17_rebase
+# $ git fetch upstream
+# $ git rebase upstream/v1.17 # rebase + fix merge conflicts
+# $ git rebase --continue
+# $ git push origin +lb/v1.17_rebase # force needed to overwrite remote. wait for CI, fix if any issues
+# $ git checkout v1.17
+# $ git reset --hard lb/v1.17_rebase
+# $ git push origin +v1.17
+#
+# This workflow automates this process, with periodic status updates over slack.
+# It will also run CI and wait for it to pass before performing the force push to v1.17.
+# In the event there's a failure in the process, it's reported to slack and the job stops.
+
+name: "Rebase jito-solana from upstream anza-xyz/agave"
+
+on:
+  #  push:
+  schedule:
+    - cron: "30 18 * * 1-5"
+
+jobs:
+  rebase:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        include:
+          - branch: master
+            rebase: upstream/master
+          - branch: v1.18
+            rebase: upstream/v1.18
+          - branch: v1.17
+            rebase: upstream/v1.17
+            # note: this will always be a day behind because we're rebasing from the previous day's rebase
+            # and NOT upstream
+          - branch: v1.17-fast-replay
+            rebase: origin/v1.17
+      fail-fast: false
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ matrix.branch }}
+          submodules: recursive
+          fetch-depth: 0
+          token: ${{ secrets.JITO_SOLANA_RELEASE_TOKEN }}
+      - name: Add upstream
+        run: git remote add upstream https://github.com/anza-xyz/agave.git
+      - name: Fetch upstream
+        run: git fetch upstream
+      - name: Fetch origin
+        run: git fetch origin
+      - name: Set REBASE_BRANCH
+        run: echo "REBASE_BRANCH=ci/nightly/${{ matrix.branch }}/$(date +'%Y-%m-%d-%H-%M')" >> $GITHUB_ENV
+      - name: echo $REBASE_BRANCH
+        run: echo $REBASE_BRANCH
+      - name: Create rebase branch
+        run: git checkout -b $REBASE_BRANCH
+      - name: Setup email
+        run: |
+          git config --global user.email "[email protected]"
+          git config --global user.name "Jito Infrastructure"
+      - name: Rebase
+        id: rebase
+        run: git rebase ${{ matrix.rebase }}
+      - name: Send warning for rebase error
+        if: failure() && steps.rebase.outcome == 'failure'
+        uses: slackapi/[email protected]
+        with:
+          payload: |
+            {
+              "text": "Nightly rebase on branch ${{ matrix.branch }}\nStatus: Rebase failed to apply cleanly"
+            }
+        env:
+          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+      - name: Check if rebase applied
+        id: check_rebase_applied
+        run: |
+          PRE_REBASE_SHA=$(git rev-parse ${{ matrix.branch }})
+          POST_REBASE_SHA=$(git rev-parse HEAD)
+          if [ "$PRE_REBASE_SHA" = "$POST_REBASE_SHA" ]; then
+            echo "No rebase was applied, exiting..."
+            exit 1
+          else
+            echo "Rebase applied successfully."
+          fi
+      - name: Send warning for rebase error
+        if: failure() && steps.check_rebase_applied.outcome == 'failure'
+        uses: slackapi/[email protected]
+        with:
+          payload: |
+            {
+              "text": "Nightly rebase on branch ${{ matrix.branch }}\nStatus: Rebase not needed"
+            }
+        env:
+          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+      - name: Set REBASE_SHA
+        run: echo "REBASE_SHA=$(git rev-parse HEAD)" >> $GITHUB_ENV
+      - name: Push changes
+        uses: ad-m/github-push-action@master
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          branch: ${{ env.REBASE_BRANCH }}
+      - name: Wait for buildkite to start build
+        run: sleep 300
+      - name: Wait for buildkite to finish
+        id: wait_for_buildkite
+        timeout-minutes: 300
+        run: |
+          while true; do
+            response=$(curl -s -f -H "Authorization: Bearer ${{ secrets.BUILDKITE_TOKEN }}" "https://api.buildkite.com/v2/organizations/jito/pipelines/jito-solana/builds?commit=${{ env.REBASE_SHA }}")
+            if [ $? -ne 0 ]; then
+              echo "Curl request failed."
+              exit 1
+            fi
+
+            state=$(echo $response | jq --exit-status -r '.[0].state')
+            echo "Current build state: $state"
+
+            # Check if the state is one of the finished states
+            case $state in
+              "passed"|"finished")
+                echo "Build finished successfully."
+                exit 0
+              ;;
+              "canceled"|"canceling"|"not_run")
+                # ignoring "failing"|"failed" because flaky CI, can restart and hope it finishes or times out
+                echo "Build failed or was cancelled."
+                exit 2
+              ;;
+            esac
+
+          sleep 30
+          done
+      - name: Send failure update
+        uses: slackapi/[email protected]
+        if: failure() && steps.wait_for_buildkite.outcome == 'failure'
+        with:
+          payload: |
+            {
+              "text": "Nightly rebase on branch ${{ matrix.branch }}\nStatus: CI failed\nBranch: ${{ env.REBASE_BRANCH}}\nBuild: https://buildkite.com/jito/jito-solana/builds?commit=${{ env.REBASE_SHA }}"
+            }
+        env:
+          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+      # check to see if different branch since CI build can take awhile and these steps are not atomic
+      - name: Fetch the latest remote changes
+        run: git fetch origin ${{ matrix.branch }}
+      - name: Check if origin HEAD has changed from the beginning of the workflow
+        run: |
+          LOCAL_SHA=$(git rev-parse ${{ matrix.branch }})
+          ORIGIN_SHA=$(git rev-parse origin/${{ matrix.branch }})
+          if [ "$ORIGIN_SHA" != "$LOCAL_SHA" ]; then
+            echo "The remote HEAD of ${{ matrix.branch }} does not match the local HEAD of ${{ matrix.branch }} at the beginning of CI."
+            echo "origin sha: $ORIGIN_SHA"
+            echo "local sha: $LOCAL_SHA"
+            exit 1
+          else
+            echo "The remote HEAD matches the local REBASE_SHA at the beginning of CI. Proceeding."
+          fi
+      - name: Reset ${{ matrix.branch }} to ${{ env.REBASE_BRANCH }}
+        run: |
+          git checkout ${{ matrix.branch }}
+          git reset --hard ${{ env.REBASE_BRANCH }}
+      - name: Push rebased %{{ matrix.branch }}
+        uses: ad-m/github-push-action@master
+        with:
+          github_token: ${{ secrets.JITO_SOLANA_RELEASE_TOKEN }}
+          branch: ${{ matrix.branch }}
+          force: true
+      - name: Send success update
+        uses: slackapi/[email protected]
+        with:
+          payload: |
+            {
+              "text": "Nightly rebase on branch ${{ matrix.branch }}\nStatus: CI success, rebased, and pushed\nBranch: ${{ env.REBASE_BRANCH}}\nBuild: https://buildkite.com/jito/jito-solana/builds?commit=${{ env.REBASE_SHA }}"
+            }
+        env:
+          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
diff --git a/.github/workflows/release-artifacts.yml b/.github/workflows/release-artifacts.yml
@@ -22,6 +22,7 @@ jobs:
         with:
           ref: master
           fetch-depth: 0
+          submodules: 'recursive'
 
       - name: Setup Rust
         shell: bash

diff --git a/.gitignore b/.gitignore
@@ -4,6 +4,7 @@ target/
 /solana-release.tar.bz2
 /solana-metrics/
 /solana-metrics.tar.bz2
+**/target/
 /test-ledger/
 
 **/*.rs.bk
@@ -27,7 +28,11 @@ log-*/
 # fetch-spl.sh artifacts
 /spl-genesis-args.sh
 /spl_*.so
+/jito_*.so
 
 .DS_Store
 # scripts that may be generated by cargo *-bpf commands
 **/cargo-*-bpf-child-script-*.sh
+
+.env
+docker-output/
diff --git a/.gitmodules b/.gitmodules
@@ -0,0 +1,9 @@
+[submodule "anchor"]
+	path = anchor
+	url = https://github.com/jito-foundation/anchor.git
+[submodule "jito-programs"]
+	path = jito-programs
+	url = https://github.com/jito-foundation/jito-programs.git
+[submodule "jito-protos/protos"]
+	path = jito-protos/protos
+	url = https://github.com/jito-labs/mev-protos.git