change(consensus): Build Sprout and Sapling parameters into the zebra…

…d binary, so a download server isn't needed (#7800) * Build sapling parameters into zebrad * Load the sapling parameters from wagyu-zcash-parameters * Reword logs * wget https://github.com/zcash/zcash/raw/master/src/rust/src/sprout-groth16.vk * Load sprout from binary and remove download code * Remove download examples binaries * Remove unused features and dependencies * Remove docs and comments about downloading parameters * Add CHANGELOG entry * Remove parameter cache steps in workflows * Update comment about downloading Co-authored-by: Marek <[email protected]> * Fix outdated timeout comments and short timeouts --------- Co-authored-by: Marek <[email protected]>
ZcashFoundation · Oct 24, 2023 · aad883b · aad883b
1 parent bb7eeb6
commit aad883b
Show file tree

Hide file tree

Showing 17 changed files with 121 additions and 365 deletions.
diff --git a/.github/workflows/ci-coverage.patch.yml b/.github/workflows/ci-coverage.patch.yml
@@ -17,10 +17,6 @@ on:
 jobs:
   coverage:
     name: Coverage on stable
-    # The large timeout is to accommodate:
-    # - nightly builds (75 minutes, typically 30-50 minutes)
-    # - parameter downloads (40 minutes, but only when the cache expires)
-    timeout-minutes: 115
     runs-on: ubuntu-latest
 
     steps:

diff --git a/.github/workflows/ci-coverage.yml b/.github/workflows/ci-coverage.yml
@@ -9,6 +9,10 @@ concurrency:
 
 on:
   workflow_dispatch:
+
+  # we build Rust caches on main,
+  # so they can be shared by all branches:
+  # https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache
   push:
     branches:
       - main
@@ -28,6 +32,7 @@ on:
       # workflow definitions
       - 'codecov.yml'
       - '.github/workflows/ci-coverage.yml'
+
   pull_request:
     paths:
       - '**/*.rs'
@@ -51,9 +56,8 @@ jobs:
   coverage:
     name: Coverage on stable
     # The large timeout is to accommodate:
-    # - stable builds (typically 30-50 minutes), and
-    # - parameter downloads (an extra 90 minutes, but only when the cache expires)
-    timeout-minutes: 140
+    # - stable builds (typically 50-90 minutes), and
+    timeout-minutes: 120
     runs-on: ubuntu-latest-xl
 
     steps:
@@ -84,24 +88,6 @@ jobs:
           echo "PROPTEST_CASES=1" >> $GITHUB_ENV
           echo "PROPTEST_MAX_SHRINK_ITERS=0" >> $GITHUB_ENV
 
-      # Modified from:
-      # https://github.com/zcash/librustzcash/blob/c48bb4def2e122289843ddb3cb2984c325c03ca0/.github/workflows/ci.yml#L20-L33
-      - name: Fetch path to Zcash parameters
-        working-directory: ./zebra-consensus
-        shell: bash
-        # cargo-llvm-cov doesn't have a silent mode, so we have to extract the path from stderr
-        run: echo "ZCASH_PARAMS=$(cargo llvm-cov --lcov --no-report run --example get-params-path 2>&1 >/dev/null | tail -1)" >> $GITHUB_ENV
-      - name: Cache Zcash parameters
-        id: cache-params
-        uses: actions/cache@v3
-        with:
-          path: ${{ env.ZCASH_PARAMS }}
-          key: ${{ runner.os }}-sprout-and-sapling-params
-      - name: Fetch Zcash parameters
-        if: steps.cache-params.outputs.cache-hit != 'true'
-        working-directory: ./zebra-consensus
-        run: cargo llvm-cov --lcov --no-report run --example download-params
-
       - name: Run Zebra tests
         run: cargo llvm-cov --lcov --no-report
 

diff --git a/.github/workflows/ci-unit-tests-os.yml b/.github/workflows/ci-unit-tests-os.yml
@@ -9,9 +9,6 @@ concurrency:
 
 on:
   workflow_dispatch:
-  # we build Rust and Zcash parameter caches on main,
-  # so they can be shared by all branches:
-  # https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache
 
   pull_request:
     paths:
@@ -31,6 +28,9 @@ on:
       # workflow definitions
       - '.github/workflows/ci-unit-tests-os.yml'
 
+  # we build Rust caches on main,
+  # so they can be shared by all branches:
+  # https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache
   push:
     branches:
       - main
@@ -67,9 +67,8 @@ jobs:
   test:
     name: Test ${{ matrix.rust }} on ${{ matrix.os }}${{ matrix.features }}
     # The large timeout is to accommodate:
-    # - macOS and Windows builds (90 minutes, typically 30-70 minutes), and
-    # - parameter downloads (an extra 100 minutes, but only when the cache expires)
-    timeout-minutes: 190
+    # - macOS and Windows builds (typically 50-90 minutes), and
+    timeout-minutes: 120
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
@@ -159,29 +158,6 @@ jobs:
           echo "PROPTEST_CASES=1" >> $GITHUB_ENV
           echo "PROPTEST_MAX_SHRINK_ITERS=1024" >> $GITHUB_ENV
 
-      # Modified from:
-      # https://github.com/zcash/librustzcash/blob/c48bb4def2e122289843ddb3cb2984c325c03ca0/.github/workflows/ci.yml#L20-L33
-      #
-      # TODO:
-      # - split Fetch/Cache Zcash parameters into their own job,
-      #   and use `concurrency:` to limit it to one job per OS
-      # - split get-params-path and download-params examples into their own crate,
-      #   to speed up compilation
-      - name: Fetch path to Zcash parameters
-        working-directory: ./zebra-consensus
-        shell: bash
-        run: echo "ZCASH_PARAMS=$(cargo run --release --example get-params-path)" >> $GITHUB_ENV
-      - name: Cache Zcash parameters
-        id: cache-params
-        uses: actions/cache@v3
-        with:
-          path: ${{ env.ZCASH_PARAMS }}
-          key: ${{ runner.os }}-sprout-and-sapling-params
-      - name: Fetch Zcash parameters
-        if: steps.cache-params.outputs.cache-hit != 'true'
-        working-directory: ./zebra-consensus
-        run: cargo run --release --example download-params
-
       # Run unit and basic acceptance tests, only showing command output if the test fails.
       #
       # If some tests hang, add "-- --nocapture" for just that test, or for all the tests.

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -10,7 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org).
 Zebra's mining RPCs are now available in release builds. TODO: rest of intro
 
 This release contains the following changes:
- 
+
 ### Mining RPCs in Production Builds
 
 Zebra's mining RPCs are now available in release builds (#7740). Any Zebra instance can be used
@@ -21,6 +21,25 @@ read our [mining blog post](https://zfnd.org/experimental-mining-support-in-zebr
 Please [let us know](https://github.com/ZcashFoundation/zebra/issues/new?assignees=&labels=C-enhancement%2CS-needs-triage&projects=&template=feature_request.yml&title=feature%3A+)
 if your mining pool needs extra RPC methods or fields.
 
+### Parameters in Binary
+
+`zebrad` now bundles zk-SNARK parameters directly into its binary. This increases the binary size
+by a few megabytes, but these parameters do not need to be downloaded or stored separately.
+
+Previously, parameters were stored by default in these locations:
+
+* `~/.zcash-params` (on Linux); or
+* `~/Library/Application Support/ZcashParams` (on Mac); or
+* `C:\Users\Username\AppData\Roaming\ZcashParams` (on Windows)
+
+If you have upgraded `zebrad` to 1.4.0 or later, and `zcashd` to 5.7.0 or later, you can delete the
+parameter files in these directories to save approximately 700 MB disk space.
+
+[`zcashd` have deprecated their `fetch-params.sh` script](https://github.com/zcash/zcash/blob/master/doc/release-notes/release-notes-5.7.0.md#deprecation-of-fetch-paramssh),
+so it can't be used to retry failed downloads in `zebrad` 1.3.0 and earlier.
+
+We recommend upgrading to the latest Zebra release to avoid download issues in new installs.
+
 ### Security
 
 TODO: rest of changelog
@@ -32,9 +51,9 @@ and fixes performance issues and bugs in the mining solution rate RPCs. Progress
 bars can now be enabled using a config, please help us test them!
 
 It contains the following updates:
- 
+
 ### User Testing: Progress Bars
- 
+
 Zebra has progress bars! When progress bars are enabled, you can see Zebra's blocks,
 transactions, and peer connections in your terminal. We're asking Zebra users to test this
 feature, and give us [feedback on the forums](https://forum.zcashcommunity.com/t/zebra-progress-bars/44485).

diff --git a/Cargo.lock b/Cargo.lock
@@ -2486,19 +2486,6 @@ dependencies = [
  "adler",
 ]
 
-[[package]]
-name = "minreq"
-version = "2.11.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cb3371dfc7b772c540da1380123674a8e20583aca99907087d990ca58cf44203"
-dependencies = [
- "log",
- "once_cell",
- "rustls",
- "rustls-webpki",
- "webpki-roots",
-]
-
 [[package]]
 name = "mio"
 version = "0.8.8"
@@ -4916,6 +4903,56 @@ version = "1.0.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "6a02e4885ed3bc0f2de90ea6dd45ebcbb66dacffe03547fadbb0eeae2770887d"
 
+[[package]]
+name = "wagyu-zcash-parameters"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "61c904628658374e651288f000934c33ef738b2d8b3e65d4100b70b395dbe2bb"
+dependencies = [
+ "wagyu-zcash-parameters-1",
+ "wagyu-zcash-parameters-2",
+ "wagyu-zcash-parameters-3",
+ "wagyu-zcash-parameters-4",
+ "wagyu-zcash-parameters-5",
+ "wagyu-zcash-parameters-6",
+]
+
+[[package]]
+name = "wagyu-zcash-parameters-1"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "90bf2e21bb027d3f8428c60d6a720b54a08bf6ce4e6f834ef8e0d38bb5695da8"
+
+[[package]]
+name = "wagyu-zcash-parameters-2"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a616ab2e51e74cc48995d476e94de810fb16fc73815f390bf2941b046cc9ba2c"
+
+[[package]]
+name = "wagyu-zcash-parameters-3"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "14da1e2e958ff93c0830ee68e91884069253bf3462a67831b02b367be75d6147"
+
+[[package]]
+name = "wagyu-zcash-parameters-4"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f058aeef03a2070e8666ffb5d1057d8bb10313b204a254a6e6103eb958e9a6d6"
+
+[[package]]
+name = "wagyu-zcash-parameters-5"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3ffe916b30e608c032ae1b734f02574a3e12ec19ab5cc5562208d679efe4969d"
+
+[[package]]
+name = "wagyu-zcash-parameters-6"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a7b6d5a78adc3e8f198e9cd730f219a695431467f7ec29dcfc63ade885feebe1"
+
 [[package]]
 name = "wait-timeout"
 version = "0.2.0"
@@ -5373,7 +5410,6 @@ dependencies = [
  "jubjub",
  "known-folders",
  "lazy_static",
- "minreq",
  "rand_core 0.6.4",
  "redjubjub",
  "tracing",
@@ -5516,6 +5552,7 @@ dependencies = [
  "tracing-error",
  "tracing-futures",
  "tracing-subscriber",
+ "wagyu-zcash-parameters",
  "zcash_proofs",
  "zebra-chain",
  "zebra-node-services",

diff --git a/Cargo.toml b/Cargo.toml
@@ -47,17 +47,9 @@ opt-level = 3
 [profile.dev.package.bls12_381]
 opt-level = 3
 
-# Cryptographic and parameter download crates
-
 [profile.dev.package.zcash_proofs]
 opt-level = 3
 
-[profile.dev.package.minreq]
-opt-level = 3
-
-[profile.dev.package.rustls]
-opt-level = 3
-
 [profile.dev.package.ring]
 opt-level = 3
 

diff --git a/README.md b/README.md
@@ -163,8 +163,6 @@ There are a few bugs in Zebra that we're still working on fixing:
 
 - Zebra currently gossips and connects to [private IP addresses](https://en.wikipedia.org/wiki/IP_address#Private_addresses), we want to [disable private IPs but provide a config (#3117)](https://github.com/ZcashFoundation/zebra/issues/3117) in an upcoming release
 
-- If Zebra fails downloading the Zcash parameters, use [the Zcash parameters download script](https://github.com/zcash/zcash/blob/master/zcutil/fetch-params.sh) instead.
-
 - Block download and verification sometimes times out during Zebra's initial sync [#5709](https://github.com/ZcashFoundation/zebra/issues/5709). The full sync still finishes reasonably quickly.
 
 - Rust 1.70 [causes crashes during shutdown on macOS x86_64 (#6812)](https://github.com/ZcashFoundation/zebra/issues/6812). The state cache should stay valid despite the crash.

diff --git a/book/src/dev/continuous-integration.md b/book/src/dev/continuous-integration.md
@@ -178,14 +178,13 @@ To fix duplicate dependencies, follow these steps until the duplicate dependenci
 
 2. If there are any "skip tree root was not found in the dependency graph" warnings, delete those versions from `deny.toml`
 
-### Fixing Disk Full Errors and Zcash Parameter Errors
+### Fixing Disk Full Errors
 
 If the Docker cached state disks are full, increase the disk sizes in:
 - [deploy-gcp-tests.yml](https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/deploy-gcp-tests.yml)
 - [continous-delivery.yml](https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/continous-delivery.yml)
 
-If the GitHub Actions disks are full, or the Zcash parameter downloads time out without any network messages or errors,
-follow these steps until the errors are fixed:
+If the GitHub Actions disks are full, follow these steps until the errors are fixed:
 
 0. Check if error is also happening on the `main` branch. If it is, skip the next step.
 1. Update your branch to the latest `main` branch, this builds with all the latest dependencies in the `main` branch cache.
@@ -194,11 +193,6 @@ follow these steps until the errors are fixed:
 
 These errors often happen after a new compiler version is released, because the caches can end up with files from both compiler versions.
 
-If the Zcash Parameter downloads have an error loading the parameters:
-1. Clear the Zcash parameter caches for all branches, including `main`
-
-The correct `*-sprout-and-sapling-params` caches should be around 765 MB.
-
 You can find a list of caches using:
 ```sh
 gh api -H "Accept: application/vnd.github+json" repos/ZcashFoundation/Zebra/actions/caches