Merge branch 'main' into get-block

.github/ISSUE_TEMPLATE/release.md

@@ -16,7 +16,7 @@ They can be skipped for urgent releases.
 
 To check consensus correctness, we want to test that the state format is valid after a full sync. (Format upgrades are tested in CI on each PR.)
 
-- [ ] Make sure there has been [at least one successful full sync test](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-integration-tests-gcp.yml?query=event%3Aschedule) since the last state change, or
+- [ ] Make sure there has been [at least one successful full sync test](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-tests.yml?query=event%3Aschedule) since the last state change, or
 - [ ] Start a manual workflow run with a Zebra and `lightwalletd` full sync.
 
 State format changes can be made in `zebra-state` or `zebra-chain`. The state format can be changed by data that is sent to the state, data created within the state using `zebra-chain`, or serialization formats in `zebra-state` or `zebra-chain`. 

.github/PULL_REQUEST_TEMPLATE/release-checklist.md

@@ -9,7 +9,7 @@ assignees: ''
 
 # Prepare for the Release
 
-- [ ] Make sure there has been [at least one successful full sync test](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-integration-tests-gcp.yml?query=event%3Aschedule) since the last state change, or start a manual full sync.
+- [ ] Make sure there has been [at least one successful full sync test in the main branch](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-tests.yml?query=branch%3Amain) since the last state change, or start a manual full sync.
 - [ ] Make sure the PRs with the new checkpoint hashes and missed dependencies are already merged.
       (See the release ticket checklist for details)
 
@@ -57,7 +57,13 @@ fastmod --fixed-strings '1.58' '1.65'
 - [ ] Freeze the [`batched` queue](https://dashboard.mergify.com/github/ZcashFoundation/repo/zebra/queues) using Mergify.
 - [ ] Mark all the release PRs as `Critical` priority, so they go in the `urgent` Mergify queue.
 - [ ] Mark all non-release PRs with `do-not-merge`, because Mergify checks approved PRs against every commit, even when a queue is frozen.
+- [ ] Add the `A-release` tag to the release pull request in order for the `check_no_git_refs_in_cargo_lock` to run.
 
+## Zebra git sources dependencies
+
+- [ ] Ensure the `check_no_git_refs_in_cargo_lock` check passes.
+
+This check runs automatically on pull requests with the `A-release` label. It must pass for crates to be published to crates.io. If the check fails, you should either halt the release process or proceed with the understanding that the crates will not be published on crates.io.
 
 # Update Versions and End of Support
 
@@ -76,7 +82,7 @@ Zebra's Rust API doesn't have any support or stability guarantees, so we keep al
 
 ### Update Crate Versions
 
-If you're publishing crates for the first time, [log in to crates.io](https://github.com/ZcashFoundation/zebra/blob/doc-crate-own/book/src/dev/crate-owners.md#logging-in-to-cratesio),
+If you're publishing crates for the first time, [log in to crates.io](https://zebra.zfnd.org/dev/crate-owners.html#logging-in-to-cratesio),
 and make sure you're a member of owners group.
 
 Check that the release will work:
@@ -103,7 +109,7 @@ Crate publishing is [automatically checked in CI](https://github.com/ZcashFounda
 ## Update End of Support
 
 The end of support height is calculated from the current blockchain height:
-- [ ] Find where the Zcash blockchain tip is now by using a [Zcash explorer](https://zcashblockexplorer.com/blocks) or other tool.
+- [ ] Find where the Zcash blockchain tip is now by using a [Zcash Block Explorer](https://mainnet.zcashexplorer.app/) or other tool.
 - [ ] Replace `ESTIMATED_RELEASE_HEIGHT` in [`end_of_support.rs`](https://github.com/ZcashFoundation/zebra/blob/main/zebrad/src/components/sync/end_of_support.rs) with the height you estimate the release will be tagged.
 
 <details>
@@ -141,8 +147,7 @@ The end of support height is calculated from the current blockchain height:
 ## Test the Pre-Release
 
 - [ ] Wait until the Docker binaries have been built on `main`, and the quick tests have passed:
-    - [ ] [ci-unit-tests-docker.yml](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-unit-tests-docker.yml?query=branch%3Amain)
-    - [ ] [ci-integration-tests-gcp.yml](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-integration-tests-gcp.yml?query=branch%3Amain)
+    - [ ] [ci-tests.yml](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-tests.yml?query=branch%3Amain)
 - [ ] Wait until the [pre-release deployment machines have successfully launched](https://github.com/ZcashFoundation/zebra/actions/workflows/cd-deploy-nodes-gcp.yml?query=event%3Arelease)
 
 ## Publish Release
@@ -151,15 +156,17 @@ The end of support height is calculated from the current blockchain height:
 
 ## Publish Crates
 
-- [ ] [Run `cargo login`](https://github.com/ZcashFoundation/zebra/blob/doc-crate-own/book/src/dev/crate-owners.md#logging-in-to-cratesio)
+- [ ] [Run `cargo login`](https://zebra.zfnd.org/dev/crate-owners.html#logging-in-to-cratesio)
 - [ ] Run `cargo clean` in the zebra repo (optional)
 - [ ] Publish the crates to crates.io: `cargo release publish --verbose --workspace --execute`
 - [ ] Check that Zebra can be installed from `crates.io`:
       `cargo install --locked --force --version 1.minor.patch zebrad && ~/.cargo/bin/zebrad`
       and put the output in a comment on the PR.
 
 ## Publish Docker Images
+
 - [ ] Wait for the [the Docker images to be published successfully](https://github.com/ZcashFoundation/zebra/actions/workflows/release-binaries.yml?query=event%3Arelease).
+- [ ] Wait for the new tag in the [dockerhub zebra space](https://hub.docker.com/r/zfnd/zebra/tags)
 - [ ] Un-freeze the [`batched` queue](https://dashboard.mergify.com/github/ZcashFoundation/repo/zebra/queues) using Mergify.
 - [ ] Remove `do-not-merge` from the PRs you added it to
 

.github/workflows/ci-tests.yml

@@ -121,7 +121,7 @@ jobs:
   # workflow or repository variable is configured differently. Testnet jobs change that config to
   # testnet when running the image.
   build:
-    name: Build images
+    name: Build CI Docker
     # Skip PRs from external repositories, let them pass, and then GitHub's Merge Queue will check them
     if: ${{ !startsWith(github.event_name, 'pull') || !github.event.pull_request.head.repo.fork }}
     uses: ./.github/workflows/sub-build-docker-image.yml

.github/workflows/sub-ci-integration-tests-gcp.yml

@@ -480,29 +480,6 @@ jobs:
       saves_to_disk: false
     secrets: inherit
 
-  # Test that the scanner can continue scanning where it was left when zebrad restarts.
-  #
-  # Runs:
-  # - after every PR is merged to `main`
-  # - on every PR update
-  #
-  # If the state version has changed, waits for the new cached states to be created.
-  # Otherwise, if the state rebuild was skipped, runs immediately after the build job.
-  scan-start-where-left-test:
-    name: Scan starts where left
-    needs: [test-full-sync, get-available-disks]
-    uses: ./.github/workflows/sub-deploy-integration-tests-gcp.yml
-    if: ${{ !cancelled() && !failure() && (fromJSON(needs.get-available-disks.outputs.zebra_tip_disk) || needs.test-full-sync.result == 'success') && github.event.inputs.regenerate-disks != 'true' && github.event.inputs.run-full-sync != 'true' && github.event.inputs.run-lwd-sync != 'true' }}
-    with:
-      app_name: zebra-scan
-      test_id: scan-start-where-left
-      test_description: Test that the scanner can continue scanning where it was left when zebrad restarts.
-      test_variables: "-e NETWORK=${{ inputs.network || vars.ZCASH_NETWORK }} -e TEST_SCAN_START_WHERE_LEFT=1 -e ZEBRA_FORCE_USE_COLOR=1 -e ZEBRA_CACHED_STATE_DIR=/var/cache/zebrad-cache"
-      needs_zebra_state: true
-      needs_lwd_state: false
-      saves_to_disk: true
-    secrets: inherit
-
   # Test that the scan task registers keys, deletes keys, and subscribes to results for keys while running.
   #
   # Runs:
@@ -546,7 +523,6 @@ jobs:
         lightwalletd-grpc-test,
         get-block-template-test,
         submit-block-test,
-        scan-start-where-left-test,
         scan-task-commands-test,
       ]
     # Only open tickets for failed scheduled jobs, manual workflow runs, or `main` branch merges.

.github/workflows/sub-ci-unit-tests-docker.yml

@@ -138,7 +138,7 @@ jobs:
           NETWORK: ${{ inputs.network || vars.ZCASH_NETWORK }}
         run: |
           docker pull ${{ vars.GAR_BASE }}/${{ vars.CI_IMAGE_NAME }}@${{ inputs.image_digest }}
-          docker run --tty -e NETWORK -e ZEBRA_TEST_LIGHTWALLETD=1 ${{ vars.GAR_BASE }}/${{ vars.CI_IMAGE_NAME }}@${{ inputs.image_digest }}
+          docker run --tty -e NETWORK -e ZEBRA_TEST_LIGHTWALLETD=1 -e TEST_LWD_INTEGRATION=1 ${{ vars.GAR_BASE }}/${{ vars.CI_IMAGE_NAME }}@${{ inputs.image_digest }}
 
   # Test that Zebra works using the default config with the latest Zebra version.
   test-configuration-file:
@@ -197,3 +197,12 @@ jobs:
           # If there is already an open issue with this label, any failures become comments on that issue.
           always-create-new-issue: false
           github-token: ${{ secrets.GITHUB_TOKEN }}
+
+  run-check-no-git-refs:
+    if: contains(github.event.pull_request.labels.*.name, 'A-release')
+    runs-on: ubuntu-latest
+    steps:
+    - name: Run check_no_git_refs_in_cargo_lock
+      run: |
+        docker pull ${{ vars.GAR_BASE }}/${{ vars.CI_IMAGE_NAME }}@${{ inputs.image_digest }}
+        docker run --tty -e NETWORK -e RUN_CHECK_NO_GIT_REFS=1 ${{ vars.GAR_BASE }}/${{ vars.CI_IMAGE_NAME }}@${{ inputs.image_digest }}

book/src/SUMMARY.md

@@ -37,6 +37,7 @@
   - [Doing Mass Renames](dev/mass-renames.md)
   - [Updating the ECC dependencies](dev/ecc-updates.md)
   - [Running a Private Testnet Test](dev/private-testnet.md)
+  - [Zebra crates](dev/crate-owners.md)
   - [Zebra RFCs](dev/rfcs.md)
     - [Pipelinable Block Lookup](dev/rfcs/0001-pipelinable-block-lookup.md)
     - [Parallel Verification](dev/rfcs/0002-parallel-verification.md)

docker/entrypoint.sh

@@ -59,7 +59,7 @@ fi
 : "${RUN_ALL_EXPERIMENTAL_TESTS:=}"
 : "${TEST_FAKE_ACTIVATION_HEIGHTS:=}"
 : "${TEST_ZEBRA_EMPTY_SYNC:=}"
-: "${ZEBRA_TEST_LIGHTWALLETD:=}"
+: "${TEST_LWD_INTEGRATION:=}"
 : "${FULL_SYNC_MAINNET_TIMEOUT_MINUTES:=}"
 : "${FULL_SYNC_TESTNET_TIMEOUT_MINUTES:=}"
 : "${TEST_DISK_REBUILD:=}"
@@ -74,7 +74,6 @@ fi
 : "${TEST_LWD_TRANSACTIONS:=}"
 : "${TEST_GET_BLOCK_TEMPLATE:=}"
 : "${TEST_SUBMIT_BLOCK:=}"
-: "${TEST_SCAN_START_WHERE_LEFT:=}"
 : "${ENTRYPOINT_FEATURES:=}"
 : "${TEST_SCAN_TASK_COMMANDS:=}"
 
@@ -225,12 +224,16 @@ case "$1" in
       if [[ "${RUN_ALL_TESTS}" -eq "1" ]]; then
         # Run unit, basic acceptance tests, and ignored tests, only showing command output if the test fails.
         # If the lightwalletd environmental variables are set, we will also run those tests.
-        exec cargo test --locked --release --features "${ENTRYPOINT_FEATURES}" --workspace -- --nocapture --include-ignored
+        exec cargo test --locked --release --features "${ENTRYPOINT_FEATURES}" --workspace -- --nocapture --include-ignored --skip check_no_git_refs_in_cargo_lock
 
       elif [[ "${RUN_ALL_EXPERIMENTAL_TESTS}" -eq "1" ]]; then
         # Run unit, basic acceptance tests, and ignored tests with experimental features.
         # If the lightwalletd environmental variables are set, we will also run those tests.
-        exec cargo test --locked --release --features "${ENTRYPOINT_FEATURES_EXPERIMENTAL}" --workspace -- --nocapture --include-ignored
+        exec cargo test --locked --release --features "${ENTRYPOINT_FEATURES_EXPERIMENTAL}" --workspace -- --nocapture --include-ignored --skip check_no_git_refs_in_cargo_lock
+
+      elif [[ "${RUN_CHECK_NO_GIT_REFS}" -eq "1" ]]; then
+        # Run the check_no_git_refs_in_cargo_lock test.
+        exec cargo test --locked --release --features "${ENTRYPOINT_FEATURES}" --workspace -- --nocapture --include-ignored check_no_git_refs_in_cargo_lock
 
       elif [[ "${TEST_FAKE_ACTIVATION_HEIGHTS}" -eq "1" ]]; then
         # Run state tests with fake activation heights.
@@ -240,10 +243,6 @@ case "$1" in
         # Test that Zebra syncs and checkpoints a few thousand blocks from an empty state.
         run_cargo_test "${ENTRYPOINT_FEATURES}" "sync_large_checkpoints_"
 
-      elif [[ "${ZEBRA_TEST_LIGHTWALLETD}" -eq "1" ]]; then
-        # Test launching lightwalletd with an empty lightwalletd and Zebra state.
-        run_cargo_test "${ENTRYPOINT_FEATURES}" "lightwalletd_integration"
-
       elif [[ -n "${FULL_SYNC_MAINNET_TIMEOUT_MINUTES}" ]]; then
         # Run a Zebra full sync test on mainnet.
         run_cargo_test "${ENTRYPOINT_FEATURES}" "full_sync_mainnet"
@@ -304,6 +303,10 @@ case "$1" in
         # Since these tests use the same cached state, a state problem in the first test can fail the second test.
         run_cargo_test "${ENTRYPOINT_FEATURES}" "--test-threads" "1" "fully_synced_rpc_"
 
+      elif [[ "${TEST_LWD_INTEGRATION}" -eq "1" ]]; then
+        # Test launching lightwalletd with an empty lightwalletd and Zebra state.
+        run_cargo_test "${ENTRYPOINT_FEATURES}" "lightwalletd_integration"
+
       elif [[ "${TEST_LWD_FULL_SYNC}" -eq "1" ]]; then
         # Starting at a cached Zebra tip, run a lightwalletd sync to tip.
         check_directory_files "${ZEBRA_CACHED_STATE_DIR}"
@@ -340,11 +343,6 @@ case "$1" in
         check_directory_files "${ZEBRA_CACHED_STATE_DIR}"
         run_cargo_test "${ENTRYPOINT_FEATURES}" "submit_block"
 
-      elif [[ "${TEST_SCAN_START_WHERE_LEFT}" -eq "1" ]]; then
-        # Test that the scanner can continue scanning where it was left when zebra-scanner restarts.
-        check_directory_files "${ZEBRA_CACHED_STATE_DIR}"
-        exec cargo test --locked --release --features "zebra-test" --package zebra-scan -- --nocapture --include-ignored scan_start_where_left
-
       elif [[ "${TEST_SCAN_TASK_COMMANDS}" -eq "1" ]]; then
         # Test that the scan task commands are working.
         check_directory_files "${ZEBRA_CACHED_STATE_DIR}"

zebra-network/src/constants.rs

@@ -396,14 +396,14 @@ lazy_static! {
     ///
     /// The minimum network protocol version typically changes after Mainnet and
     /// Testnet network upgrades.
-    // TODO: Change `Nu5` to `Nu6` after NU6 activation.
+    // TODO: Change `Nu6` to `Nu7` after NU7 activation.
     // TODO: Move the value here to a field on `testnet::Parameters` (#8367)
     pub static ref INITIAL_MIN_NETWORK_PROTOCOL_VERSION: HashMap<NetworkKind, Version> = {
         let mut hash_map = HashMap::new();
 
-        hash_map.insert(NetworkKind::Mainnet, Version::min_specified_for_upgrade(&Mainnet, Nu5));
-        hash_map.insert(NetworkKind::Testnet, Version::min_specified_for_upgrade(&Network::new_default_testnet(), Nu5));
-        hash_map.insert(NetworkKind::Regtest, Version::min_specified_for_upgrade(&Network::new_regtest(None, None), Nu5));
+        hash_map.insert(NetworkKind::Mainnet, Version::min_specified_for_upgrade(&Mainnet, Nu6));
+        hash_map.insert(NetworkKind::Testnet, Version::min_specified_for_upgrade(&Network::new_default_testnet(), Nu6));
+        hash_map.insert(NetworkKind::Regtest, Version::min_specified_for_upgrade(&Network::new_regtest(None, None), Nu6));
 
         hash_map
     };

zebra-network/src/peer_set/set/tests/vectors.rs

@@ -26,7 +26,7 @@ fn peer_set_ready_single_connection() {
     let peer_versions = PeerVersions {
         peer_versions: vec![Version::min_specified_for_upgrade(
             &Network::Mainnet,
-            NetworkUpgrade::Nu5,
+            NetworkUpgrade::Nu6,
         )],
     };
 
@@ -118,7 +118,7 @@ fn peer_set_ready_single_connection() {
 #[test]
 fn peer_set_ready_multiple_connections() {
     // Use three peers with the same version
-    let peer_version = Version::min_specified_for_upgrade(&Network::Mainnet, NetworkUpgrade::Nu5);
+    let peer_version = Version::min_specified_for_upgrade(&Network::Mainnet, NetworkUpgrade::Nu6);
     let peer_versions = PeerVersions {
         peer_versions: vec![peer_version, peer_version, peer_version],
     };
@@ -182,7 +182,7 @@ fn peer_set_rejects_connections_past_per_ip_limit() {
     const NUM_PEER_VERSIONS: usize = crate::constants::DEFAULT_MAX_CONNS_PER_IP + 1;
 
     // Use three peers with the same version
-    let peer_version = Version::min_specified_for_upgrade(&Network::Mainnet, NetworkUpgrade::Nu5);
+    let peer_version = Version::min_specified_for_upgrade(&Network::Mainnet, NetworkUpgrade::Nu6);
     let peer_versions = PeerVersions {
         peer_versions: [peer_version; NUM_PEER_VERSIONS].into_iter().collect(),
     };
@@ -232,7 +232,7 @@ fn peer_set_route_inv_empty_registry() {
     let test_hash = block::Hash([0; 32]);
 
     // Use two peers with the same version
-    let peer_version = Version::min_specified_for_upgrade(&Network::Mainnet, NetworkUpgrade::Nu5);
+    let peer_version = Version::min_specified_for_upgrade(&Network::Mainnet, NetworkUpgrade::Nu6);
     let peer_versions = PeerVersions {
         peer_versions: vec![peer_version, peer_version],
     };
@@ -315,7 +315,7 @@ fn peer_set_route_inv_advertised_registry_order(advertised_first: bool) {
     let test_change = InventoryStatus::new_available(test_inv, test_peer);
 
     // Use two peers with the same version
-    let peer_version = Version::min_specified_for_upgrade(&Network::Mainnet, NetworkUpgrade::Nu5);
+    let peer_version = Version::min_specified_for_upgrade(&Network::Mainnet, NetworkUpgrade::Nu6);
     let peer_versions = PeerVersions {
         peer_versions: vec![peer_version, peer_version],
     };
@@ -423,7 +423,7 @@ fn peer_set_route_inv_missing_registry_order(missing_first: bool) {
     let test_change = InventoryStatus::new_missing(test_inv, test_peer);
 
     // Use two peers with the same version
-    let peer_version = Version::min_specified_for_upgrade(&Network::Mainnet, NetworkUpgrade::Nu5);
+    let peer_version = Version::min_specified_for_upgrade(&Network::Mainnet, NetworkUpgrade::Nu6);
     let peer_versions = PeerVersions {
         peer_versions: vec![peer_version, peer_version],
     };
@@ -525,7 +525,7 @@ fn peer_set_route_inv_all_missing_fail() {
     let test_change = InventoryStatus::new_missing(test_inv, test_peer);
 
     // Use one peer
-    let peer_version = Version::min_specified_for_upgrade(&Network::Mainnet, NetworkUpgrade::Nu5);
+    let peer_version = Version::min_specified_for_upgrade(&Network::Mainnet, NetworkUpgrade::Nu6);
     let peer_versions = PeerVersions {
         peer_versions: vec![peer_version],
     };

zebra-test/src/vectors/sapling-treestate-main-0-419-201.txt

@@ -1 +1 @@
-019eb30778ddeea84c72e69e07a1689f3c8def3dc0a1939f0edcbe47279069d9310002000150715810d52caf35471d10feb487213fbd95ff209122225b7b65d27a7fb1a44d
+019eb30778ddeea84c72e69e07a1689f3c8def3dc0a1939f0edcbe47279069d931001f000150715810d52caf35471d10feb487213fbd95ff209122225b7b65d27a7fb1a44d0000000000000000000000000000000000000000000000000000000000

zebrad/tests/acceptance.rs

@@ -3538,3 +3538,15 @@ async fn nu6_funding_streams_and_coinbase_balance() -> Result<()> {
 
     Ok(())
 }
+
+/// Check that Zebra does not depend on any crates from git sources.
+#[test]
+#[ignore]
+fn check_no_git_refs_in_cargo_lock() {
+    let cargo_lock_contents =
+        fs::read_to_string("../Cargo.lock").expect("should have Cargo.lock file in root dir");
+
+    if cargo_lock_contents.contains(r#"source = "git+"#) {
+        panic!("Cargo.lock includes git sources")
+    }
+}