Release Testnet 76, via chain upgrade

[conorsch]

Testnet upgrade

Testnet chain id: penumbra-testnet-deimos-8
Release date: 2024-05-22
Testnet release manager: @conorsch

We're preparing another chain upgrade, explicitly to exercise the mechanics of migrations and coordination, and implicitly to ship a few changes.

Testnet Release Manager Checklist

Pre-release:

• Review all merged PRs with tags consensus-breaking and/or state-breaking
• Enumerate the specific PRs that should be included in the release:
  • governance: replace the halt counter with a halt_bit #4373
  • FMD Parameter Selection Breaking Changes #4137
  • auction: credit VCB when ending an auction #4394
  • final changes to circuits #4371
  • Arb execution does not update value circuit breaker #4113
  • Implement missing IBC APIs for Skip #4391
  • tonic: 🍸 patch tonic dependencies #4424
  • ibc: audit fixes #4386
• Confirm migrations exist where required for breaking changes already merged
• Identify specific test cases for before/after behavior, querying chain state, to ensure that migrations are effective
• Lower voting proposal period 24h -> 4h
• Lower voting proposal period 4h -> 1h
• Prepare upgrade-plan governance proposal
  • Decide on block height at which to halt
  • Double-check you've got bonded stake
  • Submit proposal
  • Vote for it, notify validators of same
  • Confirm proposal passed (default is 24h voting period)

On release day:

• Draft an announcement for peer review to ensure changes included are comprehensive.
• Disable testnet deploy workflow, so that chain is not reset
• Bump the version number and push its tag, via cargo-release.
  • Run cargo release minor for a new testnet, or cargo release patch for a bugfix. For the latter, make sure you're on a dedicated release branch.
  • Push the commit and newly generated tag, e.g. v0.51.0, to the remote.
• Manually trigger container-build workflow, bc deploy workflow is disabled
• Wait for the "Release" workflow to complete
• Edit the newly created release object, and add a note summarizing the intent of the release
• Close faucet (chain halt will make it inoperative anyway)
• Run migrations on all validators
• Run migrations on all fullnodes
• Update Galileo deployment, following docs
• Upload post-migration archive to https://snapshots.penumbra.zone
• Make the announcement to Discord! 🎉🎉🎉

Post-release cleanup tasks

• Ensure faucet is open
• Ensure a late-joining node can join via an archive
• Bump grpcui version for v1 reflection compatibility
• Perform Hermes maintenance for genesis restart to get relayer running again
• Confirm IBC channels are working
• PR the v0.76.0 dep bump into Hermes repo

[conorsch]

Lower voting proposal period 24h -> 4h

Submitted today:

❯ pcli q governance proposal 0 definition
title = "lower proposal voting duration to 4h"
description = "enabling faster voting in support of upgrade testing in coming weeks"

[[parameterChange.changes]]
component = "governanceParams"
key = "proposalVotingBlocks"
value = "\"2880\""

[hdevalence]

We should be sure to pull in a current snapshot of minifront cc @grod220 @turbocrime

[hdevalence]

We cannot do this until the proto messages erroneously added as part of #4391 are removed.

[conorsch]

Prepare upgrade-plan governance proposal

❯ pcli q governance proposal 4 definition
id = "4"
title = "upgrade to 0.76.0"
description = "planned upgrade, via chain migration, to testnet 76"

[upgradePlan]
height = "222200"


❯ pcli q governance proposal 4 period
{
  "voting_start_block": 221398,
  "voting_end_block": 222123
}

[conorsch]

❯ date -u
Fri May 24 10:14:26 PM UTC 2024

❯ pcli q governance proposal 4 state
{
  "finished": {
    "outcome": {
      "passed": {}
    }
  }
}

[conorsch]

Perform Hermes
Confirm IBC channels are working

@avahowell performed the hermes maintenance and confirmed working:

• https://www.mintscan.io/noble-testnet/tx/3987DA9988A7F3308070BFF924B5B2700A6688AEB8C3B770E8126651D371E834/?height=6749021 update across boundry for noble
• https://www.mintscan.io/osmosis-testnet/tx/EDF09DC7C629C474E6E27F3F899F0619DD769D47C1FBCC0270FADCBD469F3641?height=7827416 osmosis

Worth noting that the long migration time was concerning because we need to migrate the chain within the trusting period, which is 2h. We also overlooked updating the Hermes build deps for Penumbra v0.76.0, so had to rebuild.

[conorsch]

Notes from release process: used a fully-scripted approach to apply the chain upgrades this time, to reduce chance of operator error. The command I ran was:

cd deployments/
HELM_RELEASE=penumbra-testnet TO_VERSION=v0.76.0 ./scripts/k8s-perform-

That process worked well, but was pretty slow: the script is conservative, and spent most of its run time creating backups and tar-ing up post-migration state. Testing on devnets, with minimial chain state on the order of a few hundred blocks, the script's run time was ~5m. On the actual testnet with ~200k blocks, the script's run time was 43m21s. Notably the script doesn't parallize any of the upgrades, but intentionally serializes them and bails out if any fails. Not bothering to optimize that logic now, but recording these hot takes while the info is fresh in my mind.

In the future, once we're sure the logic in the scripted approach is sound, parallelization alone would get us nearly a 10x speedup: we've got 2 vals, 3 nodes backing the RPC, 1 seed node, and 3 more solo fullnodes backing the various UI frontends (block-explorer, dex-explorer, and gov-dash, the latter unused).

Also, it's worth circling back on the disk usage of the multiple backups and state archives. If left unaddressed, those will stick around until the next chain migration, at which point they'll be clobbered. Worth considering because it means the provisioned storage for each node is now consumed by a lot more data than just the live chain state.

[conorsch]

Bump grpcui version for v1 reflection compatibility

This is done, the new v1 reflection APIs are live on https://grpcui.testnet.penumbra.zone

[conorsch]

Leaving galileo off, since it's failing to send txs:

May 25 00:07:09.669 ERROR galileo::responder: Failed to send funds addr=penumbra1a3afgaqz86rmh4f7p6szaygwlskwv2zur4qt2kr2z834uvjdavu9fw3sc3wjkhkf7mzvnmjwxh56t8ga98der33zyyeeqqkev8ele08j65zrg0nqf0fqvvmxhvf0af4pnjlgvu e=status: Unavailable, message: "error getting app params: missing fmd_meta_params", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }

will circle back to it.

[conorsch]

Testnet 76 has shipped, via chain upgrade, and all the follow-up post-release tasks are complete.