| title | order |
|---|---|
Neutron v5.0.0 Upgrade |
2 |
- Chain upgrade point:
November 14th 2024, 14:30 UTCapproximately at height16738000; - Go version:
v1.22 - Release: https://github.com/neutron-org/neutron/releases/tag/v5.0.1
This document describes the steps for validators and full node operators, to upgrade successfully to the Neutron v5.0.1 release. For more details on the release, please see the release notes.
The upgrade will take place approximately on November 14th at 15:30 UTC approximately at height 16738000;
The chain-id of the network will remain the same, neutron-1. This is because an in-place migration of state will take place, i.e., this upgrade does not export any state.
- 64GB RAM is recommended to ensure a smooth upgrade.
If you have less than 64GB RAM, you might try creating a swapfile to swap an idle program onto the hard disk to free up memory. This can allow your machine to run the binary than it could run in RAM alone.
sudo fallocate -l 64G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile- Make sure you have enough disk space for upgrade, the state can grow twice durng upgrade.
Prior to the upgrade, validators are encouraged to take a full data snapshot. Snapshotting depends heavily on infrastructure, but generally this can be done by backing up the .neutrond directory.
If you use Cosmovisor to upgrade, by default, Cosmovisor will backup your data upon upgrade. See below upgrade using cosmovisor section.
It is critically important for validator operators to back-up the .neutrond/data/priv_validator_state.json file after stopping the neutrond process. This file is updated every block as your validator participates in consensus rounds. It is a critical file needed to prevent double-signing, in case the upgrade fails and the previous chain needs to be restarted.
The Neutron mainnet network, neutron-1, is currently running Neutron v4.2.4. We anticipate that operators who are running on v4.2.4, will be able to upgrade successfully. Validators are expected to ensure that their systems are up-to-date and capable of performing the upgrade. This includes running the correct binary, or if building from source, building with go 1.21.
The Neutron mainnet network, neutron-1, will run Neutron v5.0.1. Operators MUST use this version post-upgrade to remain connected to the network.
There are 2 major ways to upgrade a node:
- Manual upgrade
- Upgrade using Cosmovisor
- Either by manually preparing the new binary
- Or by using the auto-download functionality (this is not yet recommended)
If you prefer to use Cosmovisor to upgrade, some preparation work is needed before upgrade.
git clone https://github.com/neutron-org/neutron.git cd $HOME/neutron
git pull
git fetch --tags
git checkout v5.0.1
make install $ neutrond version --long
name: neutron
server_name: neutrond
version: 5.0.1
commit: 954660e40667a67dba1fb68dc347ba1ec90f9c1f
...Or check checksum of the binary if you decided to download it
$ shasum -a 256 neutrond-linux-amd64
4cb48cf146be021f9e4a2a748eaa4fc2bf0979704a7ee5b25c761b9431060570 neutrond-linux-amd64You can check the version you are currently using by running the following command:
$ neutrond q wasm libwasmvm-version
2.1.3
The proper version is 2.1.3.
If the version on your machine is different you MUST change it immediately!
- Use a statically built Neutrond binary from an official Neutron release: https://github.com/neutron-org/neutron/releases/tag/v5.0.1
- If you built Neutron binary by yourself,
libwasmvmshould be loaded dynamically in your binary and somehow, the wronglibwasmvmlibrary was present on your machine. You can change it by downloading the proper one and linking it to the Neutron binary manually:
- download a proper version of
libwasmvm:
$ wget https://github.com/CosmWasm/wasmvm/releases/download/v2.1.3/libwasmvm.x86_64.so
- tell the linker where to find it:
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib/
- check that libwasmvm version is correct:
$ neutrond q wasm libwasmvm-version
2.1.3
Make sure Neutron v5.0.1 is installed by either downloading a compatible binary, or building from source. Building from source requires Golang 1.22.x.
Run Neutron v4.2.4 till upgrade height, the node will panic:
ERR UPGRADE "v5.0.0" NEEDED at height: 16738000: upgrade to v5.0.0 and applying upgrade "v5.0.0" at height: 16705000Stop the node, and switch the binary to Neutron v5.0.1 and re-start by neutrond start.
It may take several minutes to a few hours until validators with a total sum voting power > 2/3 to complete their node upgrades. After that, the chain can continue to produce blocks.
Install the latest version of Cosmovisor (1.5.0):
go install cosmossdk.io/tools/cosmovisor/cmd/[email protected]Verify Cosmovisor Version
cosmovisor version
cosmovisor version: v1.5.0Create a cosmovisor folder:
create a Cosmovisor folder inside $NEUTRON_HOME and move Neutron v3.0.2 into $NEUTRON_HOME/cosmovisor/genesis/bin
mkdir -p $NEUTRON_HOME/cosmovisor/genesis/bin
cp $(which neutrond) $NEUTRON_HOME/cosmovisor/genesis/binbuild Neutron v5.0.1, and move neutrond v5.0.1 to $NEUTRON_HOME/cosmovisor/upgrades/v5.0.0/bin
mkdir -p $NEUTRON_HOME/cosmovisor/upgrades/v5.0.0/bin
cp $(which neutrond) $NEUTRON_HOME/cosmovisor/upgrades/v5.0.0/binThen you should get the following structure:
.
├── current -> genesis or upgrades/<name>
├── genesis
│ └── bin
│ └── neutrond #v4.2.4
└── upgrades
└── v5.0.0
└── bin
└── neutrond #v5.0.1Export the environmental variables:
export DAEMON_NAME=neutrond
# please change to your own neutron home dir
# please note `DAEMON_HOME` has to be absolute path
export DAEMON_HOME=$NEUTRON_HOME
export DAEMON_RESTART_AFTER_UPGRADE=trueStart the node:
cosmovisor run start --x-crisis-skip-assert-invariants --home $DAEMON_HOMESkipping the invariant checks is strongly encouraged since it decreases the upgrade time significantly and since there are some other improvements coming to the crisis module in the next release of the Cosmos SDK.
When the upgrade block height is reached, Neutron will panic and stop.
After upgrade, the chain will continue to produce blocks when validators with a total sum voting power > 2/3 complete their node upgrades.
Most likely it takes a couple of minutes.
During the network upgrade, core Neutron team will be keeping an ever vigilant eye and communicating with operators on the status of their upgrades. During this time, the core team will listen to operator needs to determine if the upgrade is experiencing unintended challenges. In the event of unexpected challenges, the core team, after conferring with operators and attaining social consensus, may choose to declare that the upgrade will be skipped.
Steps to skip this upgrade proposal are simply to resume the neutron-1 network with the (downgraded) v4.2.4 binary using the following command:
neutrond start --unsafe-skip-upgrade 16738000
Note: There is no particular need to restore a state snapshot prior to the upgrade height, unless specifically directed by core Neutron team.
Important: A social consensus decision to skip the upgrade will be based solely on technical merits, thereby respecting and maintaining the decentralized governance process of the upgrade proposal's successful YES vote.
Operators are encouraged to join the #validator-chat channel in the HUB VALIDATOR section of the Neutron Discord. This channel is the primary communication tool for operators to ask questions, report upgrade status, report technical issues, and to build social consensus should the need arise. This channel is restricted to known operators and requires verification beforehand. Request to join the #validator-chat channel by opening a ticket or asking in the general channel.
As a validator performing the upgrade procedure on your consensus nodes carries a heightened risk of double-signing and being slashed. The most important piece of this procedure is verifying your software version and genesis file hash before starting your validator and signing.
The riskiest thing a validator can do is discover that they made a mistake and repeat the upgrade procedure again during the network startup. If you discover a mistake in the process, the best thing to do is wait for the network to start before correcting it.
-
Q: My node restarted in the middle of upgrade process (OOM killed, hardware issues, etc), is it safe to just proceed with the upgrade
A: No. Most likely the upgrade will be completed successfully. But you get AppHash error after the network gets up. It's a lot safer to restart full process from scratch(recover the node from a backup).
To perform an upgrade you need to keep your
./data/priv_validator_state.jsonfile when you are applying a snapshot from the backup. This will help you avoid the risk of slashing due to double signing.