[Shipped] Lotus Client Release v1.27.1

[rjan90]

🚢 Estimated shipping date

• RC1: June 11th, 2024
• Stable: June 24th, 2024

✅ Release Checklist

• Fork a new branch (release/vX.Y.Z) from master and make any further release related changes to this branch. If any "non-trivial" changes get added to the release, uncheck all the checkboxes and return to this stage.
• Bump the version in build/version.go in the master branch to vX.Y.(Z+1)-dev (bump from feature release) or vX.(Y+1).0-dev (bump from mandatory release). Run make gen and make docsgen-cli before committing changes
  chore: bump version in master #12074

Prepping an RC:

• version string in build/version.go needs to be updated to end with '-rcX' (in the release/vX.Y.Z branch)
• run make gen && make docsgen-cli
• Generate changelog using the script at scripts/mkreleaselog
• Add contents of generated text to lotus/CHANGELOG.md in addition to other details
• Commit using PR
• tag commit with vX.Y.Z-rcN
• cut a pre-release here

Stable Release

• Final preparation
  • Verify that version string in version.go has been updated.
  • Verify that codegen is up to date (make gen && make docsgen-cli)
  • Ensure that CHANGELOG.md is up to date
  • Merge release-vX.Y.Z into the releases branch.
  • Tag this merge commit (on the releases branch) with vX.Y.Z
  • Cut the release here.

Post-Release

• Merge the releases branch back into master, ignoring the changes to version.go (keep the -dev version from master). Do NOT delete the releases branch when doing so!
• Update RELEASE_ISSUE_TEMPLATE.md with any improvements determined from this latest release iteration.
• Create an issue using RELEASE_ISSUE_TEMPLATE.md for the next release.

[rjan90]

Writing down steps taken here, and some timings so we can get a baseline:

• Copy/Paste issue template
  1 m

• Fork a new branch (release/vX.Y.Z) from master and make any further release related changes to this branch.
  1 m

• Bump the version in build/version.go in the master branch to vX.Y.(Z+1)-dev (bump from feature release) or vX.(Y+1).0-dev (bump from mandatory release). Run make gen and make docsgen-cli before committing changes in a PR: chore: bump version in master #12074
  5 m

Prepping an RC:

• version string in build/version.go needs to be updated to end with '-rcX' (in the release/vX.Y.Z branch)
  1m

• run make gen && make docsgen-cli
  5m

• Generate changelog using the script at scripts/mkreleaselog
  4m
  Could probably be faster, but I encountered a couple of issues a long the way:

Failed with:

./scripts/mkreleaselog > CHANGELOGHAHAH
Computing old deps...
Computing new deps...
Generating Changelog for github.com/filecoin-project/lotus v1.27.0..10025139e751a1fe6abd44dcc867af6c1ead626a
fatal: ambiguous argument 'v1.27.0..10025139e751a1fe6abd44dcc867af6c1ead626a': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

Had to cd go/github.com/github.com/filecoin-project/lotus - and git pull, then back to home/lotus/ and run it again.

Second failure: I had committed the step in run make gen && make docsgen-cli in my local branch, so the script failed. Uncommited, and the script ran.

• Add contents of generated text to lotus/CHANGELOG.md in addition to other details
  30m - Most of this consisted of sorting PRs after New Features/Improvements/Dependecies/Others. I have not added much additional information about this release. This was on the low end of the spectrum w.r.t time. I have spent 4x the time on releases with large release logs, network upgrades need additional information, etc. chore: release: Prep v1.27.1-rc1 #12077

• Updating my local Lotus node with the above branch for sanity building/starting:
  5m

• cut a pre-release here
  5m

[BigLep]

@rjan90 :

1. Thanks for creating the issue
2. I like your idea of recording every step taken and how long it takes. That will be good for seeing the opportunity here to improve and how much the improvements are actually helping. Good stuff

Not to self: once I get part of lotus-contributors team, I will:

1. Pin the issue
2. Add to FilOz board
   (if this isn't done already)

[rjan90]

Adding time I used for the Post-Release when finishing it for v1.27.0: #11938.

• Merge the releases branch back into master, ignoring the changes to version.go (keep the -dev version from master). Do NOT delete the releases branch when doing so!
  25 minutes PR here: chore: releases: merge releases branch to master #12076

• Create an issue using RELEASE_ISSUE_TEMPLATE.md for the next release.
  1 minute

Notes

• High variance in time consumption in merging releases branch to master branch depending on how many conflicts that needs to be resolved. Sometimes I probably spend 2-3x the amount, others 1/5 of the amount of the time measured.
• I skipped the Update [RELEASE_ISSUE_TEMPLATE.md](https://github.com/filecoin-project/lotus/blob/master/documentation/misc/RELEASE_ISSUE_TEMPLATE.md) with any improvements determined from this latest release iteration. step. Since that templated is planned to be completely overhauled.

[rjan90]

Timings for RC2-parts:

• Writing more on the changelog, going through PRs landed since RC1 and adding backports:
  20m

• run make gen && make docsgen-cli
  5m

[BigLep]

@aarshkshah1992 : I didn't sync with @rjan90 on this release (I've only been discussing 1.28.0). I assume you're going to handle this. Is it clear to you when (date and conditions) to make the final release? I'm bringing this up because the 1.28.0 release (#12109 ) will branch from this.

[jennijuju]

@BigLep oops sorry just saw your last comment - i will handle this one!

[BigLep]

Per 2024-06-25 verbal conversation, this is complete: https://github.com/filecoin-project/lotus/releases/tag/v1.27.1

[rvagg]

(Noting this here because I've been asked twice now and there may be others).

This release contains a migration in the events database, full details of this are in #11952

The events database is not turned on in the default Lotus configuration, you have to turn one of a few things on in order for it to be running. If your Lotus configuration has at least one of the following options then you have the database on:

• Events.DisableHistoricFilterAPI=false
• Fevm.Events.DisableHistoricFilterAPI=false

You can also check for yourself by looking at your lotus repo directory: if there is a sqlite/events.db file that has been recently modified then you have it turned on.

For users who are operating the events database and find themselves wanting to downgrade from this release to v1.27.0, this is possible, but it requires some manual work. This particular migration is (roughly) idempotent, so going back and forth through the migration should be non-destructive and should not cause errors if done correctly. The main problem is that we store a schema version in the events database once the migration is performed so we know it doesn't need to be done again. This version needs to be wound back to the previous version when starting with an older version of Lous. If you attempt to un an older version of Lotus with a migrated database then you'll get an error, something like: invalid database version: got 5, expected 4.

To downgrade

NOTE: this instructions are specific to the migration in #11952 (in Lotus v1.27.1) and only works because the schema of the database did not change. If you require help downgrading other versions, please ask, there may or may not be simply ways to achieve it.

Caveat emptor: I haven’t actually tested this myself on a live node, but it's a simple enough migration that this should all be straightforward. If you perform these steps you are taking responsibility for the outcome. If your node contains critical history and you depend on accuracy of the events APIs then you would be advised to take a backup of the entire repo as the blockstore and events database should stay in sync in order to have the right data up to the latest tipset. Leave some feedback if this goes well for you to give others some additional assurance.

1. Install SQLite 3 (should be available in most Linux package managers as sqlite3) so that we have the sqlite3 CLI tool.
2. Stop your Lotus v1.27.1 daemon; waiting until it has fully shut down
3. (As the user running the lotus daemon, or with sudo) Run sqlite3 /path/to/lotus/repo/sqlite/events.db (your Lotus repo may simply be ~/.lotus/).
4. Copy this SQL into the CLI tool: DELETE FROM _meta WHERE version = 5; and press enter.
5. Exit with .exit
6. Start your Lotus v1.27.0 daemon. Watch the logs, checking for mentions of "event index" or "database".

As soon as you make the switch from v1.27.0 to v1.27.1 (or later), the migration will be re-run and the version 5 will be re-inserted.

Additional notes

Please be aware that v1.27.1 contains a fix for this Ethereum API incompatibility bug: #11630, downgrading back to v1.27.0 will lose the fix and will also give you different event indexes to a non-migrated v1.27.0 (and previous) node. Events prior to your downgrade will be properly indexed (which v1.27.0 and prior won't have), but events after the downgrade will be improperly indexed. If you don't have an application that depends on indexes to be correct as per the Ethereum API then this is unlikely a concern. When proceeding past this migration in a future upgrade (to v1.27.1 or some future version), your indexes should be fixed again as all historical events are re-indexed.

[jennijuju]

@rvagg should we add this to release note?

[rvagg]

@jennijuju yeah, I can do that, but I also put an additional caveat just now in the middle of my text, I don't want to give the impression this is a guaranteed solution. Migrations are normally intended to go one way, this one just happens to be fairly simple.