Runtime releases management

[anaelleltd]

Overview:

This document aims to summarise proposed/existing options for formalising the process by which:

• the Fellowship actively consults and collaborates with ecosystem teams
• scheduled runtime upgrades account for potential breaking changes
• new bugs and recurring issues identified by ecosystem teams are prioritised for fixes

Considerations:

• Which areas are known for causing breaking changes? (based on past discussions/post-mortem docs) - See DevX perspectives, Coretime migration fix
• Which builders are commonly/potentially affected by these breaking changes? - See links above.
• How much notice should builders be given ahead of an upgrade with known breaking changes? (aka deadline for PR submissions) - See table below.
• Which actions are builders expected to take following notification? - See examples: instructions for v1.2.6 | New Polkadot Ledger app | Asset Hubs: auto incremented asset id | etc...
• Which steps should builders take when encountering upgrades-related bugs/issues? - See Alpha Program (Ecosystem Success)
• Who is responsible for following up on builders' requests/queries? (aka support from known experts) - See Fellowship profiles
• How can known experts and ecosystem builders contribute to improving QA for runtime releases? - See examples: Brainstorming Coretime migration fix | Checks for correctness of Whitelisting proposal | etc...

Strategies:

Note: A ☑ indicates that the item is already In Progress.

• Draft an outline of planned core developments with Polkadot 2.0 Roadmap
• Provide changelogs with contextual information for PRs with PRDoc
• Set up custom assignments and notifications for PR reviews with Review bot
• Highlight breaking changes more prominently in the changelogs
• Standardise the release process with a summary doc of key stages
• Create a distribution list of builders for targeted notifications on Element, Telegram, email, Github, and more
• Set up a release calendar for communications with ecosystem teams
• Track the schedules of minor and patch releases in Projects [Internal]
• Fast track parachains' replacement of their head state and/or wasm with Unbrick collective
• Set up a comprehensive bank of tests for runtime upgrades with Ecosystem test environment
• Make it mandatory that any Whitelisted Caller proposal include concrete steps (followable by any Rust engineer) on how to verify it, pointing to existing documentation for the basics
• Create a clone network to be used to test runtime/node interactions with Polkadot Doppelganger
• Integrate Doppelganger in Subsquare so that all Whitelisted caller proposals must (by default) have an associated test, which got verified by developers according to a checklist; with the test and test result shown in the UI
• Inform ecosystem builders about storage item changes with "Metadata diff tool"
• Others/TBC

Contributors:

• Parity: Ecosystem success, Engineering, Automation
• Fellowship: Secretary, Members, Fellows, and Architects
• Ecosystem: Parachain teams, dapp builders, Unbrick Collective, JUST OpenSource, OpenGov.Watch

Contacts:

This table can be edited & copied/pasted as a comment on scheduled runtime upgrades to ensure that relevant teams are contacted based on proposed/queued PRs.
The contact details for each team are available here.

-	Teams	Breaking changes	Notification timelines
CEXes	Binance, Bybit, Coinbase, Gateio, Kraken, Kucoin, OKX, Upbit, etc.	Transaction encoding, Event/Error encoding	4+ weeks
Wallets	Enkrypt, Fearless, Infinity, Kampela/Ledger, Kuma, Mimir, Multix, Nova/Polkadot Vault, Polkadot-JS, PolkaGate, PolkaSafe, Polkawallet, Spekter, Subwallet, Talisman/Signet, Tangem, etc.	Transaction encoding	4+ weeks
Explorers	Polkascan, Statescan, SubID, Subscan	XCM and storage format	3+ weeks
User Interfaces	Delegation dashboard, DoTreasury/Subsquare, Kodadot, NOMI, Polkadot-JS Apps, Polkassembly/Townhall, Proof of Chaos, Singular, Staking dashboard	Extrinsic usage, XCM and storage format	3+ weeks
Parachains	Polkadot & Kusama	polkadot-sdk (especially migrations) and XCM	Ad hoc
Live networks	Aleph Zero, Bittensor, Cere Network, ChainFlip, Joystream, Ternoa, Vara, etc.	polkadot-sdk (especially migrations) and XCM	Ad hoc

---

Last updated: 14th November 2024

[anaelleltd]

Please share your suggestions or recommendations so that the document above can be made as comprehensive as possible.

[joepetrowski]

Some suggestions:

• CEXs: Add Event/Error encoding
• DEXs: I don't quite get what you are grouping here. A lot of them are parachains and should just be your "Parachains" category. I would delete this row.
• Wallets: Transaction encoding
• Explorers: AFAIK these are pretty resilient to breaking changes at this point, since we've added types to metadata.
• OpenGov dApps, NFTs, and Dashboards can all be collapsed into one category: "User Interfaces".
• Parachains: Breaking changes of concern should be polkadot-sdk breaking changes (especially requiring migrations) and XCM breaking changes.
• Bridges: Would probably delete. Some of them (P<>K, Snow) are part of the release itself. Others are parachains and thus fit into the "Parachains" category.
• System Chains: Delete. They are part of the release. Who will the Fellowship notify? Itself?

[anaelleltd]

Thank you for these clarifications and corrections, @joepetrowski.
I have included them in the main document.

[bkchr]

One thing that I discussed today with @eskimor is the introduction of some sort of "metadata diff tool". This should help with informing people if for example storage items change. This tool could run for each pr merged to main and later for the release itself. There should exists "client" that can read these diffs and find out if a storage item that your app is using while change. By doing it per pull request the teams could get notified before the actual release is happening. The teams would need to run this tool maybe once per night to get notified.

@josepot you are already generating interfaces for the runtimes using metadata, not sure if you could maybe easily figure out what apis a user is using and based on this determine the storage item.

[josepot]

@josepot you are already generating interfaces for the runtimes using metadata,

yes, of course! All of the generated code comes directly from the metadata.

not sure if you could maybe easily figure out what apis a user is using and based on this determine the storage item.

In a way our type system already does this for our users, because if the types generated from the next runtime don't match, then the typescript compiler will let our users know what has changed.

One thing that I discussed today with @eskimor is the introduction of some sort of "metadata diff tool". This should help with informing people if for example storage items change

We could easily use the tools that we have built to generate a diff tool like this. We have already built similar tools that we use internally to detect common types across different chains, so that we can give them some friendly names for the developers.

So, yeah, we could totally build a "metadata diff tool", for sure! How urgent/important is this?

The teams would need to run this tool maybe once per night to get notified.

Wouldn't it be possible/desirable to use chopsticks to get the resulting metadata that will take place after a certain referenda gets enacted, and then run the diff tool against the resulting metadatas of those referendas? 🤔

[bkchr]

So, yeah, we could totally build a "metadata diff tool", for sure! How urgent/important is this?

Not a requirement that you write it. The general idea here would also be that teams are informed before the stuff is on chain, so the apps do not break in production. Better they are informed before, make the code work with both version and then be ready when the runtime upgrade is live.

[bkchr]

We also should highlight breaking changes more prominently in the changelogs. Basically it should be easier to find out when a change breaks something, like #337 which was made not clear that it breaks the transaction encoding.

[bkchr]

We should start simple by including the following information in the changelog:

• Does it break the transaction encoding?
• List all the storage keys that do not exist anymore or which type has changed.

[bkchr]

We should create multiple communication channels for pushing information around upcoming releases. When we are aware that breaking changes will hit a release, we should start to push out information and the closer we get to the actual enactment of the update the more spammy we should get. Better to over deliver and to annoy people, than people to miss the information. We should support Element, TG, email, github, maybe more? A release calendar.

[jak-pan]

Thanks 👍🏻 I would maybe add that linking to PRs and showing off the integration tests that changed in the breaking change PRs will get a long way to quickly evaluate the required effort. Alternatively tips like -> Update your .js libs would help. Having calendar to see it visually would be epic.

[anaelleltd]

@bkchr @jak-pan @SBalaguer
I have summarised the requirements for a runtime release calendar here
Please let me know if I missed anything important.

[anaelleltd]

Further suggestions from @eskimor

[Imod7]

In the past, subwasm was used during the release process to generate a comparison between different runtimes. Is there any plan to use it again or maybe another tool that generates the same or similar report? Thank you!