LimeChain's proposal for a research grant for the Polkadot Protocol Conformance Tests RFP #1950
Conversation
…onformance Tests RFP
…onformance Tests RFP
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
|
I have read and hereby sign the Contributor License Agreement. |
|
recheck |
There was a problem hiding this comment.
Thanks for the application. Could you potentially also add to the deliveries a short report that tries to outline the different requirements of the different implementations? The conformance test suit ideally should be used by all implementers, and it would be nice if you quickly sync with them as part of the grant.
|
@hgergov Bhargav also invited you to a shared channel with the Zombienet team. Could you double-check if you received the invite? |
|
|
||
| [Zombienet](https://github.com/paritytech/zombienet) aims to be a testing framework for Substrate-based blockchains, providing a CLI tool that allows users to spawn and test ephemeral networks. The assertions used in the tests can include on-chain storage, metrics, logs, and custom JavaScript scripts that interact with the chain | ||
|
|
||
| Our team has successfully utilised Zombienet to run PVF conformance tests as part of another W3F grant. This is the reason why we believe that Zombienet has the potential to be the go-to framework for running conformance tests on the Hosts. The research is going to focus on Zombienet, as well as on the new [Zombienet SDK](https://github.com/paritytech/zombienet-sdk). |
There was a problem hiding this comment.
Could you please elaborate on how Zombienet was used to integrate with the PVF conformance test-suite? And briefly compare and contrast with the current project?
There was a problem hiding this comment.
In the PVF conformance test-suite, we leveraged Zombienet's custom WASM configuration option(genesis_wasm_path) in order to inject malicious PVFs. Additionally, the team used Zombienet's DSL language in order to assert on the behaviour of the network. You can find the tests here. They're split into different folders and files mainly because it's easier to read and understand what's happening. The main difference between Polkadot and Kagome's tests is the startup command. There are also one or two differences in terms of logs emitted by each Host.
There was a problem hiding this comment.
When it comes to comparing the 'adapter' approach with the Zombienet approach, the biggest difference is in terms of maintainability and being forward-compatible. The adapter approach is dependent on each Host not changing its implementation details. If the function's name is changed or is extracted in another module/repository, the testing suite should change as well; otherwise, it won't compile. If a function's logic is changed, then this may result in tests failing. These are all cases very relevant for Hosts in early stages of their implementation. Whereas, changes to CLI parameters and public APIs (which Zombienet uses under the hood) happen less often, which makes the approach better in terms of maintainability
|
|
||
| The main focus of the research will be on investigating each of the aforementioned protocols and functionalities and how they can be tested in a host-agnostic manner. Based on our preliminary research and the work we've conducted on the PVF conformance testing suite, our team believes that Zombienet can serve as the foundation for the testing suite. The team will examine whether Hosts and Zombienet have the required feature set to support such testing scenarios. If this is not achievable, the team will document the missing components so that they can be identified and potentially contributed in a future proposal. | ||
|
|
||
| Our team acknowledges that, for certain Hosts, having the Host API, SCALE, and State Trie as standalone artifacts could be beneficial for Host developers during the early phases of their implementation. However, this proposal primarily focuses on adopting a host-agnostic approach for Hosts that are already functioning with the already existing tests. Depending on the research outcomes, our team may subsequently introduce a new proposal involving the redesign of the conformance repository. In this redesign, the tests would be separated into a standalone artifact and transformed into Zombienet tests. |
There was a problem hiding this comment.
this sounds reasonable, and aligns with the definition of conformance tests. However, it is interesting to check if the conformance tests can be easily distilled to generate stand-alone tests to support early-stage host implementers.
There was a problem hiding this comment.
We believe that generating stand-alone tests is completely feasible and it’s scope could be a completely different project as it will require delving deep in each module and understanding it to the smallest detail. Our proposal, however, aims to discover a new and better approach to running the tests for advanced Host implementations. We believe that coming up with a better approach for testing Hosts is more important than generating stand-alone artefacts as there are currently 3 advanced Host implementations(Polkadot, Kagome, Gossamer) and one in early stages of development(Fruzhin). As stated in the proposal, we have plans to work on expanding the tests and extracting them in stand-alone artefacts in the future.
| 2. Child Storage | ||
| 3. Hashing | ||
| 4. Trie | ||
| 2. SCALE |
There was a problem hiding this comment.
SCALE codec already has a comprehensive test-suite here which can be leveraged upon.
There was a problem hiding this comment.
Yes, that’s correct. If it’s an interest to the W3F and the community we can add more SCALE codec test scenarios to the PoC if it proves to be feasible. We’re trying to implement a different approach because as we mentioned the “adapter” approach is not very flexible
| 1. Trie encoding | ||
| 2. Trie decoding | ||
| 3. Trie proof generation | ||
| 4. BABE |
There was a problem hiding this comment.
Can you elaborate more on this? Does block import include forking and longest chain related properties?
There was a problem hiding this comment.
We’ve intentionally left out specific properties for BABE and GRANDPA as our main goal will be to establish a way to test any BABE functionality/property which implicitly includes forking and longest chain related properties. If you have specific properties that you’d want us to test, feel free to note them here as we won’t be able to cover all of them in the specified timeline.
| 4. BABE | ||
| 1. Block import | ||
| 2. Block validation | ||
| 5. GRANDPA |
There was a problem hiding this comment.
similarly, does this cover voting and justifications of the finalisation mechanism?
There was a problem hiding this comment.
We’ve intentionally left out specific properties for BABE and GRANDPA as our main goal will be to establish a way to test any GRANDPA functionality/property which implicitly includes voting and justifications. If you have specific properties that you’d want us to test, feel free to note them here as we won’t be able to cover all of them in the specified timeline.
|
|
||
| | Number | Deliverable | Specification | | ||
| | --- | --- | --- | | ||
| | 0a. | License | Apache 2.0 / GPLv3 / MIT / Unlicense | |
| # Polkadot Protocol Conformance Tests Research Proposal | ||
|
|
||
| - **Team Name:** [LimeChain](https://github.com/LimeChain) | ||
| - **Payment Address:** 15reUHgnkE9QeH9zUoduCVYAbksTmQXvpMd6L95rBhTNFuGw |
There was a problem hiding this comment.
| - **Payment Address:** 15reUHgnkE9QeH9zUoduCVYAbksTmQXvpMd6L95rBhTNFuGw | |
| - **Payment Address:** 15reUHgnkE9QeH9zUoduCVYAbksTmQXvpMd6L95rBhTNFuGw (USDT) |
Hey David, hopefully, I’ve understood your reply correctly. The goal of the research is to prove or disprove whether using Zombienet/Zombienet-SDK for these specific Host's modules is possible. If the research turns out positive, then we can schedule a call with each Host’s team and have a discussion regarding what needs to be added in order to support the tests. We believe that any preliminary syncs with Host implementation teams would be pointless, as the outcome could be negative. If that’s not what you meant, please let us know. |
|
I have read and hereby sign the Contributor License Agreement. |
There was a problem hiding this comment.
I'm sorry for not getting back to you sooner. Got it. However, in this case, it might make sense to remove "Create one test for each module from the previous steps if the research outcome indicates that it’s possible." and reduce the price. Also because it's unclear how much effort you actually need to put into this project at this stage.
|
Hey David, thanks for your input! It sound reasonable and we've adjusted the scope and effort in the proposal. We're ready for a re-review. |
Noc2
added
ready for review
…neration. Adapt the effort. Adapt the scope to exclude Host API (Storage, Child Storage, Hashing), and Trie Proof Generation.
Hey David, Certainly, we can further narrow down the scope and allocate certain topics for future phases. The most meaningful ones in this context would be Host API (Storage, Child Storage, Hashing), and Trie Proof Generation. We believe that we can rely on the outcomes of the other topics to gain a high level of certainty about them. So, those are great candidates to be postponed. We've already made the changes in the proposal accordingly. |
Removing the PoC development delivery, it was a leftover from a previous commit when the scope was reduced.
There was a problem hiding this comment.
Thanks @hgergov and to everyone else for the deep dives. This is a well-written proposal and I'm interested to see Zombienet become the go-to for conformance tests. Also happy to approve.
|
Congratulations and welcome to the Web3 Foundation Grants Program! Please refer to our Milestone Delivery repository for instructions on how to submit milestones and invoices, our FAQ for frequently asked questions and the support section of our README for more ways to find answers to your questions. |
Hello W3F grants team,
We are pleased to present our project application for a research grant for the Polkadot Protocol Conformance Tests RFP.
It outlines the research approach and objectives. We are looking forward your feedback and are committed to promptly integrating suggested changes, and ensuring the application's refinement.
Best Regards,
LimeChain team