SwarmNL

[sacha-l]

Project Abstract

SwarmNL is a configurable P2P networking layer designed to be used in architectures that require data transfer solutions for off-chain communication in a scalable way. It's goal is to provide a lightweight and configurable P2P networking layer for web applications designed to interact with on-chain data and will be used to further the development of SamaritanDB.

Grant level

• Level 1: Up to $10,000, 2 approvals
• Level 2: Up to $30,000, 3 approvals
• Level 3: Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

• The application template has been copied and aptly renamed (project_name.md).
• I have read the application guidelines.
• Payment details have been provided (bank details via email or Polkadot (USDC & USDT) address in the application).
• I am aware that, in order to receive a grant, I (and the entity I represent) have to successfully complete a KYC/KYB check.
• The software delivered for this grant will be released under an open-source license specified in the application.
• The initial PR contains only one commit (squash and force-push if needed).
• The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
• I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

[github-actions]

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

[sacha-l]

I have read and hereby sign the Contributor License Agreement.

[takahser]

@sacha-l thx, this looks potentially interesting, but I'd like to learn more about your project:

• what kind of data is samaritian db capable of storing? it looks like it's a key-value db, is that correct (relevant code sample: set <application_DID> [samaritan_DID] <key> <value>)?

• could you add more details do the non-mandatory deliverable specifications (1.+)? for example:

  Implementation of the configuration interfaces to define the behaviour of the nodes and the network.

  => which parameters would be included that define the node behaviour?
  => how would developers configure it? a config file (JSON/YML/..)? env variables?

• the live demo mentioned PoC repo is returning a HTTP 404; do you happen to have a working link?

[takahser]

Could you name some other good use cases where swarmNL could be leveraged?

[thewoodfish]

Thank you for your question. swarmNL could be leveraged in Content Delivery Networks (CDNs), distributed storage systems, IoT (Internet of Things), multiplayer online games and file hosting and sharing systems

[PieWol]

The way I see it the gossipsub libp2p version already has strategies (e.g. heartbeat mechanism) built in to deal with varying amount and availability of nodes. Whats the benefit of you creating new strategies?

[thewoodfish]

Absolutely, the gossipsub implementation in libp2p provides valuable strategies like the heartbeat mechanism for addressing varying node availability. However, our approach isn't about reinventing these strategies; it's about leveraging and enhancing them for more specialized use cases.
While libp2p handles liveness through mechanisms like ping/heartbeat/keep-alive, our library extends this by offering additional strategies to address scenarios beyond mere liveness detection. For instance, consider the following use cases:

• Node Failure Handling:
  In the event of a node going down, our library provides customizable options for developers to define reconnection strategies, automatic peer discovery, and failover mechanisms. This ensures that the network can gracefully adapt to failures without compromising overall system performance.
• Sharding Support:
  For scenarios where nodes are dedicated to specific shards or portions of the network, our library will offer functionality to seamlessly integrate and manage such specialised nodes. This allows for more fine-grained control over the network's structure, enabling efficient scaling and resource allocation.
• Replication Nodes:
  In cases where replication nodes are required for data redundancy, our library facilitates the addition of nodes dedicated solely to replication tasks. This ensures that data is appropriately distributed and replicated across the network.

[PieWol]

You know what I think the bullet points you listed are so helpful and I'm amazed that you are able to so excplicitly state the features. I wish you take those listed features for the categories you named and include them as deliverabled in the respective subsection. Very helpful to understand what you envision!

[sacha-l]

I've updated the PR with @thewoodfish's comments here as I think they do add more clarity on what we mean for implementing gossipsub.

[PieWol]

SwarmNL is the network layer you are proposing itself, right? Why would it be generic over the network layer? I assume you meant to say that it will be generic over the storage layer, correct?

[thewoodfish]

YES, we meant the storage layer. Thank you

[PieWol]

can you please fix this then?

[sacha-l]

Updated.

[PieWol]

Can you please elaborate on the fault tolerance you want to introduce that has a benefit over the already existing fault tolerance / error correction in the TCP / IP stack? I assume re-routing strategies are already handled by gossipsub but node replication sounds especially interesting.

[thewoodfish]

The fault tolerance mechanisms introduced in SwarmNL are designed to complement and extend the fault tolerance and error correction features provided by the traditional TCP/IP stack. While TCP/IP ensures reliable point-to-point communication, decentralized and peer-to-peer networks face unique challenges, such as dynamic node behaviors, varying network conditions, and potential node failures. SwarmNL enhances fault tolerance by introducing additional strategies beyond those typically found in the TCP/IP stack.
Replication is a key feature in SwarmNL's fault tolerance arsenal. Unlike traditional networks, where replication may not be as prevalent, SwarmNL allows nodes to serve as replicas, mirroring portions of the network. This replication approach enhances availability, as these replicas can seamlessly take over in case of a node failure. Replicated nodes may shadow specific network segments or act as backup nodes that consistently receive updates, ready to replace a failed node.
Also while gossipsub handles message dissemination, SwarmNL introduces additional message re-routing capabilities. In case of a node failure or unavailability, the system can intelligently reroute messages to specific nodes, ensuring that critical information reaches its destination without disruption.

[thewoodfish]

@sacha-l thx, this looks potentially interesting, but I'd like to learn more about your project:

• what kind of data is samaritian db capable of storing? it looks like it's a key-value db, is that correct (relevant code sample: set <application_DID> [samaritan_DID] <key> <value>)?

• could you add more details do the non-mandatory deliverable specifications (1.+)? for example:

  Implementation of the configuration interfaces to define the behaviour of the nodes and the network.

  => which parameters would be included that define the node behaviour?
  => how would developers configure it? a config file (JSON/YML/..)? env variables?

• the live demo mentioned PoC repo is returning a HTTP 404; do you happen to have a working link?

1. SamaritanDB is a NoSQL database designed to store both document-oriented and key-value data. While the earlier versions may have had a key-value focus, the latest version, as indicated in the repository, has evolved into a document database.
   The key-value implementation was initially adopted to optimize for fast iteration and meet specific networking requirements. However, the current focus is on providing a document-oriented database model. Users can store and retrieve data in the form of documents, allowing for flexible and schema-less data structures.

2. Thank you. SwarmNL is meant to be the subsystem that handles the majority of a node's distributed networking. Hence it needs to be configurable. The configurations would be in either JSON or env. Variables. Here's a non-exhaustive list of what would be configurable:
   First, static node parameters like serialised cryptographic key pairs, network shared authentication secret, bootstrapping parameters etc.
   Second and majorly, are parameters that would assist the scaling process e.g sharding and/or replication parameters.
   There are also simple dynamic configurable interfaces like:

• Registration of handlers to be called on network events.
• Gossiping Config
• Protocol reg and handlers
• Fault tolerance config
  Etc.

3. The demonstration revolved around a straightforward web game designed to evaluate the iteration of SamaritanDB during that particular period. The primary emphasis of the game was on the submission of on-chain transactions, with the database promptly responding to these changes and enforcing them locally. Unfortunately, there is currently no available link to access the demonstration.

[sacha-l]

Thanks for the quick review! @thewoodfish (lead engineer) and I spoke and thought it made most sense that he responded directly to your questions/comments. We look forward to your return.

[takahser]

@sacha-l @thewoodfish thanks for ellaborating, sounds good to me. Adding my approval here.

[keeganquigley]

Thanks for your thorough answers @sacha-l I think this is a good use case for the networking layer and since it is also based on libp2p I'm good to go ahead with it.

[PieWol]

Hey @thewoodfish ,
I left some comments and questions that I still have. Please refrain from tagging w3f members in any personal repositories for reviews. I'm sure everybody tries his best to get back to any applicant asap :)

Fyi I'm not a member of the committee so my approval wouldn't get this grant accepted anyway ;)

Looking forward to your responses and adjustments 👍

[PieWol]

Can you please explain what the deliverable 2 means? What do mean with "configurable interfaces for application level data filtering and authentication"?

[thewoodfish]

Our goal is to create a seamless and user-friendly experience for setting up network nodes, facilitating communication between nodes, and ensuring fault tolerance and scalability.

Deliverable 2 focuses on enhancing the Gossipsub protocol, which serves as the backbone of our networking infrastructure. However, we aim to shield developers from its complexities, such as implementing features like backoff, pruning, and cancellation transparently.

One key aspect of this deliverable is to empower developers to customize their nodes' behavior in response to incoming messages. This includes implementing custom data filtering logic and defining actions to be taken based on message content or validation results. Additionally, developers will have the flexibility to implement custom authentication logic for verifying data received from peers in the network.

[sacha-l]

I added some more details @thewoodfish wrote in a previous comment into the "Features" section to add more clarity on this deliverable (under "node communication").

[PieWol]

May I ask what led you to this milestone structuring? To me it would make sense to structure the deliverables in a way that each milestone is one coherent block of deliverables that work together and can be tested together. In this very project I can see it making a lot of sense to first implement the tech needed for a single node and build all the functionalities one node should incorporate and then later on in milestone 2 deliver on the tech thats needed to make the network as a whole work.

Yet it looks like you have now mixed it. In milestone 2 you have the ipfs integration which would be one of the node-specific functionalities and less those of the network.

When you say "define the behaviour of the nodes and the network" what does that mean exactly? To my understanding that is defining the behaviour of a node which collectively but not centrally influenced define the behaviour of the network itself. What is a configuration interface for the network?

[thewoodfish]

You’re absolutely correct. Our milestone structuring aligns with the approach you mentioned. We've organized the deliverables in a way that each milestone forms a coherent unit, ensuring that the components work together seamlessly and can be tested effectively.

Initially, we focus on implementing the technology required for a single node, encompassing all the functionalities that a node should incorporate. This includes configuration parameters that influence both scaling and communication aspects. By addressing these in Milestone 1, we establish a foundation where configured nodes can communicate with peers, paving the way for scaling techniques.

Milestone 2 builds upon this foundation by refining node communication to accommodate our scaling strategies. This includes implementing techniques like sharding and replication, which are interdependent and require integrated testing. While IPFS integration is indeed a node-specific functionality, we've positioned it within Milestone 2 to balance the workload, considering that a significant portion of effort is dedicated to achieving flexible communication in Milestone 1.

Furthermore, interfaces for posting data to remote servers are crucial for enabling nodes to respond to various network events effectively. By including this in Milestone 2, we ensure that these interfaces are ready for integration, contributing to the overall flexibility of the system.

Due to the interconnected nature of the deliverables, i think combining the milestones into a continuous 12-week period appears to be the best decision. This approach will allow for comprehensive testing at each stage, ensuring that the system functions smoothly across all components.

[thewoodfish]

When you say "define the behaviour of the nodes and the network" what does that mean exactly? To my understanding that is defining the behaviour of a node which collectively but not centrally influenced define the behaviour of the network itself. What is a configuration interface for the network?

When we talk about defining the behavior of nodes and the network in the context of SwarmNL, it essentially refers to configuring various parameters and settings that govern how nodes interact with each other and collectively form the network's behavior. Let me break it down further:

Static Node Parameters: These are fixed settings that are typically set during node initialization and remain constant during operation. Examples include cryptographic key pairs, network authentication secrets, and bootstrapping parameters. These parameters establish the fundamental identity and connectivity of the node within the network.

Scaling Parameters: These parameters are crucial for achieving scalability in the network. They encompass settings related to sharding, replication, and other scaling techniques. By adjusting these parameters, nodes can adapt to changes in network size and workload, ensuring efficient resource utilization and performance optimization.

Dynamic Configurable Interfaces: These interfaces provide flexible mechanisms for nodes to interact with the network and respond to dynamic conditions and events. They include functionalities such as:

• Registration of event handlers: Nodes can register callback functions to be executed in response to specific network events, allowing them to react and adapt accordingly.
• Gossiping Configuration: Parameters related to gossip protocols, such as message dissemination strategies, gossiping thresholds, and gossiping interval settings.
• Protocol Registration and Handlers: Nodes can register custom protocols and associated handlers to define how they communicate and exchange data with other nodes.
• Fault Tolerance Configuration: Settings that determine how nodes handle and recover from faults, errors, or network failures, ensuring robustness and resilience in the face of adversities.

These configurations play a pivotal role in shaping the behavior of both individual nodes and the network as a whole. By adjusting these settings, developers can fine-tune the network's performance, scalability, fault tolerance, and overall behavior to suit specific use cases and requirements. Ultimately, the goal is to provide a flexible and adaptable networking framework that empowers developers to build resilient and efficient decentralized applications.

[sacha-l]

As @thewoodfish mentioned, the structure is around first delivering a way to make this library flexible and configurable. Then, delivering a way to make it scalable and usable. We think these building it this way stays closer to our design goals for making this networking layer as flexible and configurable as it can be.

@PieWol we would happily include any suggestions you might have on breaking down the milestones further if you think anything is missing. Note that a lot of this is detailed in the "Features" section which we plan to use as our north start for the deliverables.

[PieWol]

Can you please tell us more about the applications you reference here?

[thewoodfish]

The reason we want SwarmNL is because it plays an important role in what we’ve been trying to build (SamaritanOS).

SamaritanOS or SamOS for short is a collection of technologies that work together to give users self-sovereignty and control of data they input (and can retrieve) across applications compatible with it.

It can be thought of as software that operates as a hub of online applications.

Let’s say every user visiting an application on the internet is authenticated cryptographically and is represented by a DID. The application recognises this DID and stores every data inputted into it in relation to the DID, allowing the user to retain control over that underlying data stored.

In our architecture for SamOS, the data is stored in a DB called SamaritanDB. A document (for now) database that is built to store data and cryptographically bind it to a DID. This DB will be made capable of listening to on-chain state changes made by the user concerning the data. The onchain state in this design contains DID-based permissioned sources of truth, allowing other applications to access application data, in a permissioned way. As a result, this database must be distributed across nodes. It must be capable of retrieving data from IPFS and adopt network protocols for scaling, replication, sharding support to ensure data availability and censorship resistance..

This is majorly where SwarmNL comes into play, to help achieve this and function at the heart of the databases.

The applications we are referring to here are the applications that are built to be compatible with the SamaritanOS infrastructure and achieve our goals of allowing users to interact with them, preserving the users sovereignty over what is being stored and shared within the application.

[sacha-l]

I added this into the "Ecosystem fit" section:

Examples could include Content Delivery Networks (CDNs); distributed storage systems; IoT (Internet of Things); multiplayer online games; and file hosting and sharing systems

[PieWol]

can you please fix this then?

[PieWol]

You know what I think the bullet points you listed are so helpful and I'm amazed that you are able to so excplicitly state the features. I wish you take those listed features for the categories you named and include them as deliverabled in the respective subsection. Very helpful to understand what you envision!

[PieWol]

Have you decided on your desired form of payment already? That should be represented here :)

[PieWol]

I thought the whole key difference between the previous storage solutions you mentioned e.g. polkadot native storage is that you planning to create a decentralized networking layer that is completely off-chain. Thats how I understand this section:

SwarmNL is a configurable P2P networking layer designed to be used in distributed system architectures that require data transfer solutions for off-chain communication in a scalable way.

But in the next sentence you mention that this will be used to interact with on-chain data

It's goal is to provide a lightweight and configurable P2P networking layer for web applications designed to interact with on-chain data.

Where I'm wondering why would anybody want to use this or the distributed storage solution you envision. Because as soon as on-chain data is involved there is the need to make sure this data is correct. Hence you will probably just ask the nodes for the on-chain data directly, wouldn't you? I every entity in this network equipped with a light-client to stay up to date on the chainstate? As you can see I'm a little clueless how this will play out and I'm thankful for any explanation you can give me :)

[thewoodfish]

What we meant is that it could be extended to serve as the networking stack for on-chain data transfer. Of course, there will be a need for a subsystem for verification etc. But with what we envision for SwarmNL, adding things like that would not have too much overhead.
Maybe we should make that correction and add it as something that COULD be done, or remove it altogether.

[sacha-l]

Indeed, adding "could" would be more accurate. The networking layer could be extended to integrate some on-chain verification mechanism an application requires it.

[PieWol]

Storage and retrieval of keypair locally.

Why would you want to have this configurable? Whats the benefit over a static solution? Or is this referring to an initial setup instead of configurability?

[thewoodfish]

I believe this has been addressed here

[PieWol]

Do you already know which protocol you plan to implement?

[thewoodfish]

Yes, indeed. We are planning to implement sharding as part of our network scaling strategy. While we have a clear understanding of what sharding entails and the different techniques available, we haven't finalized the specific protocol yet. However, we aim to support a diverse range of sharding techniques to cater to various use cases and requirements.

Some of the sharding techniques we are considering include Range-based, key-based and hash-based sharding.

[semuelle]

@sacha-l, could you fill out the KYB form?

[sacha-l]

Left some comments to add clarity and will follow up with the updates requested from the reviews including updating a payment address (in incoming commits). Hopefully the comments provide more clarity in addition to the minor updates I made. Let us know @PieWol if you want us to modify the list of deliverables to make them more detailed.

@semuelle : regarding the legal entity, I've updated our submission and removed the mention of Algorealm, Inc. as we are still in the process of setting the entity up and yet to complete some necessary filings to be able to do a KYB. In case you need any more information, we're more than happy to provide it.

[sacha-l]

Indeed, adding "could" would be more accurate. The networking layer could be extended to integrate some on-chain verification mechanism an application requires it.

[sacha-l]

I've updated the PR with @thewoodfish's comments here as I think they do add more clarity on what we mean for implementing gossipsub.

[sacha-l]

Updated.

[sacha-l]

I added some more details @thewoodfish wrote in a previous comment into the "Features" section to add more clarity on this deliverable (under "node communication").

[sacha-l]

I added this into the "Ecosystem fit" section:

Examples could include Content Delivery Networks (CDNs); distributed storage systems; IoT (Internet of Things); multiplayer online games; and file hosting and sharing systems

[sacha-l]

As @thewoodfish mentioned, the structure is around first delivering a way to make this library flexible and configurable. Then, delivering a way to make it scalable and usable. We think these building it this way stays closer to our design goals for making this networking layer as flexible and configurable as it can be.

@PieWol we would happily include any suggestions you might have on breaking down the milestones further if you think anything is missing. Note that a lot of this is detailed in the "Features" section which we plan to use as our north start for the deliverables.

[semuelle]

regarding the legal entity, I've updated our submission and removed the mention of Algorealm, Inc. as we are still in the process of setting the entity up and yet to complete some necessary filings to be able to do a KYB.

Hey @sacha-l, thanks for the update. In that case, I have to ask all beneficiaries of the grant to fill out the KYC form. The team name is still listed at the top of the document, by the way.

[sacha-l]

Hey @sacha-l, thanks for the update. In that case, I have to ask all beneficiaries of the grant to fill out the KYC form. The team name is still listed at the top of the document, by the way.

Can confirm that both @thewoodfish and I have completed the KYC process. I've also updated the team name to just Algorealm.

[keeganquigley]

re-approving

[takahser]

re-approving here

[semuelle]

Thanks for the updates, @sacha-l & @thewoodfish. I'm generally happy to support the project. I just have two more questions. The first one, see below. The other: after you applied, we started paying grants partly in vested DOT. Would you be okay with that, and if so, would you like to receive the DOT payment to the AssetHub address listed above, or would you like to add a second one?

[semuelle]

What network are you referring to here?

[thewoodfish]

Thank you. We are talking about the custom network of the application which uses SwarmNL as its underlying backbone. When SwarmNL is in use, you require the benefits it provides, hence you would be configuring your own network as you configure and tweak SwarmNL configurable interfaces.

[semuelle]

I understand. But for the tutorial and testing deliverables, you will have to provide an example. What will this example be based on? SamaritanDB?

Also, could you answer my first question from the top comment?

[thewoodfish]

Okay, i will say no. The goal is to provide a detailed means of understanding for the users/devs. Hence, the example will be based on a very simple easy to understand use-case and model. It is unlikely SamaritanDB will be able to achieve that effectively, so i can say it will be something else, simple to understand e.g a simple Ti-Tac-Toe game. What we have is a goal in mind (the mechanism -> a detailed and easy to understand tutorial, test, example etc.), we will find the best and most effective means (the policy -> which project/use-case/application to use for the tutorials, tests, examples etc) to carry that out.

@sacha-l will respond to your first question shortly. Thank you

[sacha-l]

I see what you're asking @semuelle but it's true that an actual integration to a dummy app is outside the scope of this milestone. For this milestone, our focus is a lot more on the configurability feature set. The envisioned tutorial will outline how to configure the Node Communication features we'll be delivering in this milestone. So for the tutorial its not: "how to configure your nodes for a simple tic-tac-toe game" but rather "what are all the configuration options with SwarmNL you have".

[semuelle]

an actual integration to a dummy app is outside the scope of this milestone

So what functionality will be covered by the testing and Docker deliverables?

[sacha-l]

The testing will cover the functionalities in the deliverables 1 and 2 for Milestone 1. The Docker will contain simple programs (Rust binaries) that will showcase these functionalities. Hope that clarifies things!

[sacha-l]

Hi @semuelle thank you for your recent review and questions. For the payment, it would be best for us to receive the full amount in USDC upon approval of our milestone delivery.

[semuelle]

Thanks for the clarifications, @sacha-l & @thewoodfish. Happy to proceed.

[github-actions]

Congratulations and welcome to the Web3 Foundation Grants Program! Please refer to our Milestone Delivery repository for instructions on how to submit milestones and invoices, our FAQ for frequently asked questions and the support section of our README for more ways to find answers to your questions.

Before you start, take a moment to read through our announcement guidelines for all communications related to the grant or make them known to the right person in your organisation. In particular, please don't announce the grant publicly before at least the first milestone of your project has been approved. At that point or shortly before, you can get in touch with us at [email protected] and we'll be happy to collaborate on an announcement about the work you’re doing.

Lastly, please remember to let us know in case you run into any delays or deviate from the deliverables in your application. You can either leave a comment here or directly request to amend your application via PR. We wish you luck with your project! 🚀

[sacha-l]

Hi @semuelle - I hope you're having a good start to the month. Just wanted to leave an update here that we're in week 6 of our milestone 1 delivery plan and very close to having completed all deliverables for milestone 1. Although we're close, we expect that we'll be ready to submit our deliverables sometime next week. We hope this doesn't cause any trouble. Best!

[semuelle]

Hi @sacha-l. Thanks for the offer. Small deviations from the original timeline are no problem. We'd rather see it done well than done quick.
Have a good weekend!

[sacha-l]

Hi @semuelle - just wanted to give you a quick update. We're still a little behind from the original timeline we'd set ourselves. The request/response protocol implementation took a little longer than expected. Then updating docs, unit tests and integration tests accordingly added a fair amount on our plates over the past week. That being said, we're aiming to submit milestone 1 by the end of this week, hopefully by Friday. We really look forward to getting it in for review from the W3F team. I hope the new timeline is ok on your end. Have a great week!

[sacha-l]

Hi @semuelle - hope you're having a great week. I wanted to share a quick update here. We've been spending the past few weeks rethinking a few things we'd want to amend in our milestone 2 delivery which is a discussion we'd first like to bring to an email thread. Should I use this email [email protected] ? Cheers!

[semuelle]

@sacha-l, please send to grants at web3.foundation. Thanks!