From 0f655da2d86ac7752ae1d144c44f9937715d43f7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Forn=C3=B3s?= Date: Thu, 14 Sep 2023 16:51:02 +0200 Subject: [PATCH 1/3] Ocelloids XCM Transfer Monitoring Service --- .../ocelloids_xcm_monitoring_service.md | 270 ++++++++++++++++++ 1 file changed, 270 insertions(+) create mode 100644 applications/ocelloids_xcm_monitoring_service.md diff --git a/applications/ocelloids_xcm_monitoring_service.md b/applications/ocelloids_xcm_monitoring_service.md new file mode 100644 index 00000000000..788328cefb5 --- /dev/null +++ b/applications/ocelloids_xcm_monitoring_service.md @@ -0,0 +1,270 @@ +# Ocelloids XCM Transfer Monitoring Service + +- **Team Name:** SO/DA zone +- **Payment Address:** Fiat 04.05.2023, 16:37 +- **[Level](https://github.com/w3f/Grants-Program/tree/master#level_slider-levels):** 2 + +## Project Overview + +This grant proposal is a follow-up to the Ocelloids Monitoring SDK, previously delivered and available here: https://github.com/w3f/Grant-Milestone-Delivery/pull/934 + +The objective of this grant is to develop an open-source monitoring service using the Ocelloids SDK. This service will monitor XCM transfers across Asset Hub and selected parachains. The primary purpose is to offer exchanges, custodians and other institutions on Asset Hub an efficient means of tracking asset transfers across different parachains. Additionally, we will abstract the underlying pallets responsible for initiating XCM transfers and managing assets, which varies for each parachain. Users will have access to an HTTP API to subscribe to XCM transfers and manage their subscriptions. A Docker image will be provided to facilitate service deployment. + +## Project Details + +This service is designed for use alongside an Asset Hub node deployment and the Asset Transfer API. The goal is to provide exchanges, custodians and institutions with an easily deployable and manageable monitoring solution for tracking their asset flows across parachains. The service will also be available for any user interested in monitoring XCM transfers between Asset Hub and parachains. + +The service will support bidirectional XCM transfers between Asset Hub and selected parachains. It is not feasible to provide generic support for all Asset Hub-parachain channels due to differences in pallets used for asset management and XCM message initialization. For asset management, some parachains use FRAME `assets` pallet while others use ORML `tokens` pallet. There are also networks, like Equilibrium, who uses their own custom pallet. For XCM messages, some parachains use Polkadot `xcm` pallet while others use ORML `xtokens` pallet. To keep the project manageable, our initial focus will be on supporting Moonbeam, Astar, and Acala. In the future, we plan to expand support to include more channels and XCM protocols. + +Before using the service, users will need to configure the supported networks and connection providers. An example configuration can be found in the section [Service Configuration](#service-configuration). We will support connections to the network using both RPC clients and light clients, although light client support may be limited due to technical challenges. Further details on supported Substrate clients can be found in the [Supported Substrate Clients](#supported-substrate-clients) section. + +The service will expose an HTTP API for users to add subscriptions, specifying the channels they want to monitor and the accounts involved. Users can create multiple subscriptions and modify or delete them as needed. Users can also set up webhook endpoints to receive notifications related to their transfers. Detailed information about the API will be provided in the [Subscriptions API](#subscriptions-api) section. + +### Notifications + +As previously mentioned, the monitoring service will accept webhook endpoints for delivering notifications. Initially, notifications will be provided for XCM message reception (or the lack thereof). + +The following types of notifications correspond to different scenarios: + +**XCM Execution Success** + +The XCM message sent from the origin chain was received and executed successfully. There are two sub-cases in this scenario: + +1. The received asset amount minus fees matches the sent asset amount, and the accounts where assets are deposited correspond to the beneficiary account in the XCM message. In this case, we will send a "XCM Transfer Success" notification, including contextual data such as the XCM message hash, block numbers of the XCM message on the origin and destination chains, sender account, beneficiary account, transferred asset, amount, and fees paid. + +2. If either the received asset amount or the deposit account on the destination chain does not match the specified values in the XCM message, a "XCM Transfer Executed with Discrepancies" notification will be sent, providing details of the discrepancy and additional contextual data. + +**XCM Execution Fail** + +The XCM message was received at the destination chain but failed to execute correctly. In this case, a "XCM Transfer Execution Failed" notification will be sent, including the XCM error returned in the event and additional contextual data. + +**XCM Message Not Received** + +Users can configure a "maxBlock" field in the subscription to specify the interval, in blocks, that the service should wait before triggering a "XCM Message Not Received" notification if the XCM message is not received on the destination chain within the defined time frame. The subscription will continue to monitor the XCM message in case it arrives later than expected. + +### Service Configuration + +The service will load a configuration file at startup, similar to the example provided below: + +```json +{ + "networks": [ + { + "name": "assethub", + "id": 1000, + "relay": "polkadot", + "provider": { + "type": "ws", + "url": "wss://asethub-rpc" + } + }, + { + "name": "parallel", + "id": 2012, + "relay": "polkadot", + "provider": { + "type": "smoldot" + } + } + ] +} +``` + +Other common service options, such as the listening port for the Subscriptions API, will be configurable using environment variables. + +### Subscriptions API + +#### API Methods + +`POST /api/subscriptions/{id}` + +Creates a new subscription document. To avoid the need for the client to store a server generated identifier, we will require the client to provide a unique subscription identifier. This will also allow the client to use meaningful identifiers e.g. `1000-2004-cohort1`. + +Example request body: + +```json +{ + "subject": "xcm-hrmp-transfers", + "args":{ + "origin": { + "network": "assethub", + "senders": [ + { + "accountId": "5GEWco3EGfM27Z9cAmVnzFnZ6Y7vkqNVvWM4NgQZj4n84Wh6", + "type": "AccountId32" + } + ] + }, + "destination": { + "network": "moonbeam" + }, + "maxBlocks": 500 + }, + "notification": { + "type": "webhook", + "url": "https://my-callback", + "maxRetries": 50 + } +} +``` + +`PATCH /api/subscriptions/{id}` + +Accepts a JSON Patch operations array according to [RFC6902](https://datatracker.ietf.org/doc/html/rfc6902) + +Example: + +```json +[ + { + "op": "add", + "path": "/args/origin/senders/1", + "value": { + "accountId": "5HWSEZr3DQXaN4Tk2Y9pYyAPKWeu28P94qeWWgUZ4k2mrbGB", + "type": "AccountId32" + } + } +] +``` + +The resulting patched document will be validated against the subscription document schema. + +`DELETE /api/subscriptions/{id}` + +Removes a subscription by ID. + +`GET /api/subscriptions` + +Returns the list of subscriptions. + +`GET /api/subscriptions/{id}` + +Returns the subscription document under the specified ID. + +`PUT /api/subscription/{id}` + +Replaces an existing subscription document by ID. + +### Supported Substrate Clients + +We will support WebSocket RPC endpoints and light clients. However, please note that light client support may be limited due to its experimental nature. + +During preliminary testing, we identified some limitations: + +1. Using the Smoldot through Subtrate-Connect only support bootnodes configured with secure WebSocket connection. In our exploration, we've seen that only Astar and Acala have bootnodes with secure WebSocket connections. However, it is easy to overcome this limitation since Smoldot supports all the connection types. Reference: [Github issue](https://github.com/smol-dot/smoldot/issues/1152) + +2. Some runtimes, such as Moonbeam, currently cannot be compiled by the light client. Reference: [Github issue](https://github.com/paritytech/substrate-connect/issues/1543) + +Additional issues may arise, so we plan to provide light client support on a best-effort basis. + +### Storage + +We will provide persistent storage for operational data, including subscription configurations, processed block number and hashes, XCM messages and notifications. We will use [Level](https://github.com/Level/level) as the database abstraction backed by LevelDB. + +### Catch-up Mechanism + +We will implement a mechanism to process missed blocks in case the monitoring service experiences downtime. For each processed block, its hash and block height will be saved in the database. When the service restarts, it will begin following the current finalized blocks. If the parent hash of the finalized block does not match the last-known hash stored in the database, the service will start processing the missing parent hash blocks until all missing ones are processed. + +### Management Tools + +Since notifications and XCM message matching tasks are stored in the database and retried until success, we will provide a method to clear pending states. This is crucial to prevent indefinite retries of pending tasks. For example, if a webhook endpoint is changed while a notification is pending, it may never succeed. + +We will supply a script enabling administrators to inspect and remove XCM messages matching and notification tasks from the database. + +### Tech Stack + +- Typescript +- Node.js +- Polkadot.js API +- Smoldot +- [Fastify](https://github.com/fastify/fastify) +- [Level](https://github.com/Level/level) + +### Out of Scope + +It is important to note that we will only support XCM transfers originating from an extrinsic of the `polkadotXcm` pallet or `xtokens` pallet. Other forms of sending XCM transfers, such as via an `ethereum.transact` call to an XCM contract, will not be supported in this iteration. + +## Ecosystem Fit + +This project aligns well with the growing demand for robust tooling and infrastructure necessary to fulfill Asset Hub's vision of becoming the main place for asset management within the Polkadot ecosystem. To expand its user base, Asset Hub must provide a solution that abstracts away the technical intricacies of cross-chain transfers and enhances the user experience for managing assets across the expansive Polkadot ecosystem. + +The Integrations Team at Parity is already taking initial steps toward achieving this goal by working on the Assets Transfer API and the NoS launch script. These efforts target custodians, exchanges, and institutions, simplifying integration with Asset Hub and the construction of XCM messages. The importance of these tools, along with the overarching vision for Asset Hub, is comprehensively outlined in Iker's [Polkadot forum post](https://forum.polkadot.network/t/new-tooling-and-infrastructure-to-facilitate-the-statemint-roadmap/1997). + +The addition of the Ocelloids XCM monitoring service will enhance this toolkit, providing institutions with a straightforward method to track their cross-chain transfers. This corresponds to step 4 of the "Withdrawals User Journey Example" in Iker's post, as detailed [here](https://forum.polkadot.network/t/new-tooling-and-infrastructure-to-facilitate-the-statemint-roadmap/1997#withdrawals-user-journey-example-5). + +With the inclusion of support for light clients in the service, institutions will experience a substantial reduction in infrastructure overhead since they will no longer need to run parachain nodes to monitor their transfers. + +Furthermore, the demand for a generalized monitoring solution with XCM support has been highlighted in discussions within the Polkadot community, as illustrated in this Polkadot forum post: [Generalized multichain monitoring solution](https://forum.polkadot.network/t/generalized-multichain-monitoring-solution/1297). + +## Team :busts_in_silhouette: + +### Team members + +- Marc Fornós +- Xueying Wang + +### Contact + +- **Contact Name:** Marc Fornós +- **Contact Email:** frozen.pomelo@gmail.com + +### Legal Structure + +N/A + +### Team's experience + +**Marc Fornós** has been designing and implementing software systems for 20 years. He is an expert in the area of distributed systems and data-intensive applications. His experience ranges from warehouse automation with radio-frequency terminals, to being the technical director of an airline post-sale revenue optimization software-as-a-service platform, generating millions in incremental revenue during eight years of operation. In the recent past, he was in charge of evolving a commercial Ethereum block explorer and bootstrapping an explorer for Substrate-based networks focused on the contracts pallet and ink! smart contracts. + +**Xueying Wang** pivoted to software development after completing an MSc. in Aerospace Engineering and has been in the industry for the past eight years. During this time, she pioneered conversational AI assistants for airlines, counting more than 20 assistants in production covering ten languages for customer service, FAQ and in-chat purchases. She also built a scalable publish-subscribe system to trigger actions on flight feed events for the automated agents. She participated in designing a composable Solid POD/RDF data browser and bootstrapping an explorer for Substrate-based networks focused on the contracts pallet and ink! smart contracts. + +### Team Code Repos + +- https://github.com/sodazone +- https://github.com/mfornos +- https://github.com/XY-Wang +- https://github.com/frozenpomelo (shared account) + +## Development Roadmap :nut_and_bolt: + +### Overview + +- **Total Estimated Duration:** 2 months +- **Full-Time Equivalent (FTE):** 2 +- **Total Costs:** 28,000 EUR + +### Milestone 1 + +- **Estimated duration:** 2 months +- **FTE:** 2 +- **Costs:** 28,000 EUR + +| Number | Deliverable | Specification | +| -----: | ----------- | ------------- | +| **0a.** | License | Apache 2.0 | +| **0b.** | Documentation | We will provide both inline documentation of the code and a basic guide that explains how to set up and run the monitoring service. | +| **0c.** | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. | +| **0d.** | Docker | We will provide a Dockerfile to ease the deployment and execution of the service. | +| 1. | XCM Monitoring Service | The XCM monitoring service that supports XCM transfers between Asset Hub and the following 3 parachains: Moonbeam, Astar and Acala. The service will feature what was described in [Project Details](#project-details). | +| 2. | Management Tools | Administrator scripts to inspect and delete pending XCM messages matching and notification tasks, as described in [Management Tools](#management-tools). | + +## Future Plans + +For the XCM monitoring service, we have plans to expand its capabilities and reach which include: + +1. **Support for More Networks:** We plan to broaden the range of networks supported by the XCM monitoring service, enabling a more extensive and inclusive monitoring ecosystem. + +2. **Support for More XCM Protocols:** We will add support for VMP and eventually, XCMP. + +3. **Enhanced Notifications:** Depending on user requirements and community feedback, we will extend our notification capabilities. This may involve providing notifications for asset transfers' initiation, such as when the block containing the XCM transfer is finalized on the origin chain. We are also considering notifications for when XCM messages sent through HRMP are processed on intermediate chains. + +Our long-term vision for Ocelloids extends beyond just monitoring XCM transfers. We aim to create a hassle-free, comprehensive monitoring portal for Substrate networks and smart contracts within the ecosystem. This portal will offer a set of advanced features, including: + +- Ready-to-use monitoring logic, i.e. monitoring programs, for the entire ecosystem. The XCM monitoring logic built through this grant could be one such monitoring program available. +- Marketplace for monitoring programs where users can subscribe to the programs that correspond to their needs. +- Advanced capabilities: anomaly and attack detection, machine learning, and forecasting. +- Real-time actionable insights on network, contract performance, compliance, and security. +- Multiple data centres for global, high-availability monitoring. +- Seamless integration with existing systems like Prometheus and SIEM. + +In summary, our future plans encompass not only expanding the technical capabilities of the XCM monitoring service but also positioning as a central hub for monitoring and managing Substrate networks and smart contracts within the broader multi-chain ecosystem. From 71fedb23efa81d9c630d708295647cb878b911a5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Forn=C3=B3s?= Date: Mon, 18 Sep 2023 12:53:12 +0200 Subject: [PATCH 2/3] update ocelloids_xcm_monitoring_service.md --- .../ocelloids_xcm_monitoring_service.md | 46 +++++++++---------- 1 file changed, 21 insertions(+), 25 deletions(-) diff --git a/applications/ocelloids_xcm_monitoring_service.md b/applications/ocelloids_xcm_monitoring_service.md index 788328cefb5..aa1eb51c433 100644 --- a/applications/ocelloids_xcm_monitoring_service.md +++ b/applications/ocelloids_xcm_monitoring_service.md @@ -8,18 +8,25 @@ This grant proposal is a follow-up to the Ocelloids Monitoring SDK, previously delivered and available here: https://github.com/w3f/Grant-Milestone-Delivery/pull/934 -The objective of this grant is to develop an open-source monitoring service using the Ocelloids SDK. This service will monitor XCM transfers across Asset Hub and selected parachains. The primary purpose is to offer exchanges, custodians and other institutions on Asset Hub an efficient means of tracking asset transfers across different parachains. Additionally, we will abstract the underlying pallets responsible for initiating XCM transfers and managing assets, which varies for each parachain. Users will have access to an HTTP API to subscribe to XCM transfers and manage their subscriptions. A Docker image will be provided to facilitate service deployment. +The objective of this grant is to develop an open-source monitoring service using the Ocelloids SDK. This service will monitor XCM transfers across selected parachains. The primary purpose is to offer service providers integrating with a single chain (Asset Hub as starting point) and monitoring effects on other chains that are connected via HRMP and that use XCM as their message format. The service will support connecting to the configured networks through light clients in order to reduce infrastructure overhead for service providers. Users will have access to a self-hosted HTTP API to subscribe to XCM transfers and manage their subscriptions. A public Docker image will be published to facilitate service deployment. ## Project Details -This service is designed for use alongside an Asset Hub node deployment and the Asset Transfer API. The goal is to provide exchanges, custodians and institutions with an easily deployable and manageable monitoring solution for tracking their asset flows across parachains. The service will also be available for any user interested in monitoring XCM transfers between Asset Hub and parachains. +The service will support bidirectional XCM reserve-based transfers between selected parachains. -The service will support bidirectional XCM transfers between Asset Hub and selected parachains. It is not feasible to provide generic support for all Asset Hub-parachain channels due to differences in pallets used for asset management and XCM message initialization. For asset management, some parachains use FRAME `assets` pallet while others use ORML `tokens` pallet. There are also networks, like Equilibrium, who uses their own custom pallet. For XCM messages, some parachains use Polkadot `xcm` pallet while others use ORML `xtokens` pallet. To keep the project manageable, our initial focus will be on supporting Moonbeam, Astar, and Acala. In the future, we plan to expand support to include more channels and XCM protocols. +The flow of the monitoring will work as follows: -Before using the service, users will need to configure the supported networks and connection providers. An example configuration can be found in the section [Service Configuration](#service-configuration). We will support connections to the network using both RPC clients and light clients, although light client support may be limited due to technical challenges. Further details on supported Substrate clients can be found in the [Supported Substrate Clients](#supported-substrate-clients) section. +1. The service will monitor on the origin chain for the event `xcmpqueue.xcmpMessageSent` associated to the extrinsic sent by accounts of interest. The service will extract the XCM message hash from this event. +3. The service will query the storage in `parachainSystem.hrmpOutboundMessages` at the block of the event to get all outbound messages and filter for recipient chain IDs that are supported. Subsequently it will decode the message data to get the set of XCM instructions to filter for combinations of instructions related to reserve-based transfers (i.e. `ReserveAssetDeposited`, `WithdrawAsset`, and `DepositAsset`). Then, it will get the `blake2-256` hash of the message data to match it with the message hash obtained in Step 1. The service will store a persistent task to be matched in subsequent steps. +4. At the destination chain, the service will monitor for the events `xcmpqueue.success` and `xcmpqueue.fail`. It will match the message hash extracted from these events with the message hash of the origin. +5. It will send a notification to the configured webhook to inform of the status of the XCM transfer along with contextual information. See section [Notifications](#notifications) for details. + +Before using the service, users will need to configure the supported networks and connection providers. An example configuration can be found in the section [Service Configuration](#service-configuration). We will support connections to the network using both RPC clients and light clients. Further details on supported Substrate clients can be found in the [Supported Substrate Clients](#supported-substrate-clients) section. The service will expose an HTTP API for users to add subscriptions, specifying the channels they want to monitor and the accounts involved. Users can create multiple subscriptions and modify or delete them as needed. Users can also set up webhook endpoints to receive notifications related to their transfers. Detailed information about the API will be provided in the [Subscriptions API](#subscriptions-api) section. +To keep the project manageable, the current scope includes support for Asset Hub, Astar and Acala. In the future, we plan to expand support to include more chains. + ### Notifications As previously mentioned, the monitoring service will accept webhook endpoints for delivering notifications. Initially, notifications will be provided for XCM message reception (or the lack thereof). @@ -28,19 +35,11 @@ The following types of notifications correspond to different scenarios: **XCM Execution Success** -The XCM message sent from the origin chain was received and executed successfully. There are two sub-cases in this scenario: - -1. The received asset amount minus fees matches the sent asset amount, and the accounts where assets are deposited correspond to the beneficiary account in the XCM message. In this case, we will send a "XCM Transfer Success" notification, including contextual data such as the XCM message hash, block numbers of the XCM message on the origin and destination chains, sender account, beneficiary account, transferred asset, amount, and fees paid. - -2. If either the received asset amount or the deposit account on the destination chain does not match the specified values in the XCM message, a "XCM Transfer Executed with Discrepancies" notification will be sent, providing details of the discrepancy and additional contextual data. +The XCM message sent from the origin chain was received and executed successfully. In this case, we will send an "XCM Transfer Success" notification, including contextual data such as the XCM message hash, block numbers of the XCM message on the origin and destination chains, sender account, beneficiary account, transferred asset and amount. **XCM Execution Fail** -The XCM message was received at the destination chain but failed to execute correctly. In this case, a "XCM Transfer Execution Failed" notification will be sent, including the XCM error returned in the event and additional contextual data. - -**XCM Message Not Received** - -Users can configure a "maxBlock" field in the subscription to specify the interval, in blocks, that the service should wait before triggering a "XCM Message Not Received" notification if the XCM message is not received on the destination chain within the defined time frame. The subscription will continue to monitor the XCM message in case it arrives later than expected. +The XCM message was received at the destination chain but failed to execute correctly. In this case, an "XCM Transfer Fail" notification will be sent, including the XCM error returned in the event and additional contextual data. ### Service Configuration @@ -98,7 +97,6 @@ Example request body: "destination": { "network": "moonbeam" }, - "maxBlocks": 500 }, "notification": { "type": "webhook", @@ -155,7 +153,7 @@ During preliminary testing, we identified some limitations: 2. Some runtimes, such as Moonbeam, currently cannot be compiled by the light client. Reference: [Github issue](https://github.com/paritytech/substrate-connect/issues/1543) -Additional issues may arise, so we plan to provide light client support on a best-effort basis. +We will prioritize chains that are light client ready. ### Storage @@ -169,7 +167,7 @@ We will implement a mechanism to process missed blocks in case the monitoring se Since notifications and XCM message matching tasks are stored in the database and retried until success, we will provide a method to clear pending states. This is crucial to prevent indefinite retries of pending tasks. For example, if a webhook endpoint is changed while a notification is pending, it may never succeed. -We will supply a script enabling administrators to inspect and remove XCM messages matching and notification tasks from the database. +We will supply a script enabling administrators to inspect and remove XCM message matching and notification tasks from the database. ### Tech Stack @@ -180,10 +178,6 @@ We will supply a script enabling administrators to inspect and remove XCM messag - [Fastify](https://github.com/fastify/fastify) - [Level](https://github.com/Level/level) -### Out of Scope - -It is important to note that we will only support XCM transfers originating from an extrinsic of the `polkadotXcm` pallet or `xtokens` pallet. Other forms of sending XCM transfers, such as via an `ethereum.transact` call to an XCM contract, will not be supported in this iteration. - ## Ecosystem Fit This project aligns well with the growing demand for robust tooling and infrastructure necessary to fulfill Asset Hub's vision of becoming the main place for asset management within the Polkadot ecosystem. To expand its user base, Asset Hub must provide a solution that abstracts away the technical intricacies of cross-chain transfers and enhances the user experience for managing assets across the expansive Polkadot ecosystem. @@ -244,8 +238,8 @@ N/A | **0a.** | License | Apache 2.0 | | **0b.** | Documentation | We will provide both inline documentation of the code and a basic guide that explains how to set up and run the monitoring service. | | **0c.** | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. | -| **0d.** | Docker | We will provide a Dockerfile to ease the deployment and execution of the service. | -| 1. | XCM Monitoring Service | The XCM monitoring service that supports XCM transfers between Asset Hub and the following 3 parachains: Moonbeam, Astar and Acala. The service will feature what was described in [Project Details](#project-details). | +| **0d.** | Docker | We will provide a Dockerfile to ease the deployment and execution of the service. A Docker image of the service will be published in Docker Hub and Github Container Repository. | +| 1. | XCM Monitoring Service | The XCM monitoring service that supports XCM reserve-based transfers between the following parachains: Asset Hub, Astar and Acala. The service will feature what was described in [Project Details](#project-details). | | 2. | Management Tools | Administrator scripts to inspect and delete pending XCM messages matching and notification tasks, as described in [Management Tools](#management-tools). | ## Future Plans @@ -253,10 +247,12 @@ N/A For the XCM monitoring service, we have plans to expand its capabilities and reach which include: 1. **Support for More Networks:** We plan to broaden the range of networks supported by the XCM monitoring service, enabling a more extensive and inclusive monitoring ecosystem. + +2. **Support for More XCM Instructions:** We plan to support more XCM instructions, such as the ones related to asset teleport. -2. **Support for More XCM Protocols:** We will add support for VMP and eventually, XCMP. +3. **Support for More XCM Protocols:** We will add support for XCMP when ready. -3. **Enhanced Notifications:** Depending on user requirements and community feedback, we will extend our notification capabilities. This may involve providing notifications for asset transfers' initiation, such as when the block containing the XCM transfer is finalized on the origin chain. We are also considering notifications for when XCM messages sent through HRMP are processed on intermediate chains. +4. **Enhanced Notifications:** Depending on user requirements and community feedback, we will extend our notification capabilities. This may involve providing notifications for asset transfers' initiation, such as when the block containing the XCM transfer is finalized on the origin chain. We are also considering notifications for when XCM messages sent through HRMP are processed on intermediate chains. Our long-term vision for Ocelloids extends beyond just monitoring XCM transfers. We aim to create a hassle-free, comprehensive monitoring portal for Substrate networks and smart contracts within the ecosystem. This portal will offer a set of advanced features, including: From 8fbc6d3b6bc6e31c3228fb9e1a52c7b722943a91 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Forn=C3=B3s?= Date: Tue, 19 Sep 2023 13:15:02 +0200 Subject: [PATCH 3/3] update ocelloids_xcm_monitoring_service.md --- .../ocelloids_xcm_monitoring_service.md | 24 +++++++++---------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/applications/ocelloids_xcm_monitoring_service.md b/applications/ocelloids_xcm_monitoring_service.md index aa1eb51c433..9ddc44b43b3 100644 --- a/applications/ocelloids_xcm_monitoring_service.md +++ b/applications/ocelloids_xcm_monitoring_service.md @@ -12,14 +12,14 @@ The objective of this grant is to develop an open-source monitoring service usin ## Project Details -The service will support bidirectional XCM reserve-based transfers between selected parachains. +The service will support bidirectional XCM transfers, namely asset teleports and reserve-based transfers, between selected parachains. The flow of the monitoring will work as follows: 1. The service will monitor on the origin chain for the event `xcmpqueue.xcmpMessageSent` associated to the extrinsic sent by accounts of interest. The service will extract the XCM message hash from this event. -3. The service will query the storage in `parachainSystem.hrmpOutboundMessages` at the block of the event to get all outbound messages and filter for recipient chain IDs that are supported. Subsequently it will decode the message data to get the set of XCM instructions to filter for combinations of instructions related to reserve-based transfers (i.e. `ReserveAssetDeposited`, `WithdrawAsset`, and `DepositAsset`). Then, it will get the `blake2-256` hash of the message data to match it with the message hash obtained in Step 1. The service will store a persistent task to be matched in subsequent steps. -4. At the destination chain, the service will monitor for the events `xcmpqueue.success` and `xcmpqueue.fail`. It will match the message hash extracted from these events with the message hash of the origin. -5. It will send a notification to the configured webhook to inform of the status of the XCM transfer along with contextual information. See section [Notifications](#notifications) for details. +2. The service will query the storage in `parachainSystem.hrmpOutboundMessages` at the block of the event to get all outbound messages and filter for recipient chain IDs that are supported. Subsequently it will decode the message data to get the set of XCM instructions to filter for combinations of instructions related to asset teleports or reserve-based transfers (i.e. `ReserveAssetDeposited`, `ReceiveTeleportedAsset`, `WithdrawAsset`, and `DepositAsset`). Then, it will get the `blake2-256` hash of the message data to match it with the message hash obtained in Step 1. The service will store a persistent task to be matched in subsequent steps. +3. At the destination chain, the service will monitor for the events `xcmpqueue.success` and `xcmpqueue.fail`. It will match the message hash extracted from these events with the message hash of the origin. +4. It will send a notification to the configured webhook to inform of the status of the XCM transfer along with contextual information. See section [Notifications](#notifications) for details. Before using the service, users will need to configure the supported networks and connection providers. An example configuration can be found in the section [Service Configuration](#service-configuration). We will support connections to the network using both RPC clients and light clients. Further details on supported Substrate clients can be found in the [Supported Substrate Clients](#supported-substrate-clients) section. @@ -29,7 +29,7 @@ To keep the project manageable, the current scope includes support for Asset Hub ### Notifications -As previously mentioned, the monitoring service will accept webhook endpoints for delivering notifications. Initially, notifications will be provided for XCM message reception (or the lack thereof). +As previously mentioned, the monitoring service will accept webhook endpoints for delivering notifications. Initially, notifications will be provided for XCM message reception. The following types of notifications correspond to different scenarios: @@ -149,9 +149,9 @@ We will support WebSocket RPC endpoints and light clients. However, please note During preliminary testing, we identified some limitations: -1. Using the Smoldot through Subtrate-Connect only support bootnodes configured with secure WebSocket connection. In our exploration, we've seen that only Astar and Acala have bootnodes with secure WebSocket connections. However, it is easy to overcome this limitation since Smoldot supports all the connection types. Reference: [Github issue](https://github.com/smol-dot/smoldot/issues/1152) - -2. Some runtimes, such as Moonbeam, currently cannot be compiled by the light client. Reference: [Github issue](https://github.com/paritytech/substrate-connect/issues/1543) +1. Using the Smoldot through Subtrate-Connect only support bootnodes configured with secure WebSocket connection. In our exploration, we've seen that only Astar and Acala have bootnodes with secure WebSocket connections. However, it is easy to overcome this limitation since Smoldot supports all the connection types. Reference: [Github issue](https://github.com/paritytech/substrate-connect/issues/1543) + +2. Some runtimes, such as Moonbeam, currently cannot be compiled by the light client. Reference: [Github issue](https://github.com/smol-dot/smoldot/issues/1152) We will prioritize chains that are light client ready. @@ -239,7 +239,7 @@ N/A | **0b.** | Documentation | We will provide both inline documentation of the code and a basic guide that explains how to set up and run the monitoring service. | | **0c.** | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. | | **0d.** | Docker | We will provide a Dockerfile to ease the deployment and execution of the service. A Docker image of the service will be published in Docker Hub and Github Container Repository. | -| 1. | XCM Monitoring Service | The XCM monitoring service that supports XCM reserve-based transfers between the following parachains: Asset Hub, Astar and Acala. The service will feature what was described in [Project Details](#project-details). | +| 1. | XCM Monitoring Service | The XCM monitoring service that supports asset teleports and reserve-based transfers between the following parachains: Asset Hub, Astar and Acala. The service will feature what was described in [Project Details](#project-details). | | 2. | Management Tools | Administrator scripts to inspect and delete pending XCM messages matching and notification tasks, as described in [Management Tools](#management-tools). | ## Future Plans @@ -248,11 +248,9 @@ For the XCM monitoring service, we have plans to expand its capabilities and rea 1. **Support for More Networks:** We plan to broaden the range of networks supported by the XCM monitoring service, enabling a more extensive and inclusive monitoring ecosystem. -2. **Support for More XCM Instructions:** We plan to support more XCM instructions, such as the ones related to asset teleport. - -3. **Support for More XCM Protocols:** We will add support for XCMP when ready. +2. **Support for More XCM Protocols:** We will add support for XCMP when ready. -4. **Enhanced Notifications:** Depending on user requirements and community feedback, we will extend our notification capabilities. This may involve providing notifications for asset transfers' initiation, such as when the block containing the XCM transfer is finalized on the origin chain. We are also considering notifications for when XCM messages sent through HRMP are processed on intermediate chains. +3. **Enhanced Notifications:** Depending on user requirements and community feedback, we will extend our notification capabilities. This may involve providing notifications for asset transfers' initiation, such as when the block containing the XCM transfer is finalized on the origin chain. We are also considering notifications for when XCM messages sent through HRMP are processed on intermediate chains. Our long-term vision for Ocelloids extends beyond just monitoring XCM transfers. We aim to create a hassle-free, comprehensive monitoring portal for Substrate networks and smart contracts within the ecosystem. This portal will offer a set of advanced features, including: