Define ping/pong payload extensions

beacon-chain/beacon-network.md

@@ -76,12 +76,7 @@ The beacon chain network supports the following protocol messages:
 
 #### `Ping.custom_data` & `Pong.custom_data`
 
-In the beacon chain network the `custom_payload` field of the `Ping` and `Pong` messages is the serialization of an SSZ Container specified as `custom_data`:
-
-```python
-custom_data = Container(data_radius: uint256)
-custom_payload = serialize(custom_data)
-```
+In the beacon chain network the `custom_payload` field of the `Ping` and `Pong` messages is the serialization of an SSZ Container specified as [Type 3 Ping Custom Payload Extension](../ping-payload-extensions/extensions/type-3.md)
 
 ### Routing Table
 

canonical-transaction-index/canonical-transaction-index-network.md

@@ -56,12 +56,7 @@ The canonical transaction index network supports the following protocol messages
 
 #### `Ping.custom_data` & `Pong.custom_data`
 
-In the canonical transaction index network the `custom_payload` field of the `Ping` and `Pong` messages is the serialization of an SSZ Container specified as `custom_data`:
-
-```
-custom_data = Container(data_radius: uint256)
-custom_payload = SSZ.serialize(custom_data)
-```
+In the canonical transaction index network the `custom_payload` field of the `Ping` and `Pong` messages is the serialization of an SSZ Container specified as [Type 4 Ping Custom Payload Extension](../ping-payload-extensions/extensions/type-4.md)
 
 
 ### Routing Table 

history/history-network.md

@@ -61,12 +61,8 @@ The history network supports the following protocol messages:
 
 #### `Ping.custom_data` & `Pong.custom_data`
 
-In the history network the `custom_payload` field of the `Ping` and `Pong` messages is the serialization of an SSZ Container specified as `custom_data`:
+In the history network the `custom_payload` field of the `Ping` and `Pong` messages is the serialization of an SSZ Container specified as [Type 1 Ping Custom Payload Extension](../ping-payload-extensions/extensions/type-1.md)
 
-```python
-custom_data = Container(data_radius: uint256)
-custom_payload = SSZ.serialize(custom_data)
-```
 
 ### Routing Table
 

ping-payload-extensions/extensions.md

@@ -0,0 +1,14 @@
+# This document lists defined extensions
+Some extensions are network specific some are generic
+
+This is a list and short description of all the extensions
+
+
+| Type number  |  Name | Supported sub-networks  | Short Description  | Is this call Required to Implement  |
+|---|---|---|---|---|
+| [1](extensions/type-1.md)  |  History Distance | History  | Allows you to get the radius and other nodes metadata  |  Yes  |
+| [2](extensions/type-2.md)  | State Distance  |  State | Allows you to get the state nodes radius  |  Yes  |
+| [3](extensions/type-3.md)  | Beacon Distance  |  Beacon | Allows you to get the beacon nodes radius  |  Yes  |
+| [4](extensions/type-4.md)  | Portal Canonical Transaction Distance  | Portal Canonical Transaction | Allows you to get the Portal Canonical Transaction nodes radius  |  Yes  |
+| [5](extensions/type-5.md)  | Transaction Gossip Distance  |  Transaction Gossip | Allows you to get the Transaction Gossip nodes radius  |  Yes  |
+| [10](extensions/type-10.md)  |  Client Info | All  | It will return you something like `trin/0.1.1-2b00d730/linux-x86_64/rustc1.81.0`  |  Yes  |

ping-payload-extensions/extensions/template.md

@@ -0,0 +1,25 @@
+# [Title]
+
+[Description]
+
+## version [Version Number]
+
+[Version Description]
+
+Ping payload
+```python
+[Container Name] = Container(
+  type: [Type Number],
+  version: [Type Version],
+  payload: Container([Version Description])
+)
+```
+
+Pong payload
+```python
+[Container Name] = Container(
+  type: [Type Number],
+  version: [Type Version],
+  payload: Container([Version Description])
+)
+```

ping-payload-extensions/extensions/type-1.md

@@ -0,0 +1,24 @@
+# Portal History Network Distance
+
+This payload is only supposed to be used for the history network
+
+## version 1
+
+Ping and Pong Payload
+```python
+HistoryNetworkDistanceV1 = Container(
+  type: 1,
+  version: 1,
+  payload: Container(data_radius: uint256)
+)
+```
+
+## version 2
+Ping and Pong Payload
+```python
+HistoryNetworkDistanceV2 = Container(
+  type: 1,
+  version: 2,
+  payload: Container(data_radius: uint256, ephemeral_header_count: uint16)
+)
+```

ping-payload-extensions/extensions/type-10.md

@@ -0,0 +1,34 @@
+# Get Client info
+
+This payload is only supposed to be usable from all Networks. This payload allows you to figure out peer's client info. Which will be useful for censuses.
+
+## version 1
+
+### Type Specifications
+
+Client info are ASCII hex encoded strings.
+
+Client info strings consist of 4 parts
+- client name (e.x. `trin`,`fluffy`)
+- client version + short commit (e.x. `0.1.1-2b00d730`)
+- operating system + cpu archtecture (e.x. `linux-x86_64`)
+- programming language + language version (e.x. `rustc1.81.0`)
+
+Example 
+- String: `trin/0.1.1-2b00d730/linux-x86_64/rustc1.81.0`
+- Hex encoding: `0x7472696E2F302E312E312D32623030643733302F6C696E75782D7838365F36342F7275737463312E38312E30`
+
+
+Max length of a client info we should accept
+- MAX_CLIENT_INFO_BYTE_LENGTH = 200
+
+
+Ping and Pong payload
+```python
+
+HistoryNetworkDistanceV1 = Container(
+  type: 10,
+  version: 1,
+  payload: Container(client_info: ByteList[MAX_CLIENT_INFO_BYTE_LENGTH])
+)
+```

ping-payload-extensions/extensions/type-2.md

@@ -0,0 +1,14 @@
+# Portal State Network Distance
+
+This payload is only supposed to be used for the state network
+
+## version 1
+
+Ping and Pong payload
+```python
+StateNetworkDistanceV1 = Container(
+  type: 2,
+  version: 1,
+  payload: Container(data_radius: uint256)
+)
+```

ping-payload-extensions/extensions/type-3.md

@@ -0,0 +1,14 @@
+# Portal Beacon Network Distance
+
+This payload is only supposed to be used for the beacon network
+
+## version 1
+
+Ping and Pong payload
+```python
+BeaconNetworkDistanceV1 = Container(
+  type: 3,
+  version: 1,
+  payload: Container(data_radius: uint256)
+)
+```

ping-payload-extensions/extensions/type-4.md

@@ -0,0 +1,14 @@
+# Portal Canonical Transaction Index Network Distance
+
+This payload is only supposed to be used for the Canonical Transaction Index Network
+
+## version 1
+
+Ping and Pong payload
+```python
+CanonicalTransactionIndexNetworkDistanceV1 = Container(
+  type: 4,
+  version: 1,
+  payload: Container(data_radius: uint256)
+)
+```

ping-payload-extensions/extensions/type-5.md

@@ -0,0 +1,14 @@
+# Portal Transaction Gossip Network Distance
+
+This payload is only supposed to be used for the Transaction Gossip network
+
+## version 1
+
+Ping and Pong payload
+```python
+Transaction GossipNetworkDistanceV1 = Container(
+  type: 5,
+  version: 1,
+  payload: Container(data_radius: uint256)
+)
+```

ping-payload-extensions/ping-custom-payload-extensions.md

@@ -0,0 +1,67 @@
+# Ping Custom Payload Extensions
+
+## The problem
+
+Over time we will need to change what is sent over ping/pong `custom_payload`'s to adjust for what is needed in the future.
+Currently the only way to update a `custom_payload` is through breaking changes which break protocol compatible between Portal implementations.
+As we get users, deploying breaking changes to mainnet will no longer be an option as we are aiming for a 100% uptime.
+
+## The Solution
+
+Ping Custom Payload Extensions. A versioned type prefixed format where we can upgrade endpoints on a new version well giving a window for all Portal implementations to update before deprecating the older version. This will allow clients to implement new functionality without breaking compatibility with the standard specification.
+
+# Type's
+
+There are 4_294_967_296 unique type ids.
+
+Types 0-10_000 and 4_294_957_295-4_294_967_296 are reserved for for future upgrades.
+
+The rest are first come first serve, but they should still be defined in this repo to avoid overlaps.
+
+
+## Requirements
+
+All payloads used in the Ping `custom_payload` MUST follow the `Ping Custom Payload Extensions` format.
+
+## Custom Payload Extensions Format
+
+- **type**: what payload type is it
+- **verison**: what version of the type it is
+- **payload**: a ssz container which contents are specified by the type and version field
+
+
+```python
+CustomPayloadExtensionsFormat = Container(
+  type: Bytes4,
+  version: Bytes4,
+  payload: Container(inner payload is defined by type and version)
+)
+```
+
+## Ping vs Pong
+The relationship between Ping and Pong message will be determined by the requirements of the type/version.
+
+Currently type 1,2,3 are mirrored formats, but there is room for a Ping `custom_payload` to specify details about what it wants to request, then pong handles it.
+
+
+### Error responses
+If the ping receiver can't handle the ping for any reason the pong should return an error payload
+
+Pong payload
+```python
+
+# Max ASCII hex encoded strings length
+MAX_ERROR_BYTE_LENGTH = 300
+
+CustomPayloadExtensionsFormat = Container(
+  type: 4_294_967_295,
+  version: 1,
+  payload: Container(error_code: Bytes4, message: ByteList[MAX_ERROR_BYTE_LENGTH])
+)
+```
+
+### Error Code's
+
+- 0: Extension not supported
+- 1: Requested data not found
+- 2: System error

state/state-network.md

@@ -56,12 +56,7 @@ The execution state network supports the following protocol messages:
 
 #### `Ping.custom_data` & `Pong.custom_data`
 
-In the execution state network the `custom_payload` field of the `Ping` and `Pong` messages is the serialization of an SSZ Container specified as `custom_data`:
-
-```
-custom_data = Container(data_radius: uint256)
-custom_payload = SSZ.serialize(custom_data)
-```
+In the execution state network the `custom_payload` field of the `Ping` and `Pong` messages is the serialization of an SSZ Container specified as [Type 2 Ping Custom Payload Extension](../ping-payload-extensions/extensions/type-2.md)
 
 #### POKE Mechanism
 

transaction-gossip/transaction-gossip.md

@@ -26,19 +26,7 @@ The network uses the PING/PONG/FINDNODES/FOUNDNODES/OFFER/ACCEPT messages from t
 
 The network uses the standard XOR distance function.
 
-### PING payload
-
-```
-Ping.custom_payload := ssz_serialize(custom_data)
-custom_data         := Container(transaction_radius: uint256)
-```
-
-### PONG payload
-
-```
-Pong.custom_payload := ssz_serialize(custom_data)
-custom_data         := Container(transaction_radius: uint256)
-```
+In the execution state network the `custom_payload` field of the `Ping` and `Pong` messages is the serialization of an SSZ Container specified as [Type 5 Ping Custom Payload Extension](../ping-payload-extensions/extensions/type-5.md)
 
 ## Content Keys