p2p2

Spacemesh p2p design notes

WARNING: This doc is outdated and needs to be updated.

p2p Routing

• Each node has hard-coded list of at least one but most likely up to 5 bootstrap nodes. One bootstrap node must be available for new nodes to join the network.
• When a node starts up, it should use khd routing protocol to add itself to the bootstrap node.
• This will return a list of nodes that are 'close' (under xor arithmetic) to the node. These are effectively random nodes as node ids are random.
• A node should most likely only keep up to n=5 online neighbors from the close peers list at any time and have additional peers on stand-by in case it fails to connect to one of the neighbors.
• Similar ideas are found in libp2p and eth peer routing algorithms.
• As the bootstrap node will add the new node to its routing table, other nodes will try to connect to the new node as it will be close to them at random.
• So joining the network requires only 1 call searchPeer(self) and 1 online bootstrap node.

p2p Networking requirements

• Node Identity - Node id is its public key. There is really no need to derive a token/id from a public key in a way that requires another at least one network roundtrip to find the node's public key. The public key is already unique and public and so is perfectly suited to be used as the id.
• Encryption - end-to-end with forward-security (sym enc keys negotiated per session between nodes)
• Authentication - Receiver node should be able to auth sender node id
• Node Discovery - DHT: random discovery of neighbors for gossip protocol. Node should be able to dial any online node and connect to it.
• Bootstrapping - hard-coded nodes-set - start neighbors walk from this list.
• Base Protocol and multi-protocols - Support app-level protocols framed w base protocol.
• Protocols Multiplexing - Support multi-protocol messages over the same virtual connection.
• Protocols versioning - Support negotiation of protocol version between nodes.
• Gossip Protocol - Allow nodes to gossip over the base protocol

Wire protocol (serialization)

We are going with protobufs as wire format for all data instead of having to write our own wire format (such as eth hand-rolled RLP format). There's great Go support for this format.

p2p protocol support - design

• We have built our own p2p stack above golang tcp/ip stack.

P2P v2.0 design

Spec evolution based on r&d into libp2p, go net libs, and friends.

p2p network architecture

• Local Node
  • Implements p2p protocols: Can process responses to req by ided callbacks.
    • ping protocol
    • echo protocol
    • all app-level other protocols...
• DeMuxer - routes remote requests (and responses) back to protocols. Protocols register on demuxer.
• Swarm forward incoming protocol messages to muxer. Let protocol send message to a remote node. Connects to random nodes.
  • Manages all remote nodes, sessions and connections
  • Handles handshake / session management using the handshake protocol
  • Remote Node - type used internally by swarm to manage sessions and connections
  • RemoteNodeData - the type visible outside of swarm for identifying nodes
• NetworkSession
• Connection
  • msgio (prefix length encoding). shared buffers - used internally for all comm.
• Network
  • tcp for now
  • utp / upd soon

Node IDs and Keys

• Node (and account) ids should be their own public keys - greatly simplify design and tests - no need for key hash. A secp256k1 key is 32 bytes (like a 256bits hash anyhow.)
• Keys system is secp256k1 as it includes encrypt/decrypt
• We might switch to ed25519 based on crypto team review
• We use base58 encoding for string rep of keys (human readable)
• In proto messages keys/ids can be []byte - no need for string encoding /decoding as we are using a binary p2p format - greatly simplifies everything.

smp2p - The SpaceMesh p2p protocol

To libp2p or not to libp2p?

• go-libp2p is not in a great shape: can't use utp, node ids must be multi-hashes (not pub keys), there are serious open issues, kad/dht seems problematic as well.
• go-libp2p should be viewed as a collection of p2p go utils that we can pick and chose from as needed.
• Some of their design decisions are not great and are derived from the ipfs product requirements.

Transport

• tcp for now, udp or utp later

• All

Wire message format

We are using our own simple length-prefix binary format:

<32 bits big-endian data-length><message binary data (protobufs-bin-encoded)>

• We need to use length prefixed protobufs messages becuase protobufs data doesn't include length and we need this to allow multiple messages on the same tcp/ip connection . see p2p2.conn.go.

Network stack design

• 100% locks, mutexes and sync.* free - only using go channel concurrency patterns
• As little external deps as possible but don't rewrite enc code - bad idea.

USwarm - Core P2P Server

• Server should be modular to support different transport protocols (mainly tcp, udp)
• Responsible for establishing sessions with neighbors on startup (connecting to peers and getting neighbors)
• Gets wire-format messages from the transport and decodes them to higher-level protobuf messages and routes them to receivers
• Maintains connections with other peers (for reuse)
• Maintain session info for sessions with other peers. Session data includes an ephemeral session key used for encrypting / decrypting p2p messages.
• Handles sessions disconnections and can create a session with a peer
• Encapsulate session and implements the session protocol
• Supports gossip protocol - sends a message to all neighbors on behalf of clients
• Can reuse non-expiring session ids when connecting to a new node (after disconnection)
• Any message that it sending timestamp is exceeding timesync.MaxAllowedMessageDrift is dropped

Unmarshaled top-level message format:

Session data:

data {
    sessionId: []byte      // if there's an established session between the peers, empty or missing otherwise
    payload: []byte        // encrypted bin data (binary protobuf) with sessoin key
    timestamp: int64       // A unix timestamp of the send time
    ... possible other fields ...
}

• Note that message only includes session id (random screen per session), encrypted data and timestamp and doesn't leak any other data.

• payload is protocol-specific. e.g. session protocol or an app-level protocol.

• for app-level protocol it includes message-author data (might be diff than receiver)

• The p2p server is responsible to decrypt the payload using the local node private key or a session sym key for the session.

• The only connection with incoming data that doesn't have session id is a session protocol message

• All other messages will be rejected

Handshake protocol (session protocol)

• Format: Encrypted protobufs binary data with the destination public key (id)
• Used to establish ephemeral session key between peers
• Used by the core p2p server to establish sessions with remote nodes
• A node will deny a handshake request if :
  • it can't unmrashal the binary packet
  • the timestamp is too far away (like any incoming msg)
  • the clientVersion is lower than our node's nodeconfig.MinClientVersion
  • the networkID of the sender is different the ours.

message HandshakeData {
    bytes sessionId  // for req - same as iv. for response - set to req id
    bytes payload // empty in handshake
    int64 timestamp // sending time
    string clientVersion // client version of the sender
    int32 networkID // network id of sending node
    string protocol // 'handshake/req' || 'handshake/resp'
    bytes nodePubKey // 65 bytes uncompressed
    bytes iv // 16 bytes - AES-256-CBC IV
    bytes pubKey // 65 bytes (uncompressed) ephemeral public key
    bytes hmac // HMAC-SHA-256 32 bytes
    string tcpAddress // ipv4 tcp address and port e.g. x.x.x.x:2424 that the remote node is accepting connections on
    string sign // hex encoded string 32 bytes sign of all above data by node public key (verifies he has the priv key and he wrote the data
}

Higher-level protocols

app-level protocol payload format:

payload: {  // this is the actual protocol message
      metadata // a metadata of the message
      // protocol specific fields (ex: ping )
}

metadata  {
     protocol       // protocol id string
     reqId          // Unique request id. Generated by caller. Returned in responses.
     clientVersion  // Author client version
     timestamp      // Unix time - authoring time (not sending time)
     gossip         // True to have receiver peer gossip the message to its neighbors
     authPubKey     // Authoring node Secp256k1 public key (32bytes) - may not be sender
     authorSign     // Signature of message data by author + method specific data by message     creator node. format: hexEncode([]bytes)
}

• Message is encrypted with session key
• Only non-encrypted part as the session id (server knows which remote node is part of session)
• Message content is protobufs
• Message content include message author authenticated data (may be diff than sender)
• Protocol handlers register callbacks with the core server
• Server authenticate payload was sent by claimed sender
• Server authenticate data was created by claimed author
• Server calls the handler based on protocol / method message data for authenticated messages.
• Server may discard messages with a send timestamp that is too far aparat from local time.

Basic flows

• An app-level p2p protocol wants to send a message to a node and process response.

• Protocol sends message to the core server

• The core server sends the message if it has a session with the remote peer or tries to establish a new session with a peer.

• The response is called back on protocol impl callback.

• App-level protocol can query the server for a list of active sessions or neighbor peers

• App-level protocol may send a gossip message to all neighbors using the core server

Node discovery protocol (higher level kad/dht)

• Nodes discovery is just another app-level protocol - no need to special case it.
• We will implement go-kadh/dht for node discovery - critical piece of the p2p stack.

Protocols discovery protocol

• Allows a node to query all the remote node implement protocols and their versions and decide which protocols to use with the remote node.
• Versioning is on the req processing level to allow updating req/resp for specific methods - no need for a global version for protocol
• Returned data includes:
  • ping/req/v1
  • ping/req/v2
  • hello/req/v1
  • node-discovery/req/v1 Based on this the node knows which req the remote nodes can process and which protocols - in this example, ping and hello.

Session protocol

• Special protocol - used to establish an encrypted session between nodes.

Misc P2p Tasks (move to issues)

1. Periodic ping all known nodes in DHT (and swarm peers) to update latency, remove down peers and update tcp address of peers with changed address (due to ISP public IP rotations). Investigate refresh and ensure it doesn't create too much network traffic too often.
2. Periodic check for node public IP change (ISP issued) and when detected - ping all peers in DHT and all nodes in swarm with the updated public ip address so they can update their swarm and DHT table with the current address.
3. Add current peer public ip address to protocol messages meta-data (both requests and responses). Local node should update the address of the peer in dht and its swarm in case that the public ip address changed.