P2P Architecture

Package `p2p`

WARNING: This doc is outdated and needs to be updated.

Description

p2p package exposes minimal peer-to-peer functionality as a service to other components in Spacemesh. Any protocol in Spacemesh can send a direct p2p message, broadcast to all peers or register a callback to handle incoming messages of a specified type.

Public API

Interface Service -

An interface that represents the methods required for a p2p service.
Interface Message -

An interface that represents a message that will be sent or received by Service's methods.
New(p2p.config) -

Creates a new swarm, swarm satisfies the Service interface.

NOTE : Service and Message are declared in p2p/service but re-declared and exposed in p2p to expose them out of the p2p package easily.

type Service interface {
   SendMessage(nodeID string, protocol string, payload []byte) error
   RegisterProtocol(Protocol string) chan Message
   Shutdown()
}

type Message interface {
   Data() []byte
   Sender() node.Node
}

Struct `Swarm`

Description

When a message is received, Swarm unpack the message from its binary form to a structured format which holds basic data about the message as Timestamp, ClientVersion, AuthorSign and Protocol. Swarm uses this metadata to check the validity of the message, after the message passed this check, it sends the payload to the registered protocol. Any message payload that is sent by Swarm is packed with this format exactly and sent using a net.Connection.

Protocols are registered to a Swarm in order to receive messages holding their protocol name.

Public API

SendMessage(nodeID string, protocol string, payload []byte) error

Sends a direct P2P message to a specific nodeID. When a message is sent using Swarm.SendMessage, Swarm will find the target peer in DHT and ask ConnectionPool for a secured connection with that peer. Once the connection is achieved it will add a header to the payload and send the message to the remote Swarm would be able to open it, read this metadata, validate signatures and route the message to the registered protocols. NOTE : this function will block and might take time to lookup the node or create the connection.
Broadcast(protocol string, payload []byte) - Disseminates a message to the node's neighbors via the gossip protocol. check out Gossip Protocol
RegisterProtocol(Protocol string) chan Message -

Associating a protocol with a channel, all messages carrying the protocol name in the header will be delivered on the returned channel and will be structured according to the Message interface.
Bootstrap() error -

Bootstrap starts by registering the pre-configurated bootstrap nodes, query them according to the FindNode protocol for the local node and continue doing the same with the results nodes until there are no new results or until the routing table is full. NOTE: Bootstrap() blocks until finished or returns an error, but it should be used as a background process

Message `Metadata`

message Metadata {
  string protocol = 1;      // Protocol id string
  bytes reqId = 2;          // Unique request id. Generated by caller. Returned in responses.
  string clientVersion = 3; // Author client version
  int64 timestamp = 4;      // Unix time - authoring time (not sending time)
  bool gossip = 5;          // True to have receiver peer gossip the message to its neighbors
  bytes authPubKey = 6;     // Authoring node Secp256k1 public key (32bytes) - may not be sender
  string authorSign = 7;    // Signature of message data by author + method specific data by message creator node. format: hex-encoded bytes
}

Dependencies

crypto
log
p2p/net
p2p/timesync
p2p/identity
p2p/dht

Package `p2p/node`

Description

Package node implements containers that hold basic information about nodes in the network. the most basic struct is Node, it also implements LocalNode and DhtID.

Public API

New(key crypto.PublicKey, address string) Node -

Creates a basic node identity from a PublicKey and an IP address.
NewNodeFromString(data string) (Node, error) -

Tries to break a node string descriptor into a Node. example string: 127.0.0.1:3572/r9gJRWVB9JVPap2HKnduoFySvHtVTfJdQ4WG8DriUD82
StringFromNode(node Node) string -

Creates the above format string from a Node
Union(list1 []Node, list2 []Node) []Node -

Combine two Node slices without duplicates.
SortByDhtID(nodes []Node, id DhtID) []Node -

Sort a node slice by DhtID.
NewLocalNode(Address string, config nodeconfig, persist bool) (*LocalNode, error) -

Tries to create a node from an existing file, if it can't be found it generates a new node with NewNodeIdentity, persist boolean - save the node to the disk or not.
`NewNodeIdentity(config config.Config, address string, persist bool) (*LocalNode, error) -

Generates a private-public key pair and creates a LocalNode from it. persist boolean - save the node to the disk or not.
NewDhtID(key []byte) DhtID -

Creates a DhtID from a PublicKey byte array
NewDhtIDFromBase58(s string) DhtID -

Creates a DhtID from a base58 encoded PublicKey
NewDhtIDFromHex(s string) (DhtID, error) -

Creates a DhtID from an hex encoded string.

Struct `Node`

Description

A basic identity node struct that is included in more complicated structs to implement basic identity features.

Public API -

PublicKey() crypto.PublicKey -

The node's public key
DhtID() DhtID -

A DhtID created from the node's ID
String() string -

An hex-encoded string representation of the node ID. Implements go's Stringer interface.
Address() string -

The Node's IP address

Struct `LocalNode`

Description

LocalNode is a struct that extends Node and log.Log and represents the local node's identity in the network. Every p2p operation happens with a LocalNode identity at its source. It holds a Private and Public key. LocalNode can be initialised from scratch or from existing persistent keys and files. It needs a local address and port. PublicKey is the node's identifier in the network.

Public API

All of Node's methods.
PrivateKey() crypto.PrivateKey -

The node's Private Key used for signing messages.
NetworkID() int8 -

The NetworkID this nodes belongs to.

Struct `DhtID`

Description

DhtID is implemented in Node to arrange all identity entities in one place and make dht agnostic of its DhtID implementation. For further details on DhtID's implementation please see The XOR Space in DHT.

Dependencies

crypto
filesystem
log

Package - `p2p/dht`

Description

dht is an integral part of the p2p package, it is used to implement the Kademlia DHT. A DHT is mainly used in p2p networks to lookup and probe an ID into an IP address or a file hash to the node it's stored on.

We broke Kademlia into two parts the node store known as the RoutingTable and the protocols, we skip most of the Kademlia protocols and implement only the FindNode protocol which is used to probe nodes in the network by IDs.

The RoutingTable is a table that holds all the nodes that we know from communications or lookups in the network, it is in charge of keeping that list of nodes healthy and ready for when the node needs to preform a lookup or get information from the network.

For Kademlia lookups we're hashing the original Node Public Keys to SHA256 hashes. every node ID has its deterministic DhtID.

Using those hashes we calculate and arrange nodes inside the RoutingTable according to their "Distance" from us in Kademlia "Distance" is determined by a space called the XOR Keyspace

The `XOR` Keyspace

The dht.IDs exist inside the XOR keyspace, dht also exposes sorting and functionality in this space which should help determine the distance to other nodes in this space.

"Distance" to a node in the XOR space is the result of XORing the IDs of both nodes. XOR is symmetric and will always produce the same result for the same set of IDs.

When XORing the dht.IDs together we use the given result and check the amount of leading zero's in the binary representation of the result. this is called the Common Prefix Length of thses two dht.IDs.

The higher the Common Prefix Length the closer the node to us.

Example:

// For this example and readability we assume the keyspace is 24bits (3 bytes) long instead of 256bits
"123" as DhtID = a665a4
binary - 10100110110010110100100

"567" as DhtID = 97a6d2
binary - 100101111010011011010010

XORing : {
"123"(a665a4) XOR "567"(97a6d2) = 31c376
binary - 110001110000111110110
}

The zero prefix length of the xor result
which is the Common Prefix Length : 2
(100101111010011011010010)

A third ID : "789"
"789" as DhtID = 35a9e3
binary - 1101011010100111100011
XORing "123" with "789" {
  "123"(a665a4) XOR "789"(35a9e3) = 93cc47
  binary - 110001110000111110110
}

The zero prefix length of the xor result
which is the Common Prefix Length : 0

567 is closer in xor space to 123 than 789

Struct `DHT`

Description

DHT implements RoutingTable which is a table that holds all the nodes that we know from communications or lookups in the network.

The RoutingTable is in charge of keeping that list of nodes healthy and ready for when the node needs to preform a lookup or get information from the network.

The nodes in the RoutingTable are arranged inside Buckets, in Kademlia they are called KBuckets. every bucket is represented by a Common Prefix Length, we store 20 Buckets which hold nodes according to our BucketSize parameter which is currently 20. The higher the Bucket's Common Prefix Length, the closer the nodes in the list to us. Most of the nodes are going to be in Bucket 0, half of the network according to Kademlia, half of the rest of the nodes will be in Bucket 1, and so forth. this is why we hold only 20 buckets.

Extra notes about DHT

Public API

New(p2p.Config) DHT -

Creates a new DHT with the config parameters passed.
Bootstrap() error -

Bootstrap the routing table with the bootstrapped nodes configured, a blocking operation.
Update(node node.Node) -

Updates a node in the routing table. If we already know this peer we move it to the top of the Bucket If we never heard of this peer, we try to insert it to the appropriate bucket. If this bucket is full we ping the last contacted peer (which should be the last in the bucket) and only if the ping fails we add the new peer.
Lookup(id string) (node.Node, error) -

Tries to lookup the node in the local RoutingTable if it wasn't found, issues a network FindNode operation to find it.

Dependencies

net
node

Struct `FindNode` (internal)

Description

Our KadDHT implementation holds a FindNode protocol. this protocol is the actual Kademlia - FindNode implementation. When it receives a FindNode request it looks up the local table for matching or close nodes to the requested ID. it then sends them back to the requester as a sorted array.

The protocol is created by passing a p2p.Service and a RoutingTable the constructing function is only exposed to the dht package.

Public API

FindNode(serverNode node.Node, target string) ([]node.Node, error) - The only method this protocol exposes, it asks the serverNode about the target ID. it returns a sorted slice with the closest nodes to this ID with the requested node ID as the first one if found. this function is blocking while sending the message and waiting for response.

Bootstrapping protocol

The Bootstrap process is what let's the node join the p2p network.

The operation will run in the background and fill our RoutingTable with fresh and active nodes, to do this the node is pre-configured with a list of Bootstrap Nodes. every node can act as Bootstrap Node so they can be replaced with node information acquired from any other source.

The bootstrapping process is just a matter of issuing the FindNode Kademlia protocol with the node's own identity as a parameter. this will return a list of nodes close to our own DhtID, we'll query those nodes (Concurrently) for the same ID until we don't get new results and we queried all nodes.

This will make every node that see our request to update its RoutingTable with our node and also fill our RoutingTable with these nodes. soon peers from all around the network will know that we join and start querying us in the future.

Package `p2p/net`

Description

Package p2p/net is the package that handles all network communications.

Struct `Net`

Description

Net is a general purpose secured (encrypted) connection factory and message listener. It is used to initiate and to accept all network operations including connections and messages.

Public API

Dial(address string, remotePublicKey crypto.PublicKey) (Connection, error) -

Initiate a secured connection with another peer , returns error or nil. Blocks until session is created or failure.
SubscribeOnNewRemoteConnections() chan Connection -

Subscribe to get updates on new established remote connections.
HandlePreSessionIncomingMessage(c Connection, message []byte) error -

Handles an incoming message which received before a session was created (expected to be the session creation request). Returns error when session creation failed.
GetClosingConnections() -

The channel where closing connection events are reported.
IncomingMessages() chan IncomingMessageEvent -

A channel that bridges incoming messages from all the connections in a fan-in fashion.

Session message format

// The initiator creates a HandshakeData object with a random IV based on the requested node Public Key and our
// own Private Key. The request also contains basic details about the initiator to decide if we can can create a
// session.
// Handshake protocol data used for both request and response - sent unencrypted over the wire
message HandshakeData {
  bytes sessionId = 1;    // for req - same as iv. for response - set to req id
  bytes payload = 2; // empty for now
  int64 timestamp = 3; // sending time
  string clientVersion = 4; // client version of the sender
  int32 networkID = 5; // network id of sending node
  string protocol = 6; // 'handshake/req' || 'handshake/resp'
  bytes nodePubKey = 7; // 65 bytes uncompressed
  bytes iv = 8; // 16 bytes - AES-256-CBC IV
  bytes pubKey = 9; // 65 bytes (uncompressed) ephemeral public key
  bytes hmac = 10; // HMAC-SHA-256 32 bytes
  string sign = 11; // hex encoded string 32 bytes sign of all above data by node public key (verifies he has the priv key and he wrote the data
}

Struct `Connection`

Description

A struct wrapping the go net.Conn TCP connection that pipes messages through the passed wire formatter. Connection is wire format agnostic, at this moment we use binary delimited protobuf messages, we do it using the delimited package. delimited wraps the connections and reads/writes length delimited messages over it.

Public API -

ID() string -

Returns the connection;s UUID
Send(message) error -

Sends a message on the connection, returns an error if failed. This method is blocking.
Session() NetworkSession -

Returns the connection's session or nil if no session.
Close() -

Closes the connection.
RemoteAddr() net.Addr -

Returns the remote peer's address.
RemotePublicKey() crypto.PublicKey -

Returns the remote peer's crypto.PublicKey

Dependencies

crypto
log
nodeconfig

Struct `NetworkSession`

Description

NetworkSession is an authenticated network session between two peers. Sessions may be used between 'connections' until they expire. Session provides the encryptor/decryptor for all messages exchanged between two peers. enc/dec is using an ephemeral symmetric key exchanged securely between the peers via the secured connection handshake protocol.

Public API

Decrypt(in []byte) ([]byte, error) -

Decrypt data using session dec key
Encrypt(in []byte) ([]byte, error) -

Encrypt data using session enc key

Package `p2p/net/delimited`

Description

We are using our own simple length-prefix binary format:

<32 bits big-endian data-length><message binary data (protobufs-bin-encoded)>

We need to use length prefixed protobufs messages because protobufs data doesn't include length and we need this to allow multiple messages on the same tcp/ip connection . see p2p2.conn.go.

Public API

NewReader(r io.Reader) *delimited.Reader -

Creates a delimited wrapped reader. not used directly.
NewWriter(r io.Writer) *delimited.Writer -

Creates a delimited wrapped writer. not used directly.
Pipe(io.ReadWriteCloser) -

Pipe an io.ReadWriteCloser through the delimited format.
In() chan []byte -

The incoming messages channel
Out(message []byte) error -

Sends out a message on the wire, it is a blocking operation.
Close() -

Closes the internal connection and the channels.

Package `p2p/connectionpool`

Struct `ConnectionPool`

Description

ConnectionPool stores all net.Connections and make them available to all users of net.Connection. All local connections are created by ConnectionPool, using Net. Also ConnectionPool is in charge of listening to new remote connections. Local and remote connections are treated the same by ConnectionPool and at any given point there is at most one connection with any given peer.

Public API

NewConnectionPool(network networker, lPub crypto.PublicKey) *ConnectionPool -

Constructs a new connection pool. Once created, the ConnectionPool will start listening to new remote connections using the networker interface.
GetConnection(address string, remotePub crypto.PublicKey) (net.Connection, error) -

Tries to fetch a connection from the pool, if the connection doesn't exist it will establish one using the networker interface. This is a blocking function, it will block until a net.Connection or an error is returned.
Shutdown() -

Gracefully shuts down the connection pool by closing all the connections in the pool and releasing all pending GetConnection queries.

Dependencies

crypto
p2p/net

`p2p/sim` - the p2p simulator

Description

The p2p simulator is used for testing and simulating a p2p network of nodes without actually listening to ports and sending network messages. it creates nodes that satisfy the p2p.Service interface.

Public API

New() *Simulator -

Creates a simulator that acts as services factory and bridge
Simulator.NewNode() *Node -

Creates a node on the receiving Simulator network.
Simulator.NewNodeFrom(node.Node) *Node -

Creates a node on the receiving Simulator network from an existing node data.

Dependencies

node
service

Package `gossip`

Information about the gossip protocol is in - Gossip Protocol

P2P Test Plan

empty field - not relevant for the test


Test Desc	Total Nodes	Faulty nodes	Faulty Behaviour	Boot nodes	Bucket Size	Buckets qty	Min Routing table nodes	Flow Description	Expected Result
Simple message exchange	2							* Send ping message from node1 to node2 * The reciever will be able to decrypt and read the message, he will send response * The receiver will update itself with the sending node * The sender will receive and succesfully decrypt the response	- Reciever output he got the message - Sender output he got the resposne
Multiple messages with one target node. (ex: to a bootnode)	500			1				- First node boot up - Boot rest of nodes - Update all nodes with first node and send it a message	- First node will output all messages - All 500 nodes will output a response form first node
Send message to unknown node ( Simple FindNode )	3			1			2	- Start 1 node - Start 2 more nodes with node 1 as updated node. - send message between the node 2 and 3	Node 3 will get the message
Bootstrap	10/100/1000			3 / 5 / 10	20-100	16-128	5-50	- Start 3 boot nodes - Start 10 nodes preconfigured with the 5 boot nodes and bootstrap	10 nodes will output bootstrap success
Bootstrap with disconnects	10/100/1000	random %	Disconnect after starting	3 / 5 / 10	20-100	16-128	5-50		10 - random nodes will output bootstrap success
Bootstrap with message drops	10/100/1000	random %	drop messages	3 / 5 / 10	20-100	16-128	5-50		10 nodes will output bootstrap success
Bootstrap with message delay	10/100/1000	random %	delay messages	3 / 5 / 10	20-100	16-128	5-50		10 nodes will output bootstrap success
Send message to node we don’t know after Bootstrap.	10/100/1000			3 / 5 / 10	20-100	16-128	5-50	- All bootstrap stages - Pick a single node - Try send a message to a node that is not in its routing table	the node will output the message
Send message to node we don’t know after Bootstrap.	10/100/1000	random %		3 / 5 / 10	20-100	16-128	5-50	- All bootstrap stages - Pick a single node - Try send a message to a node that is not in its routing table	the node will output the message
Send message to node we don’t know after Bootstrap. with disconnect	10/100/1000	random %	Disconnect after starting	3 / 5 / 10	20-100	16-128	5-50	- All bootstrap stages - Pick a single node - Try send a message to a node that is not in its routing table	the node will output the message
Send message to node we don’t know after Bootstrap. with message drops	10/100/1000	random %	drop messages	3 / 5 / 10	20-100	16-128	5-50	- All bootstrap stages - Pick a single node - Try send a message to a node that is not in its routing table	the node will output the message
Send message to node we don’t know after Bootstrap. with message delay	10/100/1000	random %	delay messages	3 / 5 / 10	20-100	16-128	5-50	- All bootstrap stages - Pick a single node - Try send a message to a node that is not in its routing table	the node will output the message
Benchmarks
Measure message latency	2							Send message from node a to b	Measure pingpong time.
Measure SendMessage to unknown after bootstrap	10/100/1000	random %		3 / 5 / 10	20-100	16-128	5-50		Metric : - Count FindNode operations (Hops) - Total time - avg findnode op time
Measure SendMessage to unknown after bootstrap with disconnect	10/100/1000	random %	disconnect	3 / 5 / 10	20-100	16-128	5-50		Metric : - Count FindNode operations (Hops) - Total time - avg findnode op time
Measure SendMessage to unknown after bootstrap with message drops	10/100/1000	random %	drops messages	3 / 5 / 10	20-100	16-128	5-50		Metric : - Count FindNode operations (Hops) - Total time - avg findnode op time
Measure SendMessage to unknown after bootstrap with message delay	10/100/1000	random %	delay messages	3 / 5 / 10	20-100	16-128	5-50		Metric : - Count FindNode operations (Hops) - Total time - avg findnode op time

Connect => discord || spacemesh.io || @teamspacemesh || Roadmap || FAQ

Getting Started

Dev Guides

Dev Milestones

Product Specs

Start here

P2P Architecture

Package p2p

Description

Public API

Struct Swarm

Description

Public API

Message Metadata

Dependencies

Package p2p/node

Description

Public API

Struct Node

Description

Public API -

Struct LocalNode

Description

Public API

Struct DhtID

Description

Dependencies

Package - p2p/dht

Description

The XOR Keyspace

Struct DHT

Description

Public API

Dependencies

Struct FindNode (internal)

Description

Public API

Bootstrapping protocol

Package p2p/net

Description

Struct Net

Description

Public API

Session message format

Struct Connection

Description

Public API -

Dependencies

Struct NetworkSession

Description

Public API

Package p2p/net/delimited

Description

Public API

Package p2p/connectionpool

Struct ConnectionPool

Description

Public API

Dependencies

p2p/sim - the p2p simulator

Description

Public API

Dependencies

Package gossip

P2P Test Plan

Getting Started

Dev Guides

Product Specs

Clone this wiki locally

Package `p2p`

Struct `Swarm`

Message `Metadata`

Package `p2p/node`

Struct `Node`

Struct `LocalNode`

Struct `DhtID`

Package - `p2p/dht`

The `XOR` Keyspace

Struct `DHT`

Struct `FindNode` (internal)

Package `p2p/net`

Struct `Net`

Struct `Connection`

Struct `NetworkSession`

Package `p2p/net/delimited`

Package `p2p/connectionpool`

Struct `ConnectionPool`

`p2p/sim` - the p2p simulator

Package `gossip`