Skip to content

Commit

Permalink
Merge pull request #70 from mycognosist/ebt_docs
Browse files Browse the repository at this point in the history 
Add documentation for EBT replication
  • Loading branch information
@mycognosist
mycognosist authored Nov 24, 2023
2 parents 8908759 + 47cc9ed commit c3e8b35
Show file tree
Hide file tree
Showing 2 changed files with 216 additions and 1 deletion.
34 changes: 33 additions & 1 deletion css/guide.css
View file
Original file line number Diff line number Diff line change
Expand Up @@ -300,6 +300,38 @@ table.defs td:first-child {
padding-right: 1rem;
}

table.clock-values {
font-size: 13px;
margin: 1rem 0 2rem;
background-color: #f9d2e9;
}

table.clock-values td, th {
text-align: center;
vertical-align: middle;
border-right: solid thin #fff;
}

table.clock-values thead {
color: #fff;
}

table.clock-values tr {
border-bottom: solid thin #fff;
}

table.clock-values thead tr:nth-child(1) {
background-color: #ca82c9;
}

table.clock-values thead tr:nth-child(2) {
background-color: #f29bce;
}

table.clock-values tbody td:first-child {
font-weight: bold;
}

.small {
font-size: 13px;
}
Expand Down Expand Up @@ -355,7 +387,7 @@ table.defs td:first-child {
grid-column: right;
}

.request, .response, .request-arrow, .response-arrow {
.request, .response, .request-arrow, .response-arrow, .clock-table {
margin: 12px 0;
}

Expand Down
183 changes: 183 additions & 0 deletions index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ <h2 id="toc">Contents</h2>
<div><a href="#structure">Structure</a></div>
<div><a href="#message-format">Message format</a></div>
<div><a href="#createHistoryStream">createHistoryStream</a></div>
<div><a href="#ebt-replication">EBT replication</a></div>
</div>
</div>
<div class="section">
Expand Down Expand Up @@ -1086,6 +1087,188 @@ <h5>Implementations</h5>
}</code></pre>
</div>
</figure>
<!-- ebt documentation -->
<h3 id="ebt-replication">Epidemic Broadcast Tree (EBT) Replication</h3>
<aside class="impl kicker">
<img class="icon" src="img/impl.png" alt=""/>
<h5>Implementations</h5>
<div class="lang">JS</div>
<div><a href="https://github.com/ssbc/epidemic-broadcast-trees/blob/master/index.js">epidemic-broadcast-trees</a></div>
<div class="vs"><a href="https://github.com/ssbc/ssb-ebt/blob/master/index.js">ssb-ebt</a></div>
<div class="lang">Go</div>
<div class="vs"><a href="https://github.com/planetary-social/scuttlego/tree/main/service/domain/replication/ebt">ebt</a></div>
</aside>
<p>In addition to classic replication using <i>createHistoryStream</i>, some Scuttlebutt clients implement a more efficient form of replication known as <i>Epidemic broadcast tree replication</i>. This is often referred to by the abbreviation <i>EBT</i>. The implementation of <i>EBT</i> used in Scuttlebutt is loosely based on the push-lazy-push multicast tree protocol, more commonly known as the <i>Plumtree</i> protocol [1].</p>
<h4 id="session-initiation">Session Initiation</h4>
<p>An <i>EBT</i> session may be initiated once two peers have completed the secret handshake and have established their respective box streams. The peer who acted as the client during the secret handshake takes on the role of the requester, sending an <i>["ebt", "replicate"]</i> request to the connected peer.</p>
<figure class="request-response">
<div class="request">
<div class="header">
<span class="key">Request number</span><span class="value">1</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">Yes</span>
<span class="key">End/err</span><span class="value">No</span>
</div>
<pre><code>{
"name": ["ebt", "replicate"],
"type": "duplex",
"args": [
{
"version": 3,
"format": "classic"
}
]
}</code></pre>
</div>
<img class="request-arrow" src="img/arrow.png" alt=""/>
</figure>
<p>The peer who acted as the server during the secret handshake takes on the role of the responder. After having received the replicate request, the responder first validates the arguments to ensure that the version is 3 and the format is "classic". If either of those values are incorrect, the responder terminates the stream with an error.</p>
<h4 id="vector-clocks">Vector Clocks</h4>
<p>The responder then sends a vector clock (also known as a "note" or "control message") to the requester. The vector clock takes the form of a JSON object with one or more key-value pairs. The key of each pair specifies a Scuttlebutt feed identified by the @-prefixed public key of the author. The value of each pair is a signed integer encoding a replicate flag, a receive flag and a feed sequence number.
<figure class="request-response">
<img class="response-arrow" src="img/arrow.png" alt=""/>
<div class="response">
<div class="header">
<span class="key">Request number</span><span class="value">-1</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">Yes</span>
<span class="key">End/err</span><span class="value">No</span>
</div>
<pre><code>{
"@qK93G/R9R5J2fiqK+kxV72HqqPUcss+rth8rACcYr4s=.ed25519": 450,
"@L/g6qZQE/2FdO2UhSJ0uyDiZb5LjJLatM/d8MN+INSM=.ed25519": 12,
"@fGHFd8rUgoznX/qS/1U7HPF3vnirbSyfaaWlS8cCWR0=.ed25519": 1,
"@HEqy940T6uB+T+d9Jaa58aNfRzLx9eRWqkZljBmnkmk=.ed25519": -1
}</code></pre>
</div>
</figure>
<p>The requester terminates the stream with an error if any of the received feed identifiers or encoded values are malformed. If the received vector clock is valid, the requester can proceed with decoding the values.</p>
<p>The value in each key-value pair of a vector clock encodes a maximum of three data points: a replicate flag, a receive flag and a sequence number. A negative value (usually -1) signals that the responder does not wish to replicate the associated feed, neither sending nor receiving messages. In this scenario, the replicate flag is set to false and both the receive flag and sequence number are irrelevant.</p>
<p>A positive value signals that the responder wishes to replicate the associated feed. If the value is positive it should be decoded as follows. First, the JSON number is parsed and converted to a signed integer. Then, the rightmost (lowest order) bit of the number is interpreted as a binary flag with 0 equal to true and 1 equal to false. This flag is referred to as the receive flag. Next, a sign-extending right shift (also called arithmetic right shift) by 1 bit is performed on the binary number, therefore discarding the rightmost (lowest order) bit. The remaining number is then interpreted as a sequence number for the associated feed.</p>
<p>If the receive flag is set to true, the peer who sent the vector clock wishes to receive messages for the associated feed. The decoded sequence number defines the latest message held by the peer for that feed.</p>
<p>Encoding of a vector clock value involves reversing the steps outlined above. If the peer does not wish to replicate a feed, the value is simply set to -1. Otherwise, the latest sequence number of the associated feed is stored as a signed integer and an arithmetic left shift is performed. The rightmost (lowest order) bit is then set according to the replicate flag as described previously.</p>
<table class="clock-values">
<thead>
<tr>
<th rowspan="2">Encoded</th>
<th colspan="3">Decoded</th>
</tr>
<tr>
<th>Replicate flag</th>
<th>Receive flag</th>
<th>Sequence</th>
</tr>
</thead>
<tbody>
<tr>
<td>-1</td>
<td>False</td>
<td>Irrelevant</td>
<td>Irrelevant</td>
</tr>
<tr>
<td>0</td>
<td>True</td>
<td>True</td>
<td>0</td>
</tr>
<tr>
<td>1</td>
<td>True</td>
<td>False</td>
<td>0</td>
</tr>
<tr>
<td>2</td>
<td>True</td>
<td>True</td>
<td>1</td>
</tr>
<tr>
<td>3</td>
<td>True</td>
<td>False</td>
<td>1</td>
</tr>
<tr>
<td>12</td>
<td>True</td>
<td>True</td>
<td>6</td>
</tr>
<tr>
<td>450</td>
<td>True</td>
<td>True</td>
<td>225</td>
</tr>
</tbody>
</table>
<p>The requester then sends their own vector clock to the responder. At this point, the initial exchange of vector clocks is complete and both peers may begin sending messages at will. Updated vector clocks may continue to be sent by both peers at any point during the session. These updated clocks may reference a subset of the feeds represented in the initial vector clock, or they may reference different feeds entirely. This provides a means for responding to state changes in the local database and follow graph.</p>
<aside class="kicker" style="align-self: start; position: relative; top: 58px;">
<p>The same request number is used for all requests occurring between a pair of peers over the course of a single EBT session.</p>
</aside>
<figure class="request-response">
<div class="request">
<div class="header">
<span class="key">Request number</span><span class="value">1</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">Yes</span>
<span class="key">End/err</span><span class="value">No</span>
</div>
<pre><code>{
"@fGHFd8rUgoznX/qS/1U7HPF3vnirbSyfaaWlS8cCWR0=.ed25519": 62,
"@qK93G/R9R5J2fiqK+kxV72HqqPUcss+rth8rACcYr4s=.ed25519": -1,
"@L/g6qZQE/2FdO2UhSJ0uyDiZb5LjJLatM/d8MN+INSM=.ed25519": 10
}</code></pre>
</div>
<img class="request-arrow" src="img/arrow.png" alt=""/>
</figure>
<p>Messages are sent in exactly the same way as when responding to a <i>createHistoryStream</i> request.
<figure class="request-response">
<img class="response-arrow" src="img/arrow.png" alt=""/>
<div class="response">
<div class="header">
<span class="key">Request number</span><span class="value">-1</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">Yes</span>
<span class="key">End/err</span><span class="value">No</span>
</div>
<pre><code>{
"previous": "%GvRqbiZFY0cNyGRD4QwjeMQvnrjz9vMPBsT5JdWrcW0=.sha256",
"author": "@L/g6qZQE/2FdO2UhSJ0uyDiZb5LjJLatM/d8MN+INSM=.ed25519",
"sequence": 5,
"timestamp": 1698824093970,
"hash": "sha256",
"content": {
"type": "contact",
"contact": "@fGHFd8rUgoznX/qS/1U7HPF3vnirbSyfaaWlS8cCWR0=.ed25519",
"blocking": false,
"following": true
},
"signature": "8+VQ7sbv0fnD8ZnbunJ19fyvtcwSpHvhlWUakj32nU4woFNNpI
qpvkAJ4GMGJdYHoqc8C7asPXPa+wMzbPR1Cw==.sig.ed25519"
}</code></pre>
</div>
</figure>
<h4 id="session-initiation">Session Termination</h4>
<p>An <i>EBT</i> session may be terminated by either peer at any point, either by sending an error response or by closing the stream. If no error has occurred, the stream is closed when a peer wishes to conclude the session (as described in the <b>Source example</b> of the <b>RPC protocol</b> section above).</p>
<h4 id="request-skipping">Request Skipping</h4>
<p><i>EBT</i> implementations rely on a mechanism known as <i>request skipping</i> to lower bandwidth overhead and increase replication efficiency. Each peer stores the vector clocks they receive from remote peers; these may be held in memory and persisted to disk to allow later retrieval. When a subsequent <i>EBT</i> session is initiated between peers, each peer first checks the stored vector clock of their remote peer before calculating an updated vector clock to be sent. If the latest locally-available sequence of a feed from the remote peer's vector clock is the same as the sequence in the saved vector clock for that peer, that feed is left out of the new vector clock in the outgoing request (hence the name <i>request skipping</i>). This provides a mechanism for limiting the total number of bytes to be sent over the wire.</p>
<aside>
<p>Approaches to tracking EBT session state may be gleaned from the JS and Go implementations.</p>
</aside>
<p>The stored vector clock for the remote peer may differ from their current vector clock. In that case, the remote peer will include the updated feed in their request and the local peer will respond by sending an additional partial vector clock including their sequence for that feed. Once both sides have exchanged their sequence for a particular feed, replication of messages in that feed may occur.</p>
<h4 id="vector-clock-partioning">Vector Clock Partitioning</h4>
<p>In order to further increase efficiency when connecting to multiple peers, feeds for which the local peer would like to receive updates are only sent to one peer at a time (in the outbound vector clock). A timeout may be used to request the feed from an alternate peer if no updates are available from the initial peer. In this way, the total set of requested feeds is spread across multiple peers.</p>
<h4 id="fallback-mechanisms">Fallback Mechanisms</h4>
<p><i>EBT</i> is the preferred means for peers to exchange messages. However, not all Scuttlebutt clients support <i>EBT</i> replication. In the case that only one of two connected peers support <i>EBT</i>, both peers may instead fallback to using <i>createHistoryStream</i> to exchange messages. There are several scenarios which may trigger initiation of <i>createHistoryStream</i> replication:</p>
<ul>
<li>If the requester attempts to initiate an <i>EBT</i> session but the session is terminated with an error by the responder</li>
<li>If a <i>createHistoryStream</i> request is immediately sent by the client upon successful connection</li>
<li>If the client doesn’t attempt to initiate an <i>EBT</i> session for a certain amount of time</li>
</ul>
<p>[1] Joao Leitao, Jose Pereira and Luis Rodrigues. 2007. Epidemic Broadcast Trees. In <i>2007 26th IEEE International Symposium on Reliable Distributed Systems (SRDS 2007)</i>, 301-310. https://doi.org/10.1109/SRDS.2007.27</p>
<!-- metafeeds documentation -->
<h2 id="metafeeds">Metafeeds</h2>
<div>
Expand Down

0 comments on commit c3e8b35

Please sign in to comment.