In this chapter we will describe the Lightning Network’s Onion Routing mechanism. The invention of onion routing precedes the Lightning Network by 25 years! Onion routing was invented by U.S. Navy researchers as a communications security protocol. Onion routing is most famously used by Tor, the onion routed internet overlay that allows researchers, activists, intelligence agents and everyone else to use the internet privately and anonymously.
In this chapter we are focusing on the "Source based Onion Routing (SPHINX)" part of the Lightning protocol architecture, highlighted by a double outline in the center (Routing Layer) of The Lightning Network Protocol Suite:
Onion routing describes a method of encrypted communication where a message sender builds successive nested layers of encryption that are "peeled" off by each intermediary node, until the innermost layer is delivered to the intended recipient. The name "onion routing" describes this use of layered encryption that is peeled off one layer at a time, like the skin of an onion.
Each of the intermediary nodes can only "peel" one layer and see who is next in the communications path. Onion routing ensures that no one except the sender knows the destination or length of the communication path. Each intermediary only knows the previous and next hop.
The Lightning Network uses an implementation of onion routing protocol based on _Sphinx_[1] developed in 2009 by George Danezis and Ian Goldberg.
The implementation of onion routing in the Lightning Network is defined in BOLT #4 - Onion Routing Protocol
There are many ways to describe onion routing, but one of the easiest is to use the physical equivalent of sealed envelopes. An envelope represents a layer of encryption, allowing only the named recipient to open it and read the contents.
Let’s say Alice wants to send a secret letter to Dina, indirectly via some intermediaries.
The Lightning Network uses source routing, which means that the payment path is selected and specified by the sender, and only the sender. In this example, Alice’s secret letter to Dina will be the equivalent of a payment. To make sure the letter reaches Dina, Alice will create a path from her to Dina, using Bob and Chan as intermediaries.
|
Tip
|
There may be many paths that make it possible for Alice to reach Dina. We will explain the process of selecting the optimum path in [path_finding]. For now, we’ll assume that the path selected by Alice uses Bob and Chan as intermediaries to get to Dina. |
As a reminder, the path selected by Alice is shown in Path: Alice to Bob to Chan to Dina, below:
Let’s see how Alice can use this path without revealing information to intermediaries Bob and Chan.
Source-based routing is not how packets are typically routed on the internet today, though source routing was possible in the early days. Internet routing is based on packet switching at each intermediary routing node. An IPv4 packet, for example, includes the sender and recipient’s IP address and every other IP routing node decides how to forward each packet towards the destination. However, the lack of privacy in such a routing mechanism, where every intermediary node sees the sender and recipient, make this a poor choice for use in a payment network.
Alice starts by writing a secret letter to Dina. She then seals the letter inside an envelope and writes "To Dina" on the outside (see Dina’s secret letter, sealed in an envelope). The envelope represents encryption with Dina’s public key, so that only Dina can open the envelope and read the letter.
Dina’s letter will be delivered to Dina by Chan, who is immediately before Dina in the "path". So, Alice puts Dina’s envelope inside an envelope addressed to Chan (see Chan’s envelope, containing Dina’s sealed envelope). The only part that Chan can read is the destination (routing instructions): "To Dina". Sealing this inside an envelope addressed to Chan represents encrypting it with Chan’s public key so that only Chan can read the envelope address. Chan still can’t open Dina’s envelope. All he sees is the instructions on the outside (the address).
Now, this letter will be delivered to Chan by Bob. So Alice puts it inside an envelope addressed to Bob (see Bob’s envelope, containing Chan’s sealed envelope). As before, the envelope represents a message encrypted to Bob that only Bob can read. Bob can only read the outside of Chan’s envelope (the address), so he knows to send it to Chan.
Now, if we could look through the envelopes (with X-rays!) we would see the envelopes nested one inside the other, as shown in Nested envelopes, below:
Alice now has an envelope that says "To Bob" on the outside. It represents an encrypted message that only Bob can open (decrypt). Alice will now begin the process by sending this to Bob. The entire process is shown in Sending the envelopes below:
As you can see, Bob receives the envelope from Alice. He knows it came from Alice, but doesn’t know if Alice is the original sender or just someone forwarding envelopes. He opens it to find an envelope inside that says "To Chan". Since this is addressed to Chan, Bob can’t open it. He doesn’t know what’s inside it and doesn’t know if Chan is getting a letter or another envelope to forward. Bob doesn’t know if Chan is the ultimate recipient or not. Bob forwards the envelope to Chan.
Chan receives the envelope from Bob. He doesn’t know that it came from Alice. He doesn’t know if Bob is an intermediary or the sender of a letter. Chan opens the envelope and finds another envelope inside addressed "To Dina", which he can’t open. Chan forwards it to Dina, not knowing if Dina is the final recipient.
Dina receives an envelope from Chan. Opening it she finds a letter inside, so now she knows she’s the intended recipient of this message. She reads the letter, knowing that none of the intermediaries know where it came from and no one else has read her secret letter!
This is the essence of onion routing. The sender wraps a message in layers, specifying exactly how it will be routed and preventing any of the intermediaries from gaining any information about the path or payload. Each intermediary peels one layer, sees only a forwarding address and doesn’t know anything other than the previous and next hop in the path.
Now, let’s look at the details of the onion routing implementation in the Lightning Network.
Onion routing in the Lightning Network appears complex at first glance, but once you understand the basic concept is really quite simple.
From a practical perspective, Alice is telling every intermediary node what HTLC to set up with the next node in the path.
The first node, which is the payment sender or Alice in our example, is called the -_origin node. The last node, which is the payment recipient or Dina in our example, is called the final node.
Each intermediary node, or Bob and Chan in our example, is called a hop. Every hop must set up an outgoing HTLC to the next hop. The information communicated to each hop by Alice is called the hop payload or hop data. The message that is routed from Alice to Dina is called an onion and consists of encrypted hop payload or hop data messages encrypted to each hop.
Now that we know the terminology used in Lightning Onion Routing, let’s restate Alice’s task: Alice must construct an onion with hop data, telling each hop how to construct an outgoing HTLC in order to send a payment to the final node (Dina).
From [routing] we know that Alice will send a 50,000 satoshi payment to Dina via Bob and Chan. This payment is transmitted via a series of HTLCs, as shown in Payment path with HTLCs from Alice to Dina, below:
As we will see in [gossip], Alice is able to construct this path to Dina because Lightning nodes announce their channels to the entire Lightning Network using the Lightning Gossip Protocol. After the initial channel announcement, Bob and Chan each sent out an additional "channel update" message with their routing fee and timelock expectations for payment routing.
From the announcements and updates, Alice knows the following information about the channels between Bob, Chan and Dina:
-
A short_channel_id (short channel ID) for each channel, that Alice can use to reference the channel, when constructing the path
-
A cltv_expiry_delta (timelock delta) which Alice can add to the expiry time for each HTLC
-
A fee_base_msat and fee_proportional_millionths which Alice can use to calculate the total routing fee expected by that node for relay on that channel.
In practice, other information is also exchanged such as the largest (htlc_maximum_msat) and smallest (htlc_minimum_msat) HTLCs a channel will carry, but these aren’t used as directly during onion route construction as the above fields are.
This information is used by Alice to identify the nodes, channels, fees, and timelocks for the following detailed path, shown in A detailed path constructed from gossiped channel and node information:
Alice already knows her own channel to Bob, and therefore doesn’t need this info to construct the path. Note also that Alice didn’t need a channel update from Dina, because she has the update from Chan for that last channel in the path.
There are two possible formats that Alice can use for the information communicated to each hop: A legacy fixed-length format called the hop data and a more flexible Type-Length-Value (TLV) based format called the hop payload. The TLV message format is explained in more detail in [tlv]. It offers flexibility by allowing fields to be added to the protocol at will.
|
Note
|
Both formats are specified in BOLT #4 - Onion Routing - Packet Structure. |
Alice will start building the hop data from the end of the path backwards: Dina, Chan, Bob.
Alice first builds the payload that will be delivered to Dina. Dina will not be constructing an "outgoing HTLC", because Dina is the final node and payment recipient. For this reason, the payload for Dina is different than all the others (uses all zeros for the short_channel_id), but only Dina will know this since it will be encrypted in the innermost layer of the onion. Essentially, this is the "secret letter to Dina" we saw in our physical envelope example.
The hop payload for Dina must match the information in the invoice generated by Dina for Alice and will contain (at least) the following fields in Type-Length-Value (TLV) format:
- amt_to_forward
-
The amount of this payment in milli-satoshis. If this is only one part of a multi-part payment, the amount is less than the total. Otherwise, this is a single full payment and it is equal to the invoice amount and total_msat value.
- outgoing_cltv_value
-
The payment expiry timelock set to the value min_final_cltv_expiry in the invoice.
- payment_secret
-
A special 256-bit secret value from the invoice, allowing Dina to recognize this incoming payment. This also prevents a class of probing that previously made 0-value invoices insecure. Probing by intermediate nodes is mitigated as this value is encrypted to only the recipient, meaning they can’t reconstruct a final packet that "looks" legitimate.
- total_msat
-
The total amount matching the invoice. This may be omitted if there is only one part, in which case it is assumed to match amt_to_forward and must equal the invoice amount.
The invoice Alice received from Dina specified the amount as 50,000 satoshis, which is 50,000,000 milli-satoshis. Dina specified the minimum expiry for the payment min_final_cltv_expiry as 18 blocks (3 hours, given 10-minute on average Bitcoin blocks). At the time Alice is attempting to make the payment, let’s say the Bitcoin blockchain has recorded 700,000 blocks. So Alice must set the outgoing_cltv_value to a minimum block height of 700,018.
Alice constructs the hop payload for Dina as follows:
amt_to_forward : 50,000,000 outgoing_cltv_value: 700,018 payment_secret: fb53d94b7b65580f75b98f10...03521bdab6d519143cd521d1b3826 total_msat: 50,000,000
Alice serializes it in TLV format as shown (simplified) Dina’s payload is constructed by Alice below:
Next, Alice constructs the hop payload for Chan. This will tell Chan how to setup an outgoing HTLC to Dina.
The hop payload for Chan includes three fields: short_channel_id, amt_to_forward and outgoing_cltv_value:
short_channel_id: 010002010a42be amt_to_forward: 50,000,000 outgoing_cltv_value: 700,018
Alice serializes this payload in TLV format, as shown (simplified) in Chan’s payload is constructed by Alice below:
Finally, Alice constructs the hop payload for Bob, which also contains the same three fields as the hop payload for Chan, but with different values:
short_channel_id: 000004040a61f0 amt_to_forward: 50,100,000 outgoing_cltv_value: 700,038
As you can see, the amt_to_forward field is 50,100,000 milli-satoshis, or 50,100 satoshis. That’s because Chan expects a fee of 100 satoshis to route a payment to Dina. In order for Chan to "earn" that routing fee, Chan’s incoming HTLC must be 100 satoshis more than Chan’s outgoing HTLC. Since Chan’s incoming HTLC is Bob’s outgoing HTLC, the instructions to Bob reflect the fee Chan earns. In simple terms, Bob needs to be told to send 50,100 satoshi to Chan, so that Chan can send 50,000 satoshi and keep 100 satoshi.
Similarly, Chan expects a timelock delta of 20 blocks. SO Chan’s incoming HTLC must expire 20 blocks later than Chan’s outgoing HTLC. To achieve this, Alice tells Bob to make his outgoing HTLC to Chan expire at block height 700,038 - 20 blocks later than Chan’s HTLC to Dina.
|
Tip
|
Fees and timelock delta expectations for a channel are set by the difference between incoming and outgoing HTLCs. Since the incoming HTLC is created by the preceding node, the fee and timelock delta is set in the onion payload to that preceding node. Bob is told how to make an HTLC that meets Chan’s fee and timelock expectations. |
Alice serializes this payload in TLV format, as shown (simplified) in Bob’s payload is constructed by Alice below:
Alice has now built the three hop payloads that will be wrapped in an onion. A simplified view of the payloads is shown in Hop payloads for all the hops, below:
Alice must now generate several keys that will be used to encrypt the various layers in the onion.
With these keys, Alice can achieve a high degree of privacy and integrity:
-
Alice can encrypt each layer of the onion so that only the intended recipient can read it.
-
Every intermediary can check that the message is not modified.
-
No one in the path will know who sent this onion or where it is going. Alice doesn’t reveal her identity as the sender or Dina’s identity as the recipient of the payment.
-
Each hop only learns about the previous and next hop.
-
No one can know how long the path is, or where in the path they are.
|
Warning
|
Like a chopped onion, the following technical details may bring tears to your eyes. Feel free to skip to the next section if you get confused. Come back to this and read BOLT #4 - Onion Routing - Packet Construction, if you want to learn more. |
The basis for all the keys used in the onion, is a shared secret that Alice and Bob can both generate independently using the Elliptic Curve Diffie-Hellman (ECDH) algorithm. From the shared secret (ss), they can independently generate four additional keys named rho, mu, um and pad:
- rho
-
Used to generate a stream of random bytes from a stream cipher (used as a CSPRNG). These bytes are used to encrypt/decrypt the message body as well as filler zero bytes during sphinx packet processing.
- mu
-
Used in the Hash-Based Message Authentication Code (HMAC) for integrity/authenticity verification.
- um
-
Used in error reporting.
- pad
-
Used to generate filler bytes for padding the onion to a fixed length.
The relationship between the various keys and how they are generated is shown in the diagram Onion Key Generation, below:
To avoid revealing her identity, Alice does not use her own node’s public key in building the onion. Instead, Alice creates a temporary 32-byte (256-bit) key called the session private key and corresponding session public key. This serves as a temporary "identity" and key for this onion only. From this session key, Alice will build all the other keys that will be used in this onion.
The key generation, random byte generation, ephemeral keys and how they are used in packet construction are specified in three sections of BOLT #4:
For simplicity and to avoid getting too technical, we have not included these details in the book. See the links above if you want to see the inner workings.
One important detail that seems almost magical is the ability for Alice to create a shared secret with another node simply by knowing their public keys. This is based on the invention of Diffie-Hellman key exchange (DH) in the 1970s that revolutionized cryptography. Lightning Onion Routing uses Elliptic Curve Diffie-Hellman (ECDH) on Bitcoin’s secp256k1 curve. It’s such a cool trick that we try to explain it in simple terms in Elliptic Curve Diffie-Hellman (ECDH) explained
Assume Alice’s private key is a and Bob’s private key is b. Using the Elliptic Curve, Alice and Bob each multiply their private key by the generator point G to produce their public keys A and B respectively:
A = aG
B = bG
Now Alice and Bob can use Elliptic Curve Diffie-Hellman Key Exchange to create a shared secret ss, a value that they can both calculate independently without exchanging any information
The shared secret ss is calculated by each by multiplying their own private key with the other’s public key, such that:
ss = aB = bA
But why would these two multiplications result in the same value ss? Follow along, as we demonstrate the math that proves this is possible:
ss
= aB
calculated by Alice who knows both a (her private key) and B (Bob’s public key)
= a(bG)
because we know that B = bG, we substitute
= (ab)G
because of associativity, we can move the parentheses
= (ba)G
because xy = yx (the curve is an abelian group)
= b(aG)
because of associativity, we can move the parentheses
= bA
because A = aG, we can substitute
The result bA can be calculated independently by Bob who knows b (his private key) and A (Alice’s public key).
We have therefore shown that
ss = aB (Alice can calculate this) ss = bA (Bob can calculate this)
Thus, they can each independently calculate ss which they can use as a shared key to symmetrically encrypt secrets between the two of them without communicating the shared secret.
A unique trait of Sphinx as a mix-net packet format is that rather than include a distinct session key for each hop in the route, which would increase the size of the mix-net packet dramatically, clever blinding scheme is used deterministically randomize the session key at each hop.
In practice, this little trick allows us to keep the onion packet as compact as possible while still retaining the desired security properties.
The session key for hop i is derived using the node public key, and derived shared secret of hop i-1:
session_key_i = session_key_{i-1} * SHA256(node_pubkey_{i-1} || shared_secret_{i-1})In other words, we take the session key of the prior hop, and multiply it by a value derived from the public key and derived shared secret for that hop.
As EC multiplication can be performed on a public key without knowledge of the private key, each hop is able to re-randomize the session key for the next hop in a deterministic fashion.
The creator of the onion packet knows all the shared secrets (as they’re encrypted the packet uniquely for each hop), and thus are able to derive all the blinding factors.
This knowledge allows them to derive all the session keys used up front during packet generation.
Note that the very first hop uses the original session key generated, as this key is used to kick off the session key blinding by each subsequent hop.
The process of wrapping the onion is detailed in BOLT #4 - Onion Routing - Packet Construction.
In this section we will describe this process at a high-level and somewhat simplified - omitting certain details.
We’ve mentioned the fact that none of the "hop" nodes know how long the path is, or where they are in the path. How is this possible?
If you have a set of directions, even if encrypted, can’t you tell how far you are from the beginning or end simply by looking at where in the list of directions you are?
The "trick" used in onion routing is to always make the path (the list of directions), the same length for every node. This is achieved by keeping the onion packet the same length at every step.
At each hop, the hop payload appears at the beginning of the onion payload, followed by what seem to be 19 more hop payloads. Every hop sees itself as the first of 20 hops.
|
Tip
|
The onion payload is 1300 bytes. Each hop payload is 65 bytes or less (padded to 65 bytes if less). So the total onion payload can fit 20 hop payloads (1300 = 20 * 65). The maximum onion routed path is therefore 20 hops. |
As each layer is "peeled off", more filler data (essentially junk) is added at the end of the onion payload so the next hop gets an onion of the same size and is once again the "first hop" in the onion.
The onion size is 1366 bytes structured as shown in The onion packet below:
-
1 byte: A version byte
-
33 bytes: A compressed public session key (Alice’s session key) from which the per-hop shared secret (Shared secret generation) can be generated without revealing Alice’s identity
-
1300 bytes: The actual onion payload containing the instructions for each hop
-
32 bytes: An HMAC integrity checksum
A unique trait of Sphinx as a mix-net packet format is that rather than include a distinct session key for each hop in the route, which would increase the size of the mix-net packet dramatically, instead a clever blinding scheme is used to deterministically randomize the session key at each hop.
In practice, this little trick allows us to keep the onion packet as compact as possible while still retaining the desired security properties.
Here is the process of wrapping the onion, outlined below. Come back to this list, as we explore each step with our real-world example.
For each hop the sender (Alice) repeats the same process:
-
Alice generates the per-hop shared secret and the rho, mu, and pad keys
-
Alice generates 1300 bytes of filler and fills the 1300-byte onion payload field with this filler.
-
Alice calculates the HMAC for the hop payload (zeros for the final hop).
-
Alice calculates the length of the hop payload + HMAC + space to store the length itself
-
Alice right shifts the onion payload by the calculated space needed to fit the hop payload. The rightmost "filler" data is discarded, making enough space on the left for the payload.
-
Alice inserts the length + hop payload + HMAC at the front of the payload field in the space made from shifting the filler.
-
Alice uses the rho key to generate a 1300 byte one-time-pad.
-
Alice obfuscates the entire onion payload by XOR-ing with the bytes generated from rho.
-
Alice calculates the HMAC of the onion payload, using the mu key.
-
Alice adds the session public key (so that the hop can calculate the shared secret)
-
Alice adds the version number.
-
Deterministically re-blinds the session key using a value derived by hashing the shared secret and prior hop’s public key.
Next, Alice repeats the process. The new keys are calculated, the onion payload is shifted (dropping more junk), the new hop payload is added to the front and the whole onion payload encrypted with the rho byte-stream for the next hop.
For the final hop, the HMAC included in step #3 over the plaintext instructions is actually all zero.
The final hop uses this signal to determine that it is indeed the final hop in the route.
Alternatively, the fact that the short_chan_id included in the payload to denote the "next hop" is all zero can be used as well.
Note that at each phase the mu key is used to generate an HMAC over the encrypted (from the PoV of the node processing the payload) onion packet, as well as over the contents of the packet with a single layer of encryption removed. This outer HMAC allows the node processing the packet to verify the integrity of the onion packet (no bytes modified). The inner HMAC is then revealed during the inverse of the "shift and encrypt" routine described above, which serves as the outer HMAC for the next hop.
As a reminder, the onion is wrapped by starting at the end of the path from Dina, the final node or recipient. Then the path is built in reverse all the way back to the sender, Alice.
Alice starts with an empty 1300 byte field, the fixed-length onion payload. Then, Alice fills the onion payload with a pseudo-random byte stream "filler", that is generated from the pad key.
|
Note
|
Random byte-stream generation uses the ChaCha20 algorithm, as a Cryptographic Secure Pseudo-Random Number Generator (CSPRNG). Such an algorithm will generate a deterministic, long non-repeating stream of seemingly random bytes from an initial seed. The details are specified in BOLT #4 - Onion Routing - Pseudo Random Byte Stream. |
This is shown in Filling the onion payload with a random byte-stream:
Alice will now insert Dina’s hop payload into the left side of the 1300 byte array, shifting the filler to the right and discarding anything that overflows. This is visualized in Adding Dina’s hop payload:
Another way to look at this is that Alice measures the length of Dina’s hop payload, shifts the filler right to create an equal space in the left side of the onion payload and inserts Dina’s payload in that space.
Next row down we see the result: the 1300 byte onion payload contains Dina’s hop payload and then the filler byte-stream filling up the rest of the space.
Next, Alice obfuscates the entire onion payload so that only Dina can read it.
To do this, Alice generates a byte-stream using the rho key (which Dina also knows). Alice uses a bitwise exclusive-or (XOR) between the bits of the onion payload and the byte-stream created from rho. The result appears like a random (or encrypted) byte stream of 1300 bytes length. This step is shown in Obfuscating the onion payload:
One of the properties of XOR is that if you do it twice you get back to the original data. As we will see in the "unwrapping" section, if Dina applies the same XOR operation with the byte-stream generated from rho, it will reveal the original onion payload.
|
Tip
|
XOR is an involutory function which means that if it is applied twice it undoes itself. Specifically XOR(XOR(a, b), b) = a. This property is used extensively in symmetric-key cryptography. |
Since only Alice and Dina have the rho key (derived from Alice and Dina’s shared secret), only they can do this. Effectively, this encrypts the onion payload for Dina’s eyes only.
Finally, Alice calculates a Hash-based Message Authentication Code (HMAC) for Dina’s payload, which uses the mu key as it’s initialization key. This is shown in Adding an HMAC integrity checksum to Dina’s hop payload:
The HMAC acts as a secure checksum and helps Dina verify the integrity of the hop payload. The 32-byte HMAC is appended to Dina’s hop payload. Note that we compute the HMAC over the encrypted data rather then over the plaintext data. This is known as "encrypt-then-mac" and is the recommended way to use a MAC, as it provides both plaintext and cihpertext integrity.
Modern authenticated encryption also allows for the use of an optional set of plaintext bytes to also be authenticated known as "associated data". In practice, this is usually something like a plaintext packet header or other auxiliary information. By including this associated data in the payload to be authenticated (MAC’d), the verifier of the MAC ensures that this associated data hasn’t been tampered with (eg: swapping out the plaintext header on an encrypted packet).
In the context of the Lightning Network, this associated data is used to strengthen the replay protection of this scheme. As we’ll learn more below, replay protection ensures that an attacker can’t retransmit (replay) a packet into the network and observe its the resulting path. Instead, intermediate nodes are able to use the defined replay protection measures to detect, and reject a replayed packet. The base Sphinx packet format uses a log of all the ephemeral secret keys used to detect replays. If a secret key is ever used again, then the node can detect it, and reject the packet.
The nature of HTLCs in the Lightning Network allows us to further strengthen the replay protection by adding an additional economic incentive. Remember that the payment hash of an HTLC can only ever safely be used (for a complete payment) once. If a payment hash is used again, and traverses a node that has already seen the payment secret for that hash, then they can simply pull the funds, and collect the entire payment amount without forwarding! We can use this fact to strengthen the replay protection by requiring that the payment hash is included in our HMAC computation as the associated data. With this added step, attempting to replay an onion packet also requires the sender to commit to using the same payment hash. As a result, on top of the normal replay protection, an attacker also stands to lose the entire amount of the HTLC replayed as well.
One consideration with the ever-increasing set of session keys stored for replay protection is: are nodes able to reclaim this space? In the context of the Lightning Network, the answer is: yes! Once again, due to the unique attributes of the HTLC construct, we can make a further improvement over the base Sphinx protocol. Given that HTLCs are time locked contracts based on the absolute block height, once an HTLC has expired, then the contract is effectively permanently closed. As a result, nodes can use this CLTV expiration height as an indicator to know when it’s safe to garbage collect an entry in the anti-replay log.
In Wrapping the onion for Chan we see the steps used to wrap Chan’s hop payload in the onion. These are the same steps Alice used to wrap Dina’s hop payload.
Alice starts with the 1300 onion payload created for Dina. The first 65 (or less) bytes of this are Dina’s payload obfuscated and the rest is filler. Alice must be careful not to overwrite Dina’s payload.
Next, Alice needs to locate the ephemeral public key (that was generated at the very start for each hop) that will be prepended to the routing packet at this hop.
Remember that rather than include a unique ephemeral public key (that the sender and intermediate node use in an ECDH operation to arrive a shared secret), Sphinx uses a single ephemeral public key that is deterministically randomized at each hop.
When processing the packet, Dina will use her shared secret and public key to derive the blinding value (b_dina) and use that to re-randomize the ephemeral public key, in an identical operation to what Alice performs during initial packet construction.
Alice adds an inner HMAC checksum to Chan’s payload and inserts it at the "front" (left side) of the onion payload, shifting the existing payload to the right by an equal amount. Remember that there are effectively two HMACs used in the scheme: the outer and the inner HMAC. In this case, Chan’s inner HMAC is actually Dina’s outer HMAC.
Now Chan’s payload is in the front of the onion. When Chan sees this he has no idea how many payloads came before or after. It looks like the 1st of 20 hops always!
Next, Alice obfuscates the entire payload by XOR with the byte-stream generated from the Alice-Chan rho key. Only Alice and Chan have this rho key and only they can produce the byte stream to obfuscate and de-obfuscate the onion. Finally, as we did in the earlier step, we compute Chan’s outer HMAC, which is what she’ll use to verify the integrity of the encrypted onion packet.
In Wrapping the onion for Bob we see the steps used to wrap Bob’s hop payload in the onion.
All right, by now this is easy!
Start with the onion payload (obfuscated) containing Chan’s and Dina’s hop payloads.
Obtain the session key for this hop dervied from the blinding factor generated by the prior hop. Include the prior hop’s outer HMAC as this hop’s inner HMAC. Insert Bob’s hop payload at the beginning and shift everything else over to the right, dropping a Bob-hop-payload-size chunk from the end (it was filler anyway).
Obfuscate the whole thing XOR with the rho key from the Alice-Bob-shared-secret so that only Bob can unwrap this.
Calculate the outer HMAC and stick it on the end of Bob’s hop payload.
The final onion payload is ready to be sent to Bob. Alice doesn’t need to add any more hop payloads.
Alice calculates an HMAC for the onion payload, to cryptographically secure it with a checksum that Bob can verify.
Alice adds a 33-byte public session key that will be used by each hop to generate a shared-secret and the rho, mu, and pad keys.
Finally Alice puts the onion version number (0 currently) in the front.
This allows for future upgrades of the onion packet format.
The result can be seen in The onion packet below:
In this section we will look at how the onion packet is forwarded and how HTLCs are deployed along the path.
Onion packets are sent as part of the update_add_htlc message. If you recall from [update_add_htlc], in [channel_operation], we saw the contents of the update_add_htlc message were as follows:
[channel_id:channel_id] [u64:id] [u64:amount_msat] [sha256:payment_hash] [u32:cltv_expiry] [1366*byte:onion_routing_packet]
You will recall that this message is sent by one channel partner to ask the other channel partner to add an HTLC. This is how Alice will ask Bob to add an HTLC to pay Dina. Now you understand the purpose of the last field onion_routing_packet, which is 1366 bytes long. It’s the fully-wrapped onion packet we just constructed!
Alice will send the update_add_htlc message to Bob. Let’s look at what this message will contain:
- channel_id
-
This field will contain the Alice-Bob channel ID, which in our example is 0000031e192ca1 (see A detailed path constructed from gossiped channel and node information).
- id
-
The ID of this HTLC in this channel, starting at 0.
- amount_msat
-
The amount of the HTLC, 50,200,000 milli-satoshis.
- payment_hash
-
The RIPEMD160(SHA256) payment hash, 9e017f6767971ed7cea17f98528d5f5c0ccb2c71.
- cltv_expiry
-
The expiry timelock for the HTLC will be 700,058. Alice adds 20 blocks to the expiry set in Bob’s payload according to Bob’s negotiated cltv_expiry_delta.
- onion_routing_packet
-
The final onion packet Alice constructed with all the hop payloads!
As we saw in [channel_operation], Bob will add the HTLC to the commitment transactions and update the state of the channel with Alice.
Bob will unwrap the onion he received from Alice as follows:
-
Bob takes the session key from the onion packet and derives the Alice-Bob shared secret.
-
Bob generates the mu key from the shared secret and uses it to verify the onion packet HMAC checksum.
Now that Bob has generated the shared key and verified the HMAC, he can start unwrapping the 1300 byte onion payload inside the onion packet. The goal is for Bob to retrieve his own hop payload and then forward the remaining onion to the next hop.
If Bob extracts and removes his hop payload, the remaining onion will not be 1300 bytes, it will be shorter! So the next hop will know that they are not the first hop and will be able to detect how long the path is. To prevent this, Bob needs to add more filler to refill the onion.
Bob generates filler in a slightly different way than Alice, but following the same general principle.
First, Bob extends the onion payload by 1300 bytes and filles them with 0 values. Now the onion packet is 2600 bytes long, with the first half containing the data Alice sent and the next half containing zeroes. This operation is shown in Bob extends the onion payload by 1300 (zero-filled) bytes:
This empty space will become obfuscated and turn into "filler", by the same process that Bob uses to deobfuscate his own hop payload. Let’s see how that works.
Next, Bob will generate the rho key from the Alice-Bob shared key. He will use this to generate a 2600 byte stream, using the ChaCha20 algorithm.
|
Tip
|
The first 1300 bytes of the byte stream generated by Bob are exactly the same as those generated by Alice using the rho key. |
Next, Bob applies the 2600 bytes of rho byte stream to the 2600 bytes onion payload with a bitwise XOR operation.
The first 1300 bytes will become de-obfuscated by this XOR operation, because it is the same operation Alice applied and XOR is involutory. So Bob will reveal his hop payload followed by some data that seems scrambled.
At the same time, applying the rho byte stream to the 1300 zeroes that were added to the onion payload will turn them into seemingly random filler data. This operation is shown in Bob de-obfuscates the onion, obfuscates the filler:
Remember that an inner HMAC is included for each hop, then will then become the outer HMAC for the next hop. In this case, Bob extracts the inner HMAC (he’s already verified the integrity of the encrypted packet w/ the outer HMAC), and puts it aside as he’ll append it to the deobfuscated packet to allow Chan to verify the HMAC of her encrypted packet.
Now Bob can remove his hop payload from the front of the onion and left-shift the remaining data. An amount of data equal to Bob’s hop payload from the second-half 1300 bytes of filler will now shift into the onion payload space. This is shown in Bob removes the hop payload and left-shifts the rest, filling the gap with new filler:
Now Bob can keep the first half 1300 bytes, and discard the extended (filler) 1300 bytes.
Bob now has a 1300 byte onion packet to send to the next hop. It is almost identical to the onion payload that Alice had created for Chan, except that the last 65 or so bytes of filler was put there by Bob and will be different.
No one can tell the difference between filler put there by Alice and filler put there by Bob. Filler is filler! It’s all random bytes anyway. Note that if Bob (or one of Bob’s other aliases) is present in the route in two disctinct locations, then they can tell the difference as the base protocol always uses the same payment hash across the entire route. Atomic Multipath Payments (AMP), and Point Time Locked Contrats (PTLCs) eliminate the correlation vector by randomizing the payment identifier across each route/hop.
Bob now copies the onion payload into the onion packet, appends the outer HMAC for chan, re-randomizes the session key (the same way Alice the sender does) with the Elliptic Curve multiplication operation, and appends a fresh version byte.
In order to re-randomize the session key, bob first computes the blinding factor for his hop, using his node public key and the shared secret he derived:
b_bob = SHA256(P_bob || shared_secret_bob)With this generated, Bob now re-randomizes the session key by performing an EC multiplication using his session key and the blinding factor:
session_key_chan = session_key_bob * b_bobThe session_key_chan public key will then be appended to the front of the onion packet for processing by Chan.
Bob’s hop payload contains the instructions needed to create an HTLC for Chan.
In the hop payload, Bob finds a short_channel_id, amt_to_forward, and cltv_expiry.
First, Bob checks to see if he has a channel with that short ID. He finds that he has such a channel with Chan.
Next, Bob confirms that the outgoing amount (50,100 satoshis) is less than the incoming amount (50,200 satoshis) and therefore Bob’s fee expectations are met.
Similarly, Bob checks that the outgoing cltv_expiry is less than the incoming cltv_expiry giving Bob enough time to claim the incoming HTLC if there is a breach.
Bob now constructs and HTLC to send to Chan, as follows:
- channel_id
-
This field will contain the Bob-Chan channel ID, which in our example is 000004040a61f0 (see A detailed path constructed from gossiped channel and node information).
- id
-
The ID of this HTLC in this channel, starting at 0.
- amount_msat
-
The amount of the HTLC, 50,100,000 milli-satoshis.
- payment_hash
-
The RIPEMD160(SHA256) payment hash, 9e017f6767971ed7cea17f98528d5f5c0ccb2c71. This is the same as the payment hash from Alice’s HTLC.
- cltv_expiry
-
The expiry timelock for the HTLC will be 700,038.
- onion_routing_packet
-
The onion packet Bob reconstructed after removing his hop payload.
Chan repeats the exact same process as Bob:
-
Chan receives the update_add_htlc and processes the HTLC request, adding it to commitment transactions.
-
Chan generates the Alice-Chan shared key and the mu subkey
-
Chan verifies the onion packet HMAC, then extracts the 1300 onion payload
-
Chan extends the onion payload by 1300 extra bytes, filling it with zeroes.
-
Chan uses the rho key to produce 2600 bytes.
-
Chan uses the generated byte stream to XOR and de-obfuscate the onion payload. Simultaneously the XOR operation obfuscates the extra 1300 zeroes, turning them into filler
-
Chan extracts the inner HMAC in the payload, which will become the outer HMAC for Dina.
-
Chan removes his hop payload and left-shifts the onion payload by the same amount. Some of the filler generated in the 1300 extended bytes move into the first-half 1300 bytes becoming part of the onion payload.
-
Chan constructs the onion packet for Dina with this onion payload.
-
Chan builds an update_add_htlc message for Dina and inserts the onion packet into it.
-
Chan sends the update_add_htlc to Dina
-
Chan re-randomizes the session key as Bob did in the prior hop for Dina.
When Dina receives the update_add_htlc message from Chan, she knows from the payment_hash that this is a payment for her. She knows she is the last hop in the onion.
Dina follows the exact same process as Bob and Chan to verify and unwrap the onion, except she doesn’t construct new filler and doesn’t forward anything. Instead, Dina responds to Chan with update_fulfill_htlc to redeem the HTLC. The update_fulfill_htlc will flow backwards along the path until it reaches Alice. All the HTLCs are redeemed and channel balances are updated. The payment is complete!
This far we’ve looked at the forward propagation of the onion establishing the HTLCs, and the backwards propagation of the payment secret unwinding the HTLCs once payment is successful.
There is another very important function of onion routing which is error return. If there is a problem with the payment, onion, or hops, we must propagate an error backwards to inform all nodes of the failure and unwind any HTLCs.
Errors generally fall into 3 categories: onion failures, node failures and channel failures. These furthermore may be subdivided into permanent and transient errors. Finally, some errors contain channel updates to help with future payment delivery attempts.
|
Note
|
Unlike messages in the Peer-to-Peer (P2P) protocol (defined in BOLT #2), errors are not sent as P2P messages but wrapped inside onion return packets and following the reverse of the onion path (back-propagating). |
Error return is defined in BOLT #4 - Onion Routing - Returning Errors.
Errors are encoded by the returning node (the one that discovered an error) encoded in a return packet a follows:
[32*byte:hmac]
[u16:failure_len]
[failure_len*byte:failuremsg]
[u16:pad_len]
[pad_len*byte:pad]
The return packet HMAC verification checksum is calculated with the um key, generated from the shared secret established by the onion.
|
Tip
|
The "um" key name is the reverse of the "mu" name, indicating the same use but in the opposite direction (back-propagation) |
Next, the returning node generates a ammag (inverse of the word "gamma") key and obfuscates the return packet using an XOR operation with a byte-stream generated from ammag.
Finally the return node sends the return packet to the hop from which it received the original onion.
Each hop receiving an error, will generate an ammag key and obfuscate the return packet again using an XOR operation with the byte-stream from ammag.
Eventually, the sender (origin node) receives a return packet. It will then generate ammag and um keys for each hop and XOR de-obfuscate the return error iteratively until it reveals the return packet.
The failuremsg is defined in BOLT #4 - Onion Routing - Failure Messages.
A failure message consists of a 2-byte failure code followed by the data applicable to that failure type.
The top byte of the failure_code is a set of binary flags that can be combined (with binary OR):
-
0x8000 (BADONION): unparsable onion encrypted by sending peer
-
0x4000 (PERM): permanent failure (otherwise transient)
-
0x2000 (NODE): node failure (otherwise channel)
-
0x1000 (UPDATE): new channel update enclosed
The following failure types are currently defined:
In the current implementation of the Lightning Network there is a possibility that a payment attempt becomes "stuck", neither fulfilled nor cancelled by an error. This can happen due to a bug on an intermediary node, a node going offline while handling HTLCs, or a malicious node holding HTLCs without reporting an error. In all of these cases, the HTLC cannot be resolved until it expires. The timelock (CLTV) that is set on every HTLC helps resolve this condition (among other possible HTLC routing and channel failures).
However, this means that the sender of the HTLC has to wait until expiry, and the funds committed to that HTLC remain unavailable until the HTLC expires. Furthermore, the sender cannot retry that same payment, because if they do they run the risk of both the original and the retried payment succeeding - the recipient gets paid twice. This is because once sent an HTLC cannot be "cancelled" by the sender - it either has to fail or expire. Stuck payments, while rare, create an unwanted user experience, where the user’s wallet cannot pay or cancel a payment.
One proposed solution to this problem is called "stuckless payments", and it depends on Point Time-Locked Contracts (PTLCs), which are payment contract that use a different cryptographic primitive than HTLCs (i.e. point addition on the elliptic curve instead of a hash and secret pre-image). PTLCs are cumbersome using ECDSA but much easier with Bitcoin’s Taproot and Schnorr Signature features which were recently locked in for activation in November 2021. It is expected that PTLCs will be implemented in the Lightning Network after these Bitcoin features become activated.
In the payment flow described earlier in the chapter, we assumed that Dina
received an invoice from Alice "out of band," or obtained it via some mechanism
unrelated to the protocol (typically copy/paste or QR code scans). This trait
means that the payment process always takes two steps: first, the sender
obtains an invoice and second, uses the payment hash (encoded in the invoice) to
successfully route an HTLC. The extra round trip required to obtain an invoice
before making a payment may be a bottleneck in applications that involve
streaming micropayments over Lightning. What if we could just "push" a payment
over spontaneously, without having to obtain an invoice from the recipient
first? The keysend protocol is an end-to-end extension (only the sender and
receiver are aware) to the Lightning protocol that enables spontaneous push
payments.
The modern Lightning protocol uses the TLV (Type-Length-Value) encoding in
the onion to encode information that tells each node where and how to
forward the payment. Leveraging the TLV format, each piece of routing information
(like the next node to which to pass the HTLC) is assigned a specific type (or key)
encoded as a BigSize variable length integer (max sized as as 64-bit
integer). These "essential" (reversed values below 65536) types are defined
in BOLT-04 along with the rest of the onion routing details. Onion types with a
value greater than 65536 are intended to be used by wallets and applications
as "custom records."
Custom records allow payment applications to attach additional metadata or
context to a payment as key/value pairs in the onion. Since the custom records
are included in the onion payload itself, like all other hop contents, the
records are end-to-end encrypted. As the custom records effectively consume a
portion of the fixed-size 1300 bytes onion packet, encoding each key and
value of each custom record reduces the amount of available space for encoding
the rest of the route. In practice, this means that the more onion space used for custom records, the shorter the route can be. Given that each HTLC
packet is fixed-size, custom records don’t add any additional data to an
HTLC; rather, they reallocate bytes that would have been filled with random data
otherwise.
A keysend payment inverts the typical flow of an HTLC where the receiver
reveals a secret pre-image to the sender. Instead, the sender includes the
pre-image within the onion to the receiver and routes the HTLC to the
receiver. The receiver then decrypts the onion payload, and uses the included
pre-image (which must match the payment hash of the HTLC) to settle the
payment. As a result, keysend payments can be carried out without first
obtaining an invoice from the receiver, as the pre-image is "pushed" over to
the receiver. A keysend payment uses a TLV custom record type of 5482373484
to encode a 32-byte pre-image value.
Many streaming Lightning applications use the keysend protocol to continually
stream satoshis to a destination identified by its public key in the network.
Typically, an application will also include metadata such as a
tipping/donation note or other application-level information in addition to
the keysend record.
The Lightning Network’s onion routing protocol is adapted from the Sphinx Protocol to better serve the needs of a payment network. As such, it offers a huge improvement in privacy and counter-surveillance compared to the public and transparent Bitcoin blockchain.
In [path_finding] we will see how the combination of source routing and onion routing is used by Alice to find a good path and route the payment to Dina. To find a path, Alice first needs to learn about the network topology, which is the topic of the next section [gossip].