Message channel over c-lightning

[AdamISZ]

I made several updates to this description on 29 Dec 2021 as it was substantially out of date.

See commit message for a brief technical summary of implementation in code.

For an explanation of the reasoning for creating this, see the first section of https://github.com/JoinMarket-Org/joinmarket-clientserver/blob/3f8fa0ea8156eeabe40c2a62a3c3e38026dae31f/docs/lightning-message-channels.md.

Some extra points relevant to review and further progress:

• Transactions work. And on regtest/localhost they are a lot faster than using our existing IRC tests (miniircd), which is encouraging, if not hugely meaningful.
• No tests have yet been added (and indeed some existing tests are broken, though that is probably trivial).
• This should work correctly with a mixture of IRC+LN message channels.

Those are more trivial points, but the bigger ones:

• I don't want to go whole-hog to a mainnet ready implementation until others like @chris-belcher @undeath @kristapsk and others have had a chance to give their thoughts. In particular: does this kind of "meta-layer" of communication need some extra features? My best guess is that for now we should embed a version number in all the LN-sent messages such that we can support an upgrade in future (i.e. new bots with a better system can happily talk to old bots on the old system .. this usually means version range). done
• The most important thing that this doesn't do yet is: pass messages over hops. Although bots can always !privmsg directly to other bots, if they have a connection, there is not currently implemented, even an attempt to connect to bots other than the directory nodes that we connect to at the start - see later comments, we now switch to direct messaging once we learn of peers via !fill and !offer messages. This means that it's more the old IRC model of "privmsgs are end to end encrypted" and not really decentralized further than the existing model; but that's just because I haven't tried to code that yet, not because it can't be done. In a similar vein, while the user can configure a list of directory nodes, the code currently only uses the first directory node in the list. This second sentence is also now out of date, multiple directory nodes are supported.
• A more long term one but important will be: the directory node's DOS control measures. I envisage issuance of tokens against small Lightning payments, but you could conceivably use one of the two methods we've already used against Sybil/DOS: PoDLE or fidelity bonds. I like the Chaumian tokens against small LN payments personally, but that's a bunch of new stuff (idea: directory issues its own tokens, user pays a tiny fraction on a per message basis, see e.g. Wabisabi MACs, although that's just one example, maybe such denomination flexibility not needed here). Anything that prevents "million messages per second from one guy" could do as a starting point.

Points of discussion at the code/design level:

About the messaging, the most important thing to read is the set of comments at the top of jmdaemon/lnonion.py (this has been moved to a PR in Joinmarket-Docs) which explains how we translate from (Joinmarket text formatted messages) -> (a slightly extend string containing to/from information to be passed into Lightning) -> (the contents of the sendcustommsg message).

Design-wise, pretty much everything here is arguable. The main goal was to have a class which inherits from jmdaemon.message_channel.MessageChannel and implement it, as irc.py does, and that is achieved here. The slightly obscure differences between privmsg and pubmsg are not as easy to reason about as they might be, but, it's OK. The layers of encoding are kind of painful, but at least we maintain consistency.

The actual message parsing is "asymmetric" in a weird way: messages come in via the plugin, via this hook:

@plugin.hook("custommsg")
on_custommsg(peer_id, payload, plugin, **kwargs):

and they are then handed over to jmdaemon over TCP port (default 49100) to a twisted.protocols.basic.LineReceiver. The messages go out, however, by sending the RPC command sendcustommsg (same format, of course). See this method in LNOnionMessageChannel:

    def _send(self, peerid: str, message: bytes) -> None:

(Since it's only referenced in the doc, tagging here: #415 ).

[AdamISZ]

Running a bot on signet here:

03e5db5523642fef5d65b9a7cedfda9cfe3a4054565c9aefbe32e5d72d086d6b34@m7t5zow7fugzjymfiacq37bxz7niq5pfwult6ukbqibtgzgbyb3hbeid.onion:9735 see below.

... which you can set as a directory node. I'll try to keep it up.

I'll be adding a patch immediately to read the getinfo output correctly for this Tor onion scenario (it currently reads binding which is wrong).

For info on how to set up for a persistent Tor v3 onion, if you're not already (and you probably are, if you run bitcoind with Tor on the machine), check the detailed instructions here: https://lightning.readthedocs.io/TOR.html

.. and then add (or create) this in your ~./lightning/config:

proxy=127.0.0.1:9050
bind-addr=127.0.0.1:9735
addr=statictor:127.0.0.1:9051
always-use-proxy=true

If you're planning on messing around testing, like me, with multiple bots, just set different ports in bind-addr.

Edit: sanity checked i could use it, seemed to work, created this tx with another bot on a different onion/LN node: https://explorer.bc-2.jp/tx/533901e14c7f3ee7532485061d4b018675d67da39c6ab7cc363c152d7323c428

[AdamISZ]

@kristapsk thanks for taking a look. actually due to me swapping around computers, i think the node I listed above is not currently running, but I'll probably bring it back up shortly after I finish something else (just in case it's useful).

[kristapsk]

I plan to setup and run a signet instance permanently (at least for some time) too.

[AdamISZ]

As of c8c2227 things will be quite broken; pushing just to help with testing of installation and setup in fresh env.

[AdamISZ]

After f0e44bf this is hopefully getting significantly easier to test. If you run the install from scratch with ./install.sh --with-ln-messaging it will build c-lightning and install it in deps (compiled with onion messaging flag; with release version it doesn't work). Then, setting sensible config in the created default [MESSAGING:lightning1] should allow you to run tests in some --datadir=.. setting; in that directory, you'll see a new sub-folder lightning under which you get all the c-lightning data and config (it's all done automatically). For example you can run a few nodes on signet all on your own machine like this.

I'll drop a note here that I think it's maybe too hard to keep the network layer separation pure here; if we end up using these lightning nodes for payments then they're not performing pure message handling any more; right now, I need the message channel to have access to the rpc socket that c-lightning gives, and it's too much of a pain to figure out how to isolate that messaging, so for now, this PR is not compatible with having jmdaemon run on a separate environment to jmclient. But I think most people aren't too bothered.

[AdamISZ]

As of 1f11993 I can do coinjoins with the lightningd instances connecting over onion services remotely. I will next alter the documentation to explain how to set it up now; it'll be quite a lot easier. done.

But there are a lot of details still to address, noting them here (and may add more):

• Include in docs the diff for setup of a directory node (just statictor not autotor basically so you can share the onion address). done.
• Make sure the lightningd instance shuts down for sendpayment; currently it's not getting the shutdown signal when sendpayment terminates. done.
• Add some tests of jmdaemon/jmdaemon/lnonion.py and get the test suite passing. done.
• Fix bug where disconnection of dn or non-dn raises RPC error. done.

[AdamISZ]

@kristapsk at some point if you could find time to run through the installation changes in install.sh I would appreciate it; you are much more knowledgeable on that so no doubt you will see some things needing improving.

Both you and maybe @openoms perhaps might find time to try the installation and see if you can get a bot running based on what's in the updated doc here. Or anyone else of course :)

[AdamISZ]

Updated my directory host and port:

0344bc51c0b0cf58ebfee222bdd8ff4855ac3becd7a9c8bff8ff08100771179047@mvm5ffyipzf4tis2g753w7q25evcqmvj6qnnwgr3dkpyukomykzvwuad.onion:9736

This is working atm, please try to break it :)

[AdamISZ]

Rebased on master as of 4fff271

Working on tests now.

[AdamISZ]

Now test suite is passing and also I've addressed the minor bug points above, I'm removed [WIP] from the title. It's not ready to be merged but definitely ready to start getting detailed test results and review.

Summary for reviewers:

• An installation change in install.sh which bundles c-lightning using the flag --with-ln-messaging. While such bundling is only preferable for certain use cases (many might prefer to use their existing node), there is currently no choice as we use the --experimental.. features and require a build with that flag, and can't expect users to compile that way themselves. The installation embeds the compiled binaries similar to how we deal with our other compiled dependencies.
• A new messaging section in the config, which is parsed in configure.py to know how to start the local node with our joinmarket plugin, and how to connect that over TCP to our actual joinmarket process.
• A new LNOnionMessageChannel child class of MessageChannel analogous, of course, to IRCMessageChannel and implementing the same interface, this is in jmdaemon/lnonion.py, which also has a number of classes to represent the interaction with the lightning node, in particular the concept of the RPC call sendonionmessage. Note that there are no code changes in jmdaemon/message_channel.py, by design. Note that the top of this module is a wall of text describing the translation between these onionmessage objects passed over lightning-rpc, and the Joinmarket's existing message protocol syntax. Some details about this, such as the integer message types, or the absence of any version or ping/pong messages, are very much open to discussion and ideally before we start using this!
• The plugin itself is also located in the jmdaemon package and is called jmcl.py. It is very simple, providing hooks for when our LN node receives onion message, or connect and disconnect calls. The data is passed completely transparently over a TCP socket to the joinmarket process. This is just passed as a parameter when we start up the lightningd process, which we do on startup of our Joinmarket script, if we can see from the config, that it is required (because we have a lightning message channel configured).
• Confused about the data flow? Understandable because it's asymmetrical. Inbound data goes (p2p ln) -> our lightningd -> our plugin hooks -> tcp socket -> the running joinmarket script -> LNOnionMessageChannel.receive_msg. Outbound data goes LNOnionMessageChannel._send() -> LightningRpc object (from new dependency pyln_client) -> our lightningd with the sendonionmessage call -> (p2p ln).
• Test suite additions in jmdaemon/test/test_lnonion_message_channel.py.

[AdamISZ]

After 6e712b2 the coinjoin peers are successfully passing messages peer to peer for the privmsg part of the conversation, i.e. bypassing the directory node(s) entirely.

While it's still fairly primitive, this was the level of decentralization I was hoping to implement here. Pubmsgs will go over multiple "public" directory nodes, acting essentially just as IRC servers do today for that part of the flow, while the transaction negotiation can be mostly or entirely happening p2p. This will take a lot of the load off of the directory nodes and be an extra layer of metadata privacy on top of the existing end-to-end encryption.

[kristapsk]

Random thought - if we build c-lightning inside jmvenv, why can't we also do the same with Tor? That would allow enabling it by default, without the need for the user to configure Tor settings manually. Tor daemon is not very hungry process, don't think there should be any problems even with running multiple instances in parallel.

[AdamISZ]

Random thought - if we build c-lightning inside jmvenv, why can't we also do the same with Tor? That would allow enabling it by default, without the need for the user to configure Tor settings manually. Tor daemon is not very hungry process, don't think there should be any problems even with running multiple instances in parallel.

It seems reasonable, but I'm loth to say "Yes!" without having tried it :) If someone else can add that into the installation that'd be cool indeed. Maybe in an additional PR.

[kristapsk]

I will try to hack it together as a separate PR.

[kristapsk]

This currently only works for Debian, somebody should look at required Darwin (macOS) deps and add them here too, like in deps_install().

[kristapsk]

Also, [[ ${use_os_deps_check} != '1' ]] check should be added too.

[AdamISZ]

Partially done in 7a5e4c3 .

I added the use_os_deps_check but for the MacOS i've just kind of "given it a start".

Probably someone who runs MacOS and can follow the instructions as per the guide here. From a brief read I note that there are more exceptional conditions to handle here, so it may need a few extra steps.

For now I have just added the basic list of brew install dependencies given there, but it's a shot in the dark. @kristapsk feel free to add commits that correct errors in this stuff.

[AdamISZ]

Rebased on master as of 97cf02c

[AdamISZ]

Given developments as per #1182 , closing this. Will not delete the branch, though, in case it's useful.