Onion-based message channels with directory nodes

Joinmarket bots run their own onion services allowing inbound connections.
Both takers and makers connect to other makers at the mentioned
onion services, over Tor.
Directory nodes run persistent onion services allowing peers to
find other (maker) peers to connect to, and also forwarding
messages where necessary.
This is implemented as an alternative to IRC, i.e. a new
implementation of the abstract class MessageChannel, in onionmc.py.

Note that using both this *and* IRC servers is supported; Joinmarket
supports multiple, redundant different communication methods,
simultaneously.

Messaging is done with a derived class of twisted's LineReceiver,
and there is an additional layer of syntax, similar to but not the
same as the IRC syntax for ensuring that messages are passed with
the same J5.. nick as is used on IRC. This allows us to keep the
message signing logic the same as before. As well as Joinmarket line
messages, we use additional control messages to communicate peer lists,
and to manage connections.
Peers which send messages not conforming to the syntax are dropped.
See JoinMarket-Org/JoinMarket-Docs#12 for
documentation of the syntax.
Connections to directory nodes are robust as for IRC servers, in
that we use a ReconnectingClientFactory to keep trying to re-establish
broken connections with exponential backoff. Connections to maker
peers do not require this feature, as they will often disconnect
in normal operation.
Multiple directory nodes can and should be configured by bots.

docs/onion-message-channels.md

@@ -0,0 +1,173 @@
+# HOW TO SETUP ONION MESSAGE CHANNELS IN JOINMARKET
+
+### Contents
+
+1. [Overview](#overview)
+
+2. [Testing, configuring for signet](#testing)
+
+4. [Directory nodes](#directory)
+
+<a name="overview" />
+
+## Overview
+
+This is a new way for Joinmarket bots to communicate, namely by serving and connecting to Tor onion services. This does not
+introduce any new requirements to your Joinmarket installation, technically, because the use of Payjoin already required the need
+to service such onion services, and connecting to IRC used a SOCKS5 proxy (by default, and used by almost all users) over Tor to
+a remote onion service.
+
+The purpose of this new type of message channel is as follows:
+
+* less reliance on any service external to Joinmarket
+* most of the transaction negotiation will be happening directly peer to peer, not passed over a central server (
+albeit it was and remains E2E encrypted data, in either case)
+* the above can lead to better scalability at large numbers
+* a substantial increase in the speed of transaction negotiation; this is mostly related to the throttling of high bursts of traffic on IRC
+
+The configuration for a user is simple; in their `joinmarket.cfg` they will add a messaging section like this:
+
+```
+[MESSAGING:onion1]
+type = onion
+onion_serving_port = 8082 
+# This is a comma separated list (comma can be omitted if only one item).
+# Each item has format host:port
+directory_nodes = rr6f6qtleiiwic45bby4zwmiwjrj3jsbmcvutwpqxjziaydjydkk5iad.onion:80
+```
+
+Here, I have deliberately omitted the several other settings in this section which will almost always be fine as default;
+see `jmclient/jmclient/configure.py` for what those defaults are, and the extensive comments explaining.
+
+The main point is the list of **directory nodes** (the one shown here is one being run on signet, right now), which will
+be comma separated if multiple directory nodes are configured (we expect there will be 2 or 3 as a normal situation).
+The `onion_serving_port` is on which port on the local machine the onion service is served.
+The `type` field must always be `onion` in this case, and distinguishes it from IRC message channels and others.
+
+### Can/should I still run IRC message channels?
+
+In short, yes.
+
+### Do I need to configure Tor, and if so, how?
+
+These message channels use both outbound and inbound connections to onion services (or "hidden services").
+
+As previously mentioned, both of these features were already in use in Joinmarket. If you never served an
+onion service before, it should work fine as long as you have the Tor service running in the background,
+and the default control port 9051 (if not, change that value in the `joinmarket.cfg`, see above.
+
+#### Why not use Lightning based onions?
+
+(*Feel free to skip this section if you don't know what "Lightning based onions" refers to!*). The reason this architecture is
+proposed as an alternative to the previously suggested Lightning-node-based network (see
+[this PR](https://github.com/JoinMarket-Org/joinmarket-clientserver/pull/1000)), is mostly that:
+
+* the latter has a bunch of extra installation and maintenance dependencies (just one example: pyln-client requires coincurve, which we just
+removed)
+* the latter requires establishing a new node "identity" which can be refreshed, but that creates more concern
+* longer term ideas to integrate Lightning payments to the coinjoin workflow (and vice versa!) are not realizable yet
+* using multi-hop onion messaging in the LN network itself is also a way off, and a bit problematic
+
+So the short version is: the Lightning based alternative is certainly feasible, but has a lot more baggage that can't really be justified
+unless we're actually using it for something.
+
+
+<a name="testing" />
+
+## Testing, and configuring for signet.
+
+This testing section focuses on signet since that will be the less troublesome way of getting involved in tests for
+the non-hardcore JM developer :)
+
+(For the latter, please use the regtest setup by running `test/e2e-coinjoin-test.py` under `pytest`,
+and pay attention to the settings in `regtest_joinmarket.cfg`.)
+
+There is no separate/special configuration for signet other than the configuration that is already needed for running
+Joinmarket against a signet backend (so e.g. RPC port of 38332).
+
+Add the `[MESSAGING:onion1]` message channel section to your `joinmarket.cfg`, as listed above, including the
+signet directory node listed above (rr6f6qtleiiwic45bby4zwmiwjrj3jsbmcvutwpqxjziaydjydkk5iad.onion:80), and,
+for the simplest test, remove the other `[MESSAGING:*]` sections that you have.
+
+Then just make sure your bot has some signet coins and try running as maker or taker or both.
+
+<a name="directory" />
+
+## Directory nodes
+
+**This last section is for people with a lot of technical knowledge in this area,
+who would like to help by running a directory node. You can ignore it if that does not apply.**.
+
+This requires a long running bot. It should be on a server you can keep running permanently, so perhaps a VPS,
+but in any case, very high uptime. For reliability it also makes sense to configure to run as a systemd service.
+
+A note: in this early stage, the usage of Lightning is only really network-layer stuff, and the usage of bitcoin, is none; feel free to add elements that remove any need for a backend bitcoin blockchain, but beware: future upgrades *could* mean that the directory node really does need the bitcoin backend.
+
+#### Joinmarket-specific configuration
+
+Add `hidden_service_dir` to your `[MESSAGING:onion1]` with a directory accessible to your user. You may want to lock this down
+a bit!
+The point to understand is: Joinmarket's `jmbase.JMHiddenService` will, if configured with a non-empty `hidden_service_dir`
+field, actually start an *independent* instance of Tor specifically for serving this, under the current user.
+(our tor interface library `txtorcon` needs read access to the Tor HS dir, so it's troublesome to do this another way).
+
+##### Question: How to configure the `directory-nodes` list in our `joinmarket.cfg` for this directory node bot?
+
+Answer: **you must only enter your own node in this list!** (otherwise you may find your bot infinitely rebroadcasting messages).
+
+
+#### Suggested setup of a service:
+
+You will need two components: bitcoind, and Joinmarket itself, which you can run as a yg.
+Since this task is going to be attempted by someone with significant technical knowledge,
+only an outline is provided here; several details will need to be filled in.
+Here is a sketch of how the systemd service files can be set up for signet:
+
+If someone wants to put together a docker setup of this for a more "one-click install", that would be great.
+
+1. bitcoin-signet.service
+
+```
+[Unit]
+Description=bitcoind signet
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+ExecStart=/usr/local/bin/bitcoind -signet
+User=user
+
+[Install]
+WantedBy=multi-user.target
+```
+
+This is deliberately a super-basic setup (see above). Don't forget to setup your `bitcoin.conf` as usual,
+for the bitcoin user, and make it match (specifically in terms of RPC) what you set up for Lightning below.
+
+
+2.
+
+```
+[Unit]
+Description=joinmarket directory node on signet
+Requires=bitcoin-signet.service
+After=bitcoin-signet.service
+
+[Service]
+Type=simple
+ExecStart=/bin/bash -c 'cd /path/to/joinmarket-clientserver && source jmvenv/bin/activate && cd scripts && echo -n "password" | python yg-privacyenhanced.py --wallet-password-stdin --datadir=/custom/joinmarket-datadir some-signet-wallet.jmdat'
+User=user
+
+[Install]
+WantedBy=multi-user.target
+```
+
+To state the obvious, the idea here is that this second service will run the JM directory node and have a dependency on the previous one,
+to ensure they start up in the correct order.
+
+Re: password echo, obviously this kind of password entry is bad;
+for now we needn't worry as these nodes don't need to carry any real coins (and it's better they don't!).
+Later we may need to change that (though of course you can use standard measures to protect the box).
+
+TODO: add some material on network hardening/firewalls here, I guess.

jmbase/jmbase/twisted_utils.py

@@ -128,16 +128,23 @@ def config_to_hs_ports(virtual_port, host, port):
 class JMHiddenService(object):
     """ Wrapper class around the actions needed to
     create and serve on a hidden service; an object of
-    type Resource must be provided in the constructor,
-    which does the HTTP serving actions (GET, POST serving).
+    type either Resource or server.ProtocolFactory must
+    be provided in the constructor, which does the HTTP
+    (GET, POST) or other protocol serving actions.
     """
-    def __init__(self, resource, info_callback, error_callback,
-                 onion_hostname_callback, tor_control_host,
+    def __init__(self, proto_factory_or_resource, info_callback,
+                 error_callback, onion_hostname_callback, tor_control_host,
                  tor_control_port, serving_host, serving_port,
-                 virtual_port = None,
-                 shutdown_callback = None):
-        self.site = Site(resource)
-        self.site.displayTracebacks = False
+                 virtual_port=None,
+                 shutdown_callback=None,
+                 hidden_service_dir=""):
+        if isinstance(proto_factory_or_resource, Resource):
+            # TODO bad naming, in this case it doesn't start
+            # out as a protocol factory; a Site is one, a Resource isn't.
+            self.proto_factory = Site(proto_factory_or_resource)
+            self.proto_factory.displayTracebacks = False
+        else:
+            self.proto_factory = proto_factory_or_resource
         self.info_callback = info_callback
         self.error_callback = error_callback
         # this has a separate callback for convenience, it should
@@ -155,26 +162,45 @@ def __init__(self, resource, info_callback, error_callback,
         # config object, so no default here:
         self.serving_host = serving_host
         self.serving_port = serving_port
+        # this is used to serve an onion from the filesystem,
+        # NB: Because of how txtorcon is set up, this option
+        # uses a *separate tor instance* owned by the owner of
+        # this script (because txtorcon needs to read the
+        # HS dir), whereas if this option is "", we set up
+        # an ephemeral HS on the global or pre-existing tor.
+        self.hidden_service_dir = hidden_service_dir
 
     def start_tor(self):
         """ This function executes the workflow
         of starting the hidden service and returning its hostname
         """
         self.info_callback("Attempting to start onion service on port: {} "
                            "...".format(self.virtual_port))
-        if str(self.tor_control_host).startswith('unix:'):
-            control_endpoint = UNIXClientEndpoint(reactor,
-                                    self.tor_control_host[5:])
+        if self.hidden_service_dir == "":
+            if str(self.tor_control_host).startswith('unix:'):
+                control_endpoint = UNIXClientEndpoint(reactor,
+                                        self.tor_control_host[5:])
+            else:
+                control_endpoint = TCP4ClientEndpoint(reactor,
+                                self.tor_control_host, self.tor_control_port)
+            d = txtorcon.connect(reactor, control_endpoint)
+            d.addCallback(self.create_onion_ep)
+            d.addErrback(self.setup_failed)
+            # TODO: add errbacks to the next two calls in
+            # the chain:
+            d.addCallback(self.onion_listen)
+            d.addCallback(self.print_host)
         else:
-            control_endpoint = TCP4ClientEndpoint(reactor,
-                            self.tor_control_host, self.tor_control_port)
-        d = txtorcon.connect(reactor, control_endpoint)
-        d.addCallback(self.create_onion_ep)
-        d.addErrback(self.setup_failed)
-        # TODO: add errbacks to the next two calls in
-        # the chain:
-        d.addCallback(self.onion_listen)
-        d.addCallback(self.print_host)
+            ep = "onion:" + str(self.virtual_port) + ":localPort="
+            ep += str(self.serving_port)
+            # endpoints.TCPHiddenServiceEndpoint creates version 2 by
+            # default for backwards compat (err, txtorcon needs to update that ...)
+            ep += ":version=3"
+            ep += ":hiddenServiceDir="+self.hidden_service_dir
+            onion_endpoint = serverFromString(reactor, ep)
+            d = onion_endpoint.listen(self.proto_factory)
+            d.addCallback(self.print_host_filesystem)
+
 
     def setup_failed(self, arg):
         # Note that actions based on this failure are deferred to callers:
@@ -195,7 +221,8 @@ def onion_listen(self, onion):
         serverstring = "tcp:{}:interface={}".format(self.serving_port,
                                                     self.serving_host)
         onion_endpoint = serverFromString(reactor, serverstring)
-        return onion_endpoint.listen(self.site)
+        print("created the onion endpoint, now calling listen")
+        return onion_endpoint.listen(self.proto_factory)
 
     def print_host(self, ep):
         """ Callback fired once the HS is available
@@ -204,6 +231,14 @@ def print_host(self, ep):
         """
         self.onion_hostname_callback(self.onion.hostname)
 
+    def print_host_filesystem(self, port):
+        """ As above but needed to respect slightly different
+        callback chain for this case (where we start our own tor
+        instance for the filesystem-based onion).
+        """
+        self.onion = port.onion_service
+        self.onion_hostname_callback(self.onion.hostname)
+
     def shutdown(self):
         self.tor_connection.protocol.transport.loseConnection()
         self.info_callback("Hidden service shutdown complete")

jmclient/jmclient/__init__.py

@@ -24,7 +24,7 @@
                            TYPE_P2PKH, TYPE_P2SH_P2WPKH, TYPE_P2WPKH, detect_script_type)
 from .configure import (load_test_config, process_shutdown,
     load_program_config, jm_single, get_network, update_persist_config,
-    validate_address, is_burn_destination, get_irc_mchannels,
+    validate_address, is_burn_destination, get_mchannels,
     get_blockchain_interface_instance, set_config, is_segwit_mode,
     is_native_segwit_mode, JMPluginService, get_interest_rate, get_bondless_makers_allowance)
 from .blockchaininterface import (BlockchainInterface,

jmclient/jmclient/client_protocol.py

@@ -15,7 +15,7 @@
 import sys
 from jmbase import (get_log, EXIT_FAILURE, hextobin, bintohex,
                     utxo_to_utxostr, bdict_sdict_convert)
-from jmclient import (jm_single, get_irc_mchannels,
+from jmclient import (jm_single, get_mchannels,
                       RegtestBitcoinCoreInterface,
                       SNICKERReceiver, process_shutdown)
 import jmbitcoin as btc
@@ -434,7 +434,7 @@ def clientStart(self):
                                                    "blockchain_source")
         #needed only for channel naming convention
         network = jm_single().config.get("BLOCKCHAIN", "network")
-        irc_configs = get_irc_mchannels()
+        irc_configs = self.factory.get_mchannels()
         #only here because Init message uses this field; not used by makers TODO
         minmakers = jm_single().config.getint("POLICY", "minimum_makers")
         maker_timeout_sec = jm_single().maker_timeout_sec
@@ -601,7 +601,7 @@ def clientStart(self):
                                                    "blockchain_source")
         #needed only for channel naming convention
         network = jm_single().config.get("BLOCKCHAIN", "network")
-        irc_configs = get_irc_mchannels()
+        irc_configs = self.factory.get_mchannels()
         minmakers = jm_single().config.getint("POLICY", "minimum_makers")
         maker_timeout_sec = jm_single().maker_timeout_sec
 
@@ -795,6 +795,14 @@ def getClient(self):
     def buildProtocol(self, addr):
         return self.protocol(self, self.client)
 
+    def get_mchannels(self):
+        """ A transparent wrapper that allows override,
+        so that a script can return a customised set of
+        message channel configs; currently used for testing
+        multiple bots on regtest.
+        """
+        return get_mchannels()
+
 def start_reactor(host, port, factory=None, snickerfactory=None,
                   bip78=False, jm_coinjoin=True, ish=True,
                   daemon=False, rs=True, gui=False): #pragma: no cover