Merge #996: Rpc api 2

7e73e4c Add websocket for subscription, OpenAPI spec (Adam Gibson) 1688d2d Adds listutxos and heartbeat route, several fixes (abhishek0405) 80e17df Add jmwalletd script as RPC server. (Adam Gibson)
JoinMarket-Org · Oct 10, 2021 · ff10262 · ff10262
2 parents 658952e + 7e73e4c
commit ff10262
Show file tree

Hide file tree

Showing 19 changed files with 3,247 additions and 33 deletions.
diff --git a/docs/JSON-RPC-API-using-jmwalletd.md b/docs/JSON-RPC-API-using-jmwalletd.md
@@ -0,0 +1,121 @@
+## JSON-RPC API for Joinmarket using jmwalletd.py
+
+### Introduction - how to start the server
+
+Create an ssl certificate and store it in `<datadir>/ssl/{key,cert}.pem`; the `datadir` is set by `--datadir` in scripts or is `~/.joinmarket` by default, or `.` by default in testing.
+
+After installing Joinmarket as per the [INSTALL GUIDE](INSTALL.md), navigate to the `scripts/` directory as usual and start the server with:
+
+```
+(jmvenv) $python jmwalletd.py
+```
+
+which with defaults will start serving the RPC over `https://` on port 28183, and a (secure) websocket server (`wss://`) on port 28283.
+
+Documentation of the websocket functionality [below](#websocket).
+
+This HTTP server does *NOT* currently support multiple sessions; it is intended as a manager/daemon for all the Joinmarket services for a single user. Note that in particular it allows only control of *one wallet at a time*.
+
+#### Rules about making requests
+
+Authentication is with the [JSON Web Token](https://jwt.io/) scheme, provided using the Python package [PyJWT](https://pypi.org/project/PyJWT/).
+
+Note that for some methods, it's particularly important to deal with the HTTP response asynchronously, since it can take some time for wallet synchronization, service startup etc. to occur; in these cases a HTTP return code of 202 is sent.
+
+### API documentation
+
+Current API version: v1.
+
+The [OpenAPI](https://github.com/OAI/OpenAPI-Specification) spec is given in [this yaml file](../jmclient/jmclient/wallet-rpc-api.yaml). Human readable documentation of the API is provided in [this document](../jmclient/jmclient/wallet-rpc-api.md), which is auto-generated with the node utility [swagger-markdown](https://www.npmjs.com/package/swagger-markdown).
+
+Those wishing to write client code should adhere to that specification.
+
+#### What is and is not provided in the current version of the API.
+
+As a brief summary, the functionality currently available is:
+
+* list existing wallets
+* create a wallet
+* unlock (decrypt) a wallet
+* lock a wallet
+* display contents of a wallet
+* list the utxos in the wallet
+* get a new address for deposit in a given account
+* send a payment without coinjoin
+* send a payment with coinjoin
+* start the yield generator
+* stop the yield generator
+* get the value of a specific config variable
+* set the value of a specific config variable (only in memory)
+* a 'heartbeat' check that also reports whether a wallet is loaded, whether the maker is running, whether a coinjoin is in process.
+
+Clearly there are several further functionalities currently available in the CLI and Qt versions of Joinmarket which are not yet supported. It is likely that several or all of these will be added in future (e.g.: payjoin, utxo freezing).
+
+In addition to the above, a websocket service currently allowing subscription only to transaction events, and coinjoining state, is provided, see next.
+
+<a name="websocket" />
+
+### Websocket
+
+When a wallet service is started via a call to `create` or `unlock` (see above), the websocket automatically starts to serve notifications to any connected client. The client must send the authentication token it has received in the create/unlock call, over the websocket, when it connects, otherwise it will not receive any notifications.
+
+Any authenticated connection is currently automatically subscribed to both of the following events:
+
+#### Coinjoin state change event
+
+When the backend switches from doing nothing, to running a coinjoin as taker over the messaging channels, or to running as a yield generator, or stopping either of these, an event is sent on the websocket noting the new current state. The message is json encoded as:
+
+```
+{"coinjoin_state": 1}
+```
+
+where the values are:
+
+0 - Taker running  
+1 - Maker running  
+2 - Neither are running
+
+#### Transaction event
+
+When a transaction is seen for the first time in the Joinmarket wallet, a notification is sent to the client over the websocket as encoded json, containing the txid and a detailed human-readable deserialization of the transaction details. See this example:
+
+```
+{"txid": "ca606efc5ba8f6669ba15e9262e5d38e745345ea96106d5a919688d1ff0da0cc", 
+ "txdetails": {
+    "hex": "02000000000102578770b2732aed421ffe62d54fd695cf281ca336e4f686d2adbb2e8c3bedb2570000000000ffffffff4719a259786b4237f92460629181edcc3424419592529103143090f07d85ec330100000000ffffffff0324fd9b0100000000160014d38fa4a6ac8db7495e5e2b5d219dccd412dd9bae24fd9b0100000000160014564aead56de8f4d445fc5b74a61793b5c8a819667af6c208000000001600146ec55c2e1d1a7a868b5ec91822bf40bba842bac502473044022078f8106a5645cc4afeef36d4addec391a5b058cc51053b42c89fcedf92f4db1002200cdf1b66a922863fba8dc1b1b1a0dce043d952fa14dcbe86c427fda25e930a53012102f1f750bfb73dbe4c7faec2c9c301ad0e02176cd47bcc909ff0a117e95b2aad7b02483045022100b9a6c2295a1b0f7605381d416f6ed8da763bd7c20f2402dd36b62dd9dd07375002207d40eaff4fc6ee219a7498abfab6bdc54b7ce006ac4b978b64bff960fbf5f31e012103c2a7d6e44acdbd503c578ec7d1741a44864780be0186e555e853eee86e06f11f00000000",
+    "inputs": [
+        {
+            "outpoint": "57b2ed3b8c2ebbadd286f6e436a31c28cf95d64fd562fe1f42ed2a73b2708757:0",
+            "scriptSig": "",
+            "nSequence": 4294967295,
+            "witness": "02473044022078f8106a5645cc4afeef36d4addec391a5b058cc51053b42c89fcedf92f4db1002200cdf1b66a922863fba8dc1b1b1a0dce043d952fa14dcbe86c427fda25e930a53012102f1f750bfb73dbe4c7faec2c9c301ad0e02176cd47bcc909ff0a117e95b2aad7b"
+        },
+        {
+            "outpoint": "33ec857df09030140391529295412434cced8191626024f937426b7859a21947:1",
+            "scriptSig": "",
+            "nSequence": 4294967295,
+            "witness": "02483045022100b9a6c2295a1b0f7605381d416f6ed8da763bd7c20f2402dd36b62dd9dd07375002207d40eaff4fc6ee219a7498abfab6bdc54b7ce006ac4b978b64bff960fbf5f31e012103c2a7d6e44acdbd503c578ec7d1741a44864780be0186e555e853eee86e06f11f"
+        }
+    ],
+    "outputs": [
+        {
+            "value_sats": 27000100,
+            "scriptPubKey": "0014d38fa4a6ac8db7495e5e2b5d219dccd412dd9bae",
+            "address": "bcrt1q6w86ff4v3km5jhj79dwjr8wv6sfdmxawzzx47z"
+        },
+        {
+            "value_sats": 27000100,
+            "scriptPubKey": "0014564aead56de8f4d445fc5b74a61793b5c8a81966",
+            "address": "bcrt1q2e9w44tdar6dg30utd62v9unkhy2sxtxr0p4md"
+        },
+        {
+            "value_sats": 146994810,
+            "scriptPubKey": "00146ec55c2e1d1a7a868b5ec91822bf40bba842bac5",
+            "address": "bcrt1qdmz4ctsarfagdz67eyvz906qhw5y9wk990rz48"
+        }
+    ],
+    "txid": "ca606efc5ba8f6669ba15e9262e5d38e745345ea96106d5a919688d1ff0da0cc",
+    "nLockTime": 0,
+    "nVersion": 2
+}}
+ ```
diff --git a/jmbase/jmbase/commands.py b/jmbase/jmbase/commands.py
@@ -70,6 +70,13 @@ class JMMsgSignatureVerify(JMCommand):
                  (b'fullmsg', Unicode()),
                  (b'hostid', Unicode())]
 
+class JMShutdown(JMCommand):
+    """ Requests shutdown of the current
+    message channel connections (to be used
+    when the client is shutting down).
+    """
+    arguments = []
+
 """TAKER specific commands
 """
 

diff --git a/jmclient/jmclient/__init__.py b/jmclient/jmclient/__init__.py
@@ -31,7 +31,8 @@
 from .snicker_receiver import SNICKERError, SNICKERReceiver
 from .client_protocol import (JMTakerClientProtocol, JMClientProtocolFactory,
                               start_reactor, SNICKERClientProtocolFactory,
-                              BIP78ClientProtocolFactory)
+                              BIP78ClientProtocolFactory,
+                              get_daemon_serving_params)
 from .podle import (set_commitment_file, get_commitment_file,
                     add_external_commitments,
                     PoDLE, generate_podle, get_podle_commitments,
@@ -58,9 +59,14 @@
     wallet_change_passphrase)
 from .wallet_service import WalletService
 from .maker import Maker
-from .yieldgenerator import YieldGenerator, YieldGeneratorBasic, ygmain
+from .yieldgenerator import YieldGenerator, YieldGeneratorBasic, ygmain, \
+     YieldGeneratorService, YieldGeneratorServiceSetupFailed
+from .snicker_receiver import SNICKERError, SNICKERReceiver, SNICKERReceiverService
 from .payjoin import (parse_payjoin_setup, send_payjoin,
                       JMBIP78ReceiverManager)
+from .websocketserver import JmwalletdWebSocketServerFactory, \
+     JmwalletdWebSocketServerProtocol
+from .wallet_rpc import JMWalletDaemon
 # Set default logging handler to avoid "No handler found" warnings.
 
 try:

diff --git a/jmclient/jmclient/client_protocol.py b/jmclient/jmclient/client_protocol.py
@@ -20,6 +20,15 @@
                       SNICKERReceiver, process_shutdown)
 import jmbitcoin as btc
 
+# module level variable representing the port
+# on which the daemon is running.
+# note that this var is only set if we are running
+# client+daemon in one process.
+daemon_serving_port = -1
+daemon_serving_host = ""
+
+def get_daemon_serving_params():
+    return (daemon_serving_host, daemon_serving_port)
 
 jlog = get_log()
 
@@ -366,6 +375,15 @@ def make_tx(self, nick_list, tx):
                             tx=tx)
         self.defaultCallbacks(d)
 
+    def request_mc_shutdown(self):
+        """ To ensure that lingering message channel
+        connections are shut down when the client itself
+        is shutting down.
+        """
+        d = self.callRemote(commands.JMShutdown)
+        self.defaultCallbacks(d)
+        return {'accepted': True}
+
 class JMMakerClientProtocol(JMClientProtocol):
     def __init__(self, factory, maker, nick_priv=None):
         self.factory = factory
@@ -779,9 +797,18 @@ def start_reactor(host, port, factory=None, snickerfactory=None,
     #(Cannot start the reactor in tests)
     #Not used in prod (twisted logging):
     #startLogging(stdout)
-    usessl = True if jm_single().config.get("DAEMON",
-                                            "use_ssl") != 'false' else False
-
+    global daemon_serving_host
+    global daemon_serving_port
+
+    # in case we are starting connections but not the
+    # reactor, we can return a handle to the connections so
+    # that they can be cleaned up properly.
+    # TODO: currently *only* used in tests, with only one
+    # server protocol listening.
+    serverconn = None
+    clientconn = None
+
+    usessl = jm_single().config.get("DAEMON", "use_ssl") != 'false'
     jmcport, snickerport, bip78port = [port]*3
     if daemon:
         try:
@@ -806,7 +833,7 @@ def start_daemon_on_port(p, f, name, port_offset):
             orgp = p[0]
             while True:
                 try:
-                    start_daemon(host, p[0] - port_offset, f, usessl,
+                    serverconn = start_daemon(host, p[0] - port_offset, f, usessl,
                         './ssl/key.pem', './ssl/cert.pem')
                     jlog.info("{} daemon listening on port {}".format(
                         name, str(p[0] - port_offset)))
@@ -819,15 +846,19 @@ def start_daemon_on_port(p, f, name, port_offset):
                                    "listen on any of them. Quitting.")
                         sys.exit(EXIT_FAILURE)
                     p[0] += 1
-            return p[0]
+            return (p[0], serverconn)
 
         if jm_coinjoin:
             # TODO either re-apply this port incrementing logic
             # to other protocols, or re-work how the ports work entirely.
-            jmcport = start_daemon_on_port(port_a, dfactory, "Joinmarket", 0)
+            jmcport, serverconn = start_daemon_on_port(port_a, dfactory,
+                                                       "Joinmarket", 0)
+            daemon_serving_port = jmcport
+            daemon_serving_host = host
         # (See above) For now these other two are just on ports that are 1K offsets.
         if snickerfactory:
-            snickerport = start_daemon_on_port(port_a, sdfactory, "SNICKER", 1000) - 1000
+            snickerport, serverconn = start_daemon_on_port(port_a, sdfactory,
+                                                    "SNICKER", 1000) - 1000
         if bip78:
             start_daemon_on_port(port_a, bip78factory, "BIP78", 2000)
 
@@ -845,14 +876,15 @@ def start_daemon_on_port(p, f, name, port_offset):
             reactor.connectSSL(host, jmcport, factory, ClientContextFactory())
         if snickerfactory:
             reactor.connectSSL(host, snickerport, snickerfactory,
-                           ClientContextFactory())
+                               ClientContextFactory())
     else:
         if factory:
-            reactor.connectTCP(host, jmcport, factory)
+            clientconn = reactor.connectTCP(host, jmcport, factory)
         if snickerfactory:
             reactor.connectTCP(host, snickerport, snickerfactory)
     if rs:
         if not gui:
             reactor.run(installSignalHandlers=ish)
         if isinstance(jm_single().bc_interface, RegtestBitcoinCoreInterface):
             jm_single().bc_interface.shutdown_signal = True
+    return (serverconn, clientconn)
diff --git a/jmclient/jmclient/maker.py b/jmclient/jmclient/maker.py
@@ -4,7 +4,7 @@
 import atexit
 
 import jmbitcoin as btc
-from jmbase import bintohex, hexbin, get_log, EXIT_FAILURE, stop_reactor
+from jmbase import bintohex, hexbin, get_log, EXIT_FAILURE
 from jmclient.wallet_service import WalletService
 from jmclient.configure import jm_single
 from jmclient.support import calc_cj_fee
@@ -39,12 +39,18 @@ def try_to_create_my_orders(self):
         if not self.wallet_service.synced:
             return
         self.freeze_timelocked_utxos()
-        self.offerlist = self.create_my_orders()
+        try:
+            self.offerlist = self.create_my_orders()
+        except AssertionError:
+            jlog.error("Failed to create offers.")
+            self.aborted = True
+            return
         self.fidelity_bond = self.get_fidelity_bond_template()
         self.sync_wait_loop.stop()
         if not self.offerlist:
-            jlog.info("Failed to create offers, giving up.")
-            stop_reactor()
+            jlog.error("Failed to create offers.")
+            self.aborted = True
+            return
         jlog.info('offerlist={}'.format(self.offerlist))
 
     @hexbin

diff --git a/jmclient/jmclient/snicker_receiver.py b/jmclient/jmclient/snicker_receiver.py
@@ -1,17 +1,55 @@
 #! /usr/bin/env python
 
+import os
+from twisted.application.service import Service
+from twisted.internet import task
 import jmbitcoin as btc
 from jmclient.configure import jm_single
 from jmbase import (get_log, utxo_to_utxostr,
                     hextobin, bintohex)
-from twisted.application.service import Service
 
 jlog = get_log()
 
 class SNICKERError(Exception):
     pass
 
-class SNICKERReceiver(Service):
+class SNICKERReceiverService(Service):
+    def __init__(self, receiver):
+        assert isinstance(receiver, SNICKERReceiver)
+        self.receiver = receiver
+        # main monitor loop
+        self.monitor_loop = task.LoopingCall(self.receiver.poll_for_proposals)
+
+    def startService(self):
+        """ Encapsulates start up actions.
+        This service depends on the receiver's
+        wallet service to start, so wait for that.
+        """
+        self.wait_for_wallet = task.LoopingCall(self.wait_for_wallet_sync)
+        self.wait_for_wallet.start(5.0)
+
+    def wait_for_wallet_sync(self):
+        if self.receiver.wallet_service.isRunning():
+            jlog.info("SNICKER service starting because wallet service is up.")
+            self.wait_for_wallet.stop()
+            self.monitor_loop.start(5.0)
+            super().startService()
+
+    def stopService(self, wallet=False):
+        """ Encapsulates shut down actions.
+        Optionally also shut down the underlying
+        wallet service (default False).
+        """
+        if self.monitor_loop:
+            self.monitor_loop.stop()
+        if wallet:
+            self.receiver.wallet_service.stopService()
+        super().stopService()
+
+    def isRunning(self):
+        return self.running == 1
+
+class SNICKERReceiver(object):
     supported_flags = []
 
     def __init__(self, wallet_service, acceptance_callback=None,
@@ -66,6 +104,10 @@ def __init__(self, wallet_service, acceptance_callback=None,
 
     def default_info_callback(self, msg):
         jlog.info(msg)
+        if not os.path.exists(self.proposals_source):
+            with open(self.proposals_source, "wb") as f:
+                jlog.info("created proposals source file.")
+
 
     def default_acceptance_callback(self, our_ins, their_ins,
                                     our_outs, their_outs):