Update guide for pcli threshold

docs/guide/book.toml

@@ -15,14 +15,14 @@ command = "mdbook-mermaid"
 default-theme = "coal"
 preferred-dark-theme = "coal"
 curly-quotes = true
-fold = { enable = true, level = 1 }
+fold = { enable = true, level = 2 }
 git-repository-url = "https://github.com/penumbra-zone/penumbra"
 additional-js =["theme/js/mermaid.min.js", "theme/js/mermaid-init.js"]
 
 [output.linkcheck]
 # Consider setting `follow-web-links=true` ad-hoc, to validate external URLs.
 follow-web-links = false
-warning-policy = "ignore"
+warning-policy = "error"
 
 # The buf.build website doesn't support HTTP HEAD calls, returning 405.
 # Only relevant when `follow-web-links=true`.

docs/guide/src/README.md

@@ -1,99 +1,68 @@
 # Penumbra Guide
 
-[Penumbra] is a fully shielded zone for the Cosmos ecosystem, allowing anyone to
-securely transact, stake, swap, or marketmake without broadcasting their
-personal information to the world.
-
-This site contains documentation on how to use, deploy, and develop the Penumbra
-software.  The description of the protocol itself can be found in the [protocol
-specification][protocol], and the API documentation can be found
-[here][rustdoc].
-
-## Test networks
-
-Penumbra is a decentralized protocol, so Penumbra Labs is [building in
-public][how-were-building], launching (and crashing) lots of work-in-progress
-testnets to allow community participation, engagement, and feedback.
-
-Currently, Penumbra only has a command line client, `pcli` (pronounced
-"pickle-y"), which bundles all of the client components in one binary, and a
-chain-scanning daemon, `pclientd`, which runs just the view service, without spend
-capability.  To get started with the Penumbra test network, all that's required
-is to download and build `pcli`, as described in
-[Installation](./pcli/install.md).
-
-The Penumbra node software is the Penumbra daemon, `pd`.  This is an ABCI
-application, which must be driven by CometBFT, so a Penumbra full node
-consists of both a `pd` instance and a `cometbft` instance.
-
-The basic architecture of Penumbra is as follows:
-
-```text
-          ╭   ┌───────┐
-  spending│   │custody│
-capability│   │service│
-          ╰   └───────┘
-               ▲     │
-               │tx   │auth
-               │plan │data
-               │     ▼
-          ╭   ┌───────┐
-   viewing│   │wallet │ tx submission
-capability│   │logic  │────────┐
-          │   └───────┘        │
-          │    ▲               │
-          │    │view private state
-          │    │               │
-          │    │               │
-          │   ┌───────┐        │
-          │   │view   │        │
-          │   │service│        │
-          ╰   └───────┘        │
-               ▲               │
-               │sync private state
-               │               │
-          ╭ ┌──┼───────────────┼──────┐
-    public│ │  │     Penumbra Fullnode│
-     chain│ │  │               │      │
-      data│ │  │               ▼      │
-          │ │ ┌──┐ app   ┌──────────┐ │
-          │ │ │pd│◀─────▶│ cometbft │ │
-          │ │ └──┘ sync  └──────────┘ │
-          │ │               ▲         │
-          ╰ └───────────────┼─────────┘
-                         .──│.
-                       ,'   │ `.
-                  .───;     │consensus
-                 ;          │sync
-               .─┤          │   ├──.
-             ,'             │       `.
-            ;   Penumbra    │         :
-            :   Network  ◀──┘         ;
-             ╲                       ╱
-              `.     `.     `.     ,'
-                `───'  `───'  `───'
-```
-
-The custody service holds signing keys and is responsible for authorizing
-transaction plans.  The view service holds viewing keys and scans the chain
-state.  Wallet logic can query the view service to get information about what
-funds are available, submit a transaction plan to the custody service for
-signing, and then use the returned signatures to build the transaction and
-submit it.
-
-As a shielded chain, Penumbra's architecture is slightly different than a
-transparent chain, because user data such as account balances, transaction
-activity, etc., is not part of the public chain state.  This means that clients
-need to synchronize with the chain to build a copy of the private user data they
-have access to.  This logic is provided by the *view service*, which is bundled
-into `pcli`, but can also be run as a standalone `pclientd` daemon.
-
-Modeling authorization as an (asynchronous) RPC to a custody service means that
-the client software is compatible with many different custody flows by default
--- an in-process "SoftHSM", a hardware wallet with user intervention, a cluster
-of online threshold signers, an offline threshold signing process, etc.
-
-[how-were-building]: https://penumbra.zone/blog/how-were-building-penumbra
+[Penumbra] is a fully private, cross-chain proof-of-stake network and
+decentralized exchange for the Cosmos and beyond. Penumbra brings privacy to
+IBC, allowing anyone to shield any IBC asset just by transferring it into
+Penumbra's multi-asset shielded pool.  Within Penumbra, users can transact,
+stake, swap, or marketmake without broadcasting their personal information to
+the world.
+
+<picture>
+  <img src="./assets/interchain-shielded-pool.jpg" />
+</picture>
+
+Unlike a transparent chain, where all information is public, Penumbra is
+end-to-end encrypted. 
+
+## Using Penumbra on the web
+
+The easiest way to get started using Penumbra is with the Penumbra web
+extension.  The [web extension][webext] runs entirely locally, and contains an
+embedded ultralight node that syncs and decrypts only the data visible to your
+wallet.  Websites can request to connect to your wallet and query your data.
+
+The [_Using Penumbra on the web_](./web.md) chapter describes how to use
+Penumbra on the web.
+
+## Using Penumbra from the command line
+
+Penumbra also has a command-line client, `pcli`.  Some protocol features, such
+as threshold custody for shielded multisigs, do not yet have support in web
+frontends and are only accessible via the command line.
+
+The [_Using Penumbra from the command line_](./pcli.md) chapter describes how to
+use `pcli`.
+
+## Running a node
+
+Running a node is not necessary to use the protocol. Both the web extension and
+`pcli` are designed to operate with any RPC endpoint. However, we've tried to
+make it as easy as possible to run nodes so that users can host their own RPC.
+
+There are two kinds of Penumbra nodes:
+
+* Penumbra fullnodes run `pd` and `cometbft` to synchronize and verify the entire chain state, as described in [_Running a node: `pd`_](./node/pd.md).
+* Penumbra ultralight nodes run `pclientd` to scan, decrypt, and synchronize a specific wallet's data, as well as build and sign transactions, as described in [_Running a node: `pclientd`_](./node/pclientd.md).
+
+The web extension and `pcli` embed the view and custody functionality provided
+by `pclientd`, so it is not necessary to run `pclientd` to use them. Instead,
+`pclientd` is intended to act as a local RPC for programmatic tooling (e.g.,
+trading bots) not written in Rust that cannot easily embed the code for working
+with Penumbra's shielded cryptography.
+
+## Participating in development
+
+Penumbra is a decentralized, open-source protocol, built in public.
+
+The [_Participating in development_](./dev.md) chapter has developer
+documentation for working on the protocol itself.
+
+## Resources
+
+The [_Resources_](./resources.md) chapter has links to other resources about the Penumbra protocol.
+
+
 [protocol]: https://protocol.penumbra.zone
 [rustdoc]: https://rustdoc.penumbra.zone
 [Penumbra]: https://github.com/penumbra-zone/penumbra
+[webext]: https://chrome.google.com/webstore/detail/penumbra-wallet/lkpmkhpnhknhmibgnmmhdhgdilepfghe

docs/guide/src/SUMMARY.md

@@ -1,23 +1,26 @@
 [Penumbra Guide](./README.md)
-- [Using the web wallet](./extension.md)
-- [Using `pcli`](./pcli.md)
+- [Using Penumbra on the web](./web.md)
+- [Using Penumbra from the command line](./pcli.md)
   - [Installing `pcli`](./pcli/install.md)
   - [Generating a Wallet](./pcli/wallet.md)
+    - [Software Custody Backend](./pcli/wallet/softkms.md)
+    - [Threshold Custody Backend](./pcli/wallet/threshold.md)
   - [Updating to the latest testnet](./pcli/update.md)
   - [Viewing Balances](./pcli/balance.md)
   - [Sending Transactions](./pcli/transaction.md)
   - [Using Governance](./pcli/governance.md)
-- [Running a `pd` fullnode](./pd.md)
-  - [Requirements](./pd/requirements.md)
-  - [Installing `pd`](./pd/install.md)
-  - [Joining a testnet](./pd/join-testnet.md)
-  - [Becoming a validator](./pd/validator.md)
-  - [Performing a chain upgrade](./pd/chain-upgrade.md)
-  - [Debugging](./pd/debugging.md)
-- [Local RPC with `pclientd`](./pclientd.md)
-  - [Configuring `pclientd`](./pclientd/configure.md)
-  - [Making RPC requests](./pclientd/rpc.md)
-  - [Building Transactions](./pclientd/build_transaction.md)
+- [Running a node](./node.md)
+  - [Full node: `pd`](./node/pd.md)
+    - [Requirements](./node/pd/requirements.md)
+    - [Installing `pd`](./node/pd/install.md)
+    - [Joining a testnet](./node/pd/join-testnet.md)
+    - [Becoming a validator](./node/pd/validator.md)
+    - [Performing a chain upgrade](./node/pd/chain-upgrade.md)
+    - [Debugging](./node/pd/debugging.md)
+  - [Ultralight node: `pclientd`](./node/pclientd.md)
+    - [Configuring `pclientd`](./node/pclientd/configure.md)
+    - [Making RPC requests](./node/pclientd/rpc.md)
+    - [Building Transactions](./node/pclientd/build_transaction.md)
 - [Development](./dev.md)
   - [Compiling from source](./dev/build.md)
   - [Devnet Quickstart](./dev/devnet-quickstart.md)

docs/guide/src/dev/devnet-quickstart.md

@@ -3,7 +3,7 @@
 This page describes a quickstart method for running `pd`+`cometbft` to test
 changes during development.
 
-To start, you'll need to install a [specific version of CometBFT](../pd/install.md#installing-cometbft).
+To start, you'll need to install a [specific version of CometBFT](../node/pd/install.md#installing-cometbft).
 
 ## Generating configs
 

docs/guide/src/dev/metrics.md

@@ -6,13 +6,12 @@ the Penumbra software is doing. Penumbra Labs runs Grafana instances for the pub
   * https://grafana.testnet.penumbra.zone
   * https://grafana.testnet-preview.penumbra.zone
 
-## Viewing Metrics
+There's a more comprehensive WIP dashboard, gated by basic auth for PL team:
 
-TODO: add details on how to use Grafana:
+  * https://metrics.penumbra.zone
 
-- [x] link to <https://grafana.testnet.penumbra.zone> for dashboard on current testnet;
-- [ ] instructions on how to run Grafana + Prometheus for local dev setup (ideally this could work without requiring that `pd` itself is Dockerized, since local development is often more convenient outside of docker);
-- [x] instructions on how to commit dashboards back to the repo.
+Check the usual place for credentials. Eventually those views should be exported
+as public references.
 
 ## Adding Metrics
 

docs/guide/src/dev/rpc.md

@@ -1,7 +1,7 @@
 # Working with gRPC for Penumbra
 
-The Penumbra [`pd`](../pd.md) application exposes a [gRPC] service for integration
-with other tools, such as [`pcli`](../pcli.md) or the [web extension](../extension.md).
+The Penumbra [`pd`](../node/pd.md) application exposes a [gRPC] service for integration
+with other tools, such as [`pcli`](../pcli.md) or the [web extension](../web.md).
 A solid understanding of how the gRPC methods work is helpful when
 building software that interoperates with Penumbra.
 
@@ -37,11 +37,11 @@ Click **Send** to submit the request and view response data in the right-hand pa
 
 Regardless of which interface you choose, you can connect to an instance of `pd` running
 on your machine, which can be useful while adding new features.
-First, make sure you've [joined a testnet](../pd/join-testnet.md)
+First, make sure you've [joined a testnet](../node/pd/join-testnet.md)
 by setting up a node on your local machine. Once it's running, you can connect directly
 to the pd port via `http://localhost:8080`.
 
-Alternatively, you can use `pclientd`. First, make sure you've [configured pclientd locally](../pclientd/configure.md)
+Alternatively, you can use `pclientd`. First, make sure you've [configured pclientd locally](../node/pclientd/configure.md)
 with your full viewing key. Once it's running, you can connect directly
 to the pclient port via `http://localhost:8081`.
 

docs/guide/src/node.md

@@ -0,0 +1,16 @@
+# Running a node
+
+Running a node is not necessary to use the protocol. Both the web extension and
+`pcli` are designed to operate with any RPC endpoint. However, we've tried to
+make it as easy as possible to run nodes so that users can host their own RPC.
+
+There are two kinds of Penumbra nodes:
+
+* Penumbra fullnodes run `pd` and `cometbft` to synchronize and verify the entire chain state, as described in [_Running a node: `pd`_](./node/pd.md).
+* Penumbra ultralight nodes run `pclientd` to scan, decrypt, and synchronize a specific wallet's data, as well as build and sign transactions, as described in [_Running a node: `pclientd`_](./node/pclientd.md).  
+
+The web extension and `pcli` embed the view and custody functionality provided
+by `pclientd`, so it is not necessary to run `pclientd` to use them. Instead,
+`pclientd` is intended to act as a local RPC for programmatic tooling (e.g.,
+trading bots) not written in Rust that cannot easily embed the code for working
+with Penumbra's shielded cryptography.

docs/guide/src/pclientd/configure.md → docs/guide/src/node/pclientd/configure.md

@@ -1,7 +1,7 @@
 # Configuring `pclientd`
 
 First, install `pclientd` following the [instructions for installing
-`pcli`](../pcli/install.md) but downloading `pclientd` rather than `pcli`.
+`pcli`](../../pcli/install.md) but downloading `pclientd` rather than `pcli`.
 
 ## Generating configs
 

docs/guide/src/pd.md → docs/guide/src/node/pd.md

@@ -5,4 +5,4 @@ Penumbra:
 
 - [Installing `pd`](./pd/install.md) describes how to build `pd`;
 - [Joining a Testnet](./pd/join-testnet.md) describes how to join the current testnet;
-- [Creating a Testnet](./dev/devnet-quickstart.md) describes how to create a custom testnet, for instance for local development.
+- [Creating a Testnet](../dev/devnet-quickstart.md) describes how to create a custom testnet, for instance for local development.

docs/guide/src/pd/chain-upgrade.md → docs/guide/src/node/pd/chain-upgrade.md

@@ -40,7 +40,7 @@ at slightly different times.
 
 ### Testnet 70 -> 71
 
-For the first chain upgrade performed on a Penumbra testnet, use this value for genesis time: `{{ #include ../upgrade_genesis_time_70_71.md }}`.
+For the first chain upgrade performed on a Penumbra testnet, use this value for genesis time: `{{ #include ../../upgrade_genesis_time_70_71.md }}`.
 See an example below for how to supply this value when performing the migration.
 
 ## Performing a chain upgrade
@@ -52,7 +52,7 @@ If your instance is using a different directory, update the paths accordingly.
 2. Using the same version of `pd` that was running when the chain halted, prepare an export directory:
    `pd export --home ~/.penumbra/testnet_data/node0/pd --export-directory ~/.penumbra/testnet_data/node0/pd-exported-state`
 3. Back up the historical state directory: `mv ~/.penumbra/testnet_data/node0/pd ~/.penumbra/testnet_data/node0/pd-state-backup`
-4. Download the latest version of `pd` and install it. Run `pd --version` and confirm you see `{{ #include ../penumbra_version.md }}` before proceeding.
+4. Download the latest version of `pd` and install it. Run `pd --version` and confirm you see `{{ #include ../../penumbra_version.md }}` before proceeding.
 
 <!--
 An example log message emitted by `pd migrate` without providing `--genesis-start`:
@@ -62,7 +62,7 @@ An example log message emitted by `pd migrate` without providing `--genesis-star
 The value after `now=` is what should be copied. In practice, for testnets, Penumbra Labs will advise on a genesis time
 and provide that value in the documentation. Or should we just pick a genesis start ahead of time, and use that for all?
 -->
-5. Apply the migration: `pd migrate --genesis-start "{{ #include ../upgrade_genesis_time_70_71.md }}" --target-directory ~/.penumbra/testnet_data/node0/pd-exported-state/ --migrate-archive ~/.penumbra/testnet_data/node0/pd-migrated-state-{{ #include ../penumbra_version.md }}.tar.gz`.
+5. Apply the migration: `pd migrate --genesis-start "{{ #include ../../upgrade_genesis_time_70_71.md }}" --target-directory ~/.penumbra/testnet_data/node0/pd-exported-state/ --migrate-archive ~/.penumbra/testnet_data/node0/pd-migrated-state-{{ #include ../../penumbra_version.md }}.tar.gz`.
    You must use that precise genesis time, otherwise your node will not be able to reach consensus with the rest of the network.
 6. Move the migrated state into place: `mkdir ~/.penumbra/testnet_data/node0/pd && mv ~/.penumbra/testnet_data/node0/pd-exported-state/rocksdb ~/.penumbra/testnet_data/node0/pd/`
 7. Move the upgrade cometbft state into place: `cp ~/.penumbra/testnet_data/node0/pd-exported-state/genesis.json ~/.penumbra/testnet_data/node0/cometbft/config/genesis.json
@@ -73,5 +73,5 @@ Finally, restart the node, e.g. `sudo systemctl restart penumbra cometbft`. Chec
 past the halt height `n`.
 
 If you want to host a snapshot for this migration, copy the file
-`~/.penumbra/testnet_data/node0/pd-migrated-state-{{ #include ../penumbra_version.md }}.tar.gz` to the appropriate hosting environment,
+`~/.penumbra/testnet_data/node0/pd-migrated-state-{{ #include ../../penumbra_version.md }}.tar.gz` to the appropriate hosting environment,
 and inform the users of your validator.

docs/guide/src/pd/install.md → docs/guide/src/node/pd/install.md

@@ -8,7 +8,7 @@ Make sure to choose the correct platform for your machine. After downloading the
 extract it, and copy its contents to your `$PATH`. For example:
 
 ```
-curl -sSfL -O https://github.com/penumbra-zone/penumbra/releases/download/{{ #include ../penumbra_version.md }}/pd-x86_64-unknown-linux-gnu.tar.xz
+curl -sSfL -O https://github.com/penumbra-zone/penumbra/releases/download/{{ #include ../../penumbra_version.md }}/pd-x86_64-unknown-linux-gnu.tar.xz
 unxz pd-x86_64-unknown-linux-gnu.tar.xz
 tar -xf pd-x86_64-unknown-linux-gnu.tar
 sudo mv pd-x86_64-unknown-linux-gnu/pd /usr/local/bin/
@@ -17,16 +17,16 @@ sudo mv pd-x86_64-unknown-linux-gnu/pd /usr/local/bin/
 pd --version
 ```
 As of v0.64.1 (released 2023-12-12), we build Linux binaries on Ubuntu 22.04. If these binaries don't work for you out of the box,
-you'll need to [build from source](../dev/build.md), or use the container images.
+you'll need to [build from source](../../dev/build.md), or use the container images.
 
 ### Installing CometBFT
 
 You'll need to have [CometBFT installed](https://docs.cometbft.com/v0.37/guides/install)
 on your system to join your node to the testnet.
 
-You must use a specific version of CometBFT, `{{ #include ../cometbft_version.md }}`, which you can download
-[from the CometBFT releases page](https://github.com/cometbft/cometbft/releases/tag/{{ #include ../cometbft_version.md }}).
-If you prefer to compile from source instead, make sure you are compiling version `{{ #include ../cometbft_version.md }}`.
+You must use a specific version of CometBFT, `{{ #include ../../cometbft_version.md }}`, which you can download
+[from the CometBFT releases page](https://github.com/cometbft/cometbft/releases/tag/{{ #include ../../cometbft_version.md }}).
+If you prefer to compile from source instead, make sure you are compiling version `{{ #include ../../cometbft_version.md }}`.
 
 Previous versions of Penumbra used Tendermint, but as of Testnet 62 (released 2023-10-10),
 only CometBFT is supported. **Do not use** any version of Tendermint, which will not work with `pd`.

docs/guide/src/pd/join-testnet.md → docs/guide/src/node/pd/join-testnet.md

@@ -102,5 +102,5 @@ sudo systemctl daemon-reload
 sudo systemctl restart penumbra cometbft
 ```
 
-See the [`deployments/`](https://github.com/penumbra-zone/penumbra/tree/{{ #include ../penumbra_version.md }}/deployments)
+See the [`deployments/`](https://github.com/penumbra-zone/penumbra/tree/{{ #include ../../penumbra_version.md }}/deployments)
 directory for more examples on configuration scripts.