Esplora - Electrs backend API

A block chain index engine and HTTP API written in Rust based on romanz/electrs.

Used as the backend for the Esplora block explorer powering blockstream.info.

API documentation is available here.

Documentation for the database schema and indexing process is available here.

Uses rust-ferrite master branch.

Installing & indexing - Ubuntu Linux 22

Install Rust, Ferrite Core (no txindex needed) and the clang and cmake packages, then:

# to install rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# if you dont have curl
sudo apt-get install curl  # do not use snap - if you did, use    sudo snap remove curl 

sudo apt install cargo

# if you hadn't already...
sudo apt install clang
sudo apt install cmake

$ git clone https://github.com/ferritecoin/electrs-fec && cd electrs-fec
$ cargo update   # important!
$ cargo run --release --bin electrs -- -vvvv --daemon-dir ~/.ferrite --cookie user:password --daemon-rpc-addr 127.0.0.1:9573 --jsonrpc-import
or alternatively $ cargo run --release --bin electrs -- -vvvv --daemon-dir ~/.ferrite

# Or for liquid:
$ cargo run --features liquid --release --bin electrs -- -vvvv --network liquid --daemon-dir ~/.liquid

Windows 10

Install dependencies

# Install rust and VS
https://static.rust-lang.org/rustup/dist/x86_64-pc-windows-msvc/rustup-init.exe
# Ensure to select both MSVC v143 VS 2022 C++ x86/x64 build tools and Windows 11 SDK (about 5.6GB)

# Install cmake
https://github.com/Kitware/CMake/releases/download/v3.30.2/cmake-3.30.2-windows-x86_64.msi

# Install git
https://github.com/git-for-windows/git/releases/download/v2.46.0.windows.1/Git-2.46.0-64-bit.exe

# Use Elevated Powershell
# Install chocolatey
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

  # Install clang
  # Note that installing clang later than version 14 will fail to build librocksdb-sys
  choco install llvm --version 14.0.6 -y

# Open Powershell in directory
git clone https://github.com/ferritecoin/electrs-fec && cd electrs-fec
cd electrs-fec
cargo update
cargo run --release --bin electrs -- -vvvv --daemon-dir ~/.ferrite --cookie user:password --daemon-rpc-addr 127.0.0.1:9573 --jsonrpc-import

Known issues

Build Errors

No method named emit_rerun_if_env_changed

error[E0599]: no method named `emit_rerun_if_env_changed` found for struct `Build` in the current scope
   --> /home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/blake3-1.5.3/build.rs:104:11
    |
104 |     build.emit_rerun_if_env_changed(false);
    |           ^^^^^^^^^^^^^^^^^^^^^^^^^ method not found in `Build`

For more information about this error, try `rustc --explain E0599`.
error: could not compile `blake3` (build script) due to previous error
warning: build failed, waiting for other jobs to finish...

# Open your Cargo.toml file and ensure the cc dependency is set to a version that includes the emit_rerun_if_env_changed method:
[build-dependencies]
cc = "1.0.72"
# then
$ cargo update
# and retry
$ cargo run --release --bin electrs -- -vvvv --daemon-dir ~/.ferrite --cookie user:password --daemon-rpc-addr 127.0.0.1:9573 --jsonrpc-import

RPC Errors

failed to read cookie

WARN - reconnecting to ferrited: failed to read cookie from "/home/ubuntu/.ferrite/.cookie"

# Assuming your RPC username is "user" and RPC password is "password" without the two enclosing double quotation marks.
# use --cookie user:password

No reply from daemon

WARN - reconnecting to ferrited: no reply from daemon
ThreadRPCServer incorrect password attempt from 127.0.0.1:56148
# from debug.log
# use --daemon-rpc-addr 127.0.0.1:9573

Failed to index blocks

thread 'blkfiles_fetcher' panicked at 'failed to index 1068560 blocks from blk*.dat files'
# use --jsonrpc-import

Server errors

Address in use

INFO - REST server running on 127.0.0.1:3000
thread 'acceptor' panicked at src/util/mod.rs:128:39:
cannot bind: Os { code: 98, kind: AddrInUse, message: "Address already in use" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aborted (core dumped)

# this means that there is already a service running on the local port 3000.
# You can change the port in electrs-fec/src/config.rs (already changed)
Network::Bitcoin => 3011,
Network::Bitcoin => 50011,

See electrs's original documentation for more detailed instructions. Note that our indexes are incompatible with electrs's and has to be created separately.

The indexes require 610GB of storage after running compaction (as of June 2020), but you'll need to have free space of about double that available during the index compaction process. Creating the indexes should take a few hours on a beefy machine with SSD.

To deploy with Docker, follow the instructions here.

Light mode

For personal or low-volume use, you may set --lightmode to reduce disk storage requirements by roughly 50% at the cost of slower and more expensive lookups.

With this option set, raw transactions and metadata associated with blocks will not be kept in rocksdb (the T, X and M indexes), but instead queried from litecoind on demand.

Notable changes from Electrs:

• HTTP REST API in addition to the Electrum JSON-RPC protocol, with extended transaction information (previous outputs, spending transactions, script asm and more).

• Extended indexes and database storage for improved performance under high load:

  • A full transaction store mapping txids to raw transactions is kept in the database under the prefix t.
  • An index of all spendable transaction outputs is kept under the prefix O.
  • An index of all addresses (encoded as string) is kept under the prefix a to enable by-prefix address search.
  • A map of blockhash to txids is kept in the database under the prefix X.
  • Block stats metadata (number of transactions, size and weight) is kept in the database under the prefix M.

  With these new indexes, litecoind is no longer queried to serve user requests and is only polled periodically for new blocks and for syncing the mempool.

• Support for Liquid and other Elements-based networks, including CT, peg-in/out and multi-asset. (requires enabling the liquid feature flag using --features liquid)

CLI options

In addition to electrs's original configuration options, a few new options are also available:

• --http-addr <addr:port> - HTTP server address/port to listen on (default: 127.0.0.1:3000).
• --lightmode - enable light mode (see above)
• --cors <origins> - origins allowed to make cross-site request (optional, defaults to none).
• --address-search - enables the by-prefix address search index.
• --index-unspendables - enables indexing of provably unspendable outputs.
• --utxos-limit <num> - maximum number of utxos to return per address.
• --electrum-txs-limit <num> - maximum number of txs to return per address in the electrum server (does not apply for the http api).
• --electrum-banner <text> - welcome banner text for electrum server.

Additional options with the liquid feature:

• --parent-network <network> - the parent network this chain is pegged to.

Additional options with the electrum-discovery feature:

• --electrum-hosts <json> - a json map of the public hosts where the electrum server is reachable, in the server.features format.
• --electrum-announce - announce the electrum server on the electrum p2p server discovery network.

See $ cargo run --release --bin electrs -- --help for the full list of options.

License

MIT