Feat: on-disk RevIndex based on RocksDB

[luizirber]

On-disk RevIndex based on RocksDB, initially implemented in https://github.com/luizirber/2022-06-26-rocksdb-eval
This is the index/core data structure backing https://mastiff.sourmash.bio

There are many changes in the Rust code, so bumping the version to 0.12.0.

This is mostly not exposed thru the FFI yet. Tests from the from the in-memory RevIndex (greyhound) from #1238 were kept working, but it is not well supported (doesn't allow saving/loading from disk, for example), and should be wholly replaced by sourmash::index::revindex::disk_revindex (the on-disk RevIndex) in the future.
It is confusing to have these different RevIndex impls in Rust, and I started converging them, but the work is not completely done yet.

#2727 is a better starting point for how Index abc/trait should work acrosss Python/Rust, and I started moving the Rust indices to start from a LinearIndex and later specialize into a RevIndex, which will make easier to translate the work from #2727 for future indices across FFI.

A couple of new concepts introduced in this PR:

• a Collection is a Manifest + Storage. So a zip file like the ones for GTDB databases fit this easily (storage = ZipStorage, manifest is read from the zipfile), but a file paths list does too (manifest built from the file paths, storage = FSStorage). This goes in a bit of different direction than thoughts on Storage, manifests, etc. #1901, which was extending Storage to support more functionality. I think Storage should stay pretty bare and mostly deal with loading/saving data, but not much knowledge of what data is there (this is covered with Manifest).

• a CollectionSet is a consistent collection of signatures. Consistent here means: same k-size, downsample-compatible for scaled, same moltype. You can create a CollectionSet by running .select() on a Collection. CollectionSet is required for building indices (because we shouldn't be building indices mixing k-size/moltype), and greatly simplifies the logic in many places because it is not necessary to check for compatibility.

• LinearIndex was rewritten based on Collection (and also the greyhound/branchwater parallelism), and this supports the "parallel search without an index" use case. There is no index construction per se here, pretty much just a thin layer on top of Collection implementing functionality expected from indices.

• Manifest, Selection, and Picklist are still incomplete, but the relevant function definitions are in place, need to barrage it with tests (and potentially exposing to Python and reusing the ones there in Feature: implement FFI for manifest, picklist and selection in Rust #2726)

Feature

• Initial implementation for Manifest, Selection, and Picklist following
  the Python API.
• Collection is a new abstraction for working with a set of signatures. A
  collection needs a Storage for holding the signatures (on-disk, in-memory,
  or remotely), and a Manifest to describe the metadata for each signature.
• Expose CSV parsing and RocksDB errors.
• New module sourmash::index::revindex::disk_revindex with the on-disk
  RevIndex implementation based on RocksDB.
• Add iter and iter_mut methods for Signature.
• Add load_sig and save_sig methods to Storage trait for higher-level data
  manipulation and caching.
• Add spec method to Storage to allow constructing a concrete Storage from
  a string description.
• Add InnerStorage for synchronizing parallel access to Storage
  implementations.
• Add MemStorage for keeping signatures in-memory (mostly for debugging and
  testing).

Refactor

• Rename HashFunctions variants to follow camel-case, so Murmur64Protein instead of murmur64_protein
• LinearIndex is now implemented as a thin layer on top of Collection.
• Move GatherResult to sourmash::index module.
• Move sourmash::index::revindex to sourmash::index::mem_revindex (this is
  the Greyhound version of revindex, in-memory only). It was also refactored
  internally to build a version of a LinearIndex that will be merged in the
  future with sourmash::index::LinearIndex
• Move select method from Index trait into a separate Select trait,
  and implement it for Signature based on the new Selection API.
• Move SigStore into sourmash::storage module, and remove the generic. Now
  it always stores Signature. Also implement Select for it.

Build

• Add new branchwater feature (enabled by default), which can be disabled by downstream projects to limit bringing heavy dependencies like rocksdb
• Add new rkyv feature (disabled by default), making MinHash serializable
  with the rkyv crate.
• Add semver checks for CI (so we bump versions accordingly, or avoid breaking changes)
• Reduce features combinations on Rust checks (takes much less time to run)
• Disable musllinux wheels (need to figure out how to build rocksdb for it)

[codecov]

Codecov Report

Attention: 318 lines in your changes are missing coverage. Please review.

Comparison is base (d6994a1) 86.23% compared to head (a748bb3) 86.30%.

Files	Patch %	Lines
src/core/src/storage.rs	51.63%	59 Missing ⚠️
src/core/src/index/revindex/mod.rs	50.84%	58 Missing ⚠️
src/core/src/index/revindex/mem_revindex.rs	48.91%	47 Missing ⚠️
src/core/src/index/linear.rs	65.16%	31 Missing ⚠️
src/core/src/manifest.rs	62.06%	22 Missing ⚠️
src/core/src/signature.rs	29.03%	22 Missing ⚠️
src/core/src/index/revindex/disk_revindex.rs	89.26%	19 Missing ⚠️
src/core/src/selection.rs	14.28%	18 Missing ⚠️
src/core/src/collection.rs	67.92%	17 Missing ⚠️
src/core/src/ffi/mod.rs	0.00%	8 Missing ⚠️
... and 7 more

Additional details and impacted files

@@            Coverage Diff             @@
##           latest    #2230      +/-   ##
==========================================
+ Coverage   86.23%   86.30%   +0.07%     
==========================================
  Files         130      135       +5     
  Lines       14756    15301     +545     
  Branches     2622     2622              
==========================================
+ Hits        12725    13206     +481     
- Misses       1731     1795      +64     
  Partials      300      300              

Flag	Coverage Δ
hypothesis-py	25.80% <0.00%> (ø)
python	92.82% <50.00%> (ø)
rust	57.99% <61.57%> (+7.11%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ctb]

@bluegenes can you remind me of the relationship between this branch and the pyo3_dependency on commit ff1092f (which may no longer exist in this repo...?)

@luizirber my main request is that you update the description of the PR so that we can refer back to this in the future. Specific thoughts for things to add (or maybe questions) -

• this bumps the sourmash-rs package to 0.12, right?
• am I right in understanding that there is no Python API access to the RevIndex? Or does the code in src/sourmash/index/revindex.py make use of the updated class?

thanks!

[luizirber]

@bluegenes can you remind me of the relationship between this branch and the pyo3_dependency on commit ff1092f (which may no longer exist in this repo...?)

I stopped rebasing this branch and latest and started merging instead, so commits should be available (even if the branch is deleted, but maybe keep it around for a bit).

sourmash-bio/sourmash_plugin_branchwater#134 uses this PR instead, and can flip to use 0.12.0 once this PR is merged and released. We can open smaller PRs to address shortcomings (and release 0.12.x versions, or 0.13.0 if we need to change the API), or add features as needed by pyo3_branchwater.

@luizirber my main request is that you update the description of the PR so that we can refer back to this in the future. Specific thoughts for things to add (or maybe questions) -

PR description updated!

this bumps the sourmash-rs package to 0.12, right?

👍

am I right in understanding that there is no Python API access to the RevIndex? Or does the code in src/sourmash/index/revindex.py make use of the updated class?

Well... there is a light FFI exposing what is now sourmash::index::revindex::mem_revindex and is the in-memory RevIndex from #1238. This is not well supported (doesn't allow saving/loading from disk, for example), and should be wholly replaced by sourmash::index::revindex::disk_revindex (the on-disk RevIndex).

It is confusing to have these different RevIndex impls in Rust, and I started converging them, but the work is not completely done yet.

Nonetheless, I kept the current tests for the Python RevIndex working, but there are not many of them. #2727 is a better starting point for how Index abc/trait should work acrosss Python/Rust, and I started moving the Rust indices to start from a LinearIndex and later specialize into a RevIndex, which will make easier to translate the work from #2727 for future indices across FFI.

[ctb]

thanks!

@bluegenes I leave this in your capable hands ;)

[ctb]

Per out of band conversation: merge now, release sourmash-rs 0.12.0, integrate/test/fix in sourmash-bio/sourmash_plugin_branchwater#134