MRG: provide --internal-storage and --no-internal-storage for index

[ctb]

Provides command-line access to the sourmash v0.15 internalize_storage() function added in sourmash-bio/sourmash#3250, via index --internal-storage (defaults to True).

Documentation is separately updated in #416.

This PR also:

• expands tests to cover both internal and exteranl behaviors;
• renames .rdb to .rocksdb internally.

Punted to other issues:

• relative paths to external storage are still not handled well: Improve support for external storage of sketches in RocksDB indexes #415 - but the breakage is tested in this PR, which is a good start.

Behavior that is not tested:

• relocatability of rocksdb w/internal storage: can we rename the index directory after --internal-storage is used, and will it still work?

[ctb]

(this comment is now outdated, but I'm leaving it in for posterity and reference ;)

With the latest trace statements, I find the problem (as expected ;) here, in the storage::InnerStorage struct.

InnerStorage::from_spec: opening zip://list.sig.zip

Conveniently 😭 this is not a new bug or problem, this is the same problem that I discussed ad nauseum over in sourmash-bio/sourmash#3008, where we found it to be a problem for manifests - triggered initially by sourmash-bio/sourmash#3053.

In brief, the problem is this: when we refer to an external storage, how do we interpret non-absolute paths?

As I wrote in #3008, "I am slowly coming around to the idea that loading things relative to the manifest path is correct."

So I think the Right Fix would be to rejigger the path to the zip file to be interpreted relative to the RocksDB location.

However, there is another fun component to this, which is that I'm not sure it's documented anywhere that RocksDB indices created by this plugin store the sketches externally, which is needed for gather (but not for manysearch)... Per @luizirber on slack,

me:

do RocksDB disk rev indexes, once created, require access to the original signatures to function?

@luizirber:

Yes for gather, no for search
You can also build the index with internal sigs, but I don’t think it’s the default

So anyway I think maybe the default for branchwater plugin should be to build a self-contained index. I'm going to dig into the code a bit more to see if that's possible.

[ctb]

There's another wrinkle that I'm having trouble sorting out - it's not clear to me that fastmultigather on the RocksDB index is actually using the external sketches? For the FS storage, I can remove the sketches and it still works 🤔 .

[ctb]

There's another wrinkle that I'm having trouble sorting out - it's not clear to me that fastmultigather on the RocksDB index is actually using the external sketches? For the FS storage, I can remove the sketches and it still works 🤔 .

OK, I was wrong. It looks like fastmultigather appropriately errors out when the sketches are not available, either via FSStorage or ZipStorage.

The real difference is that ZipStorage errors out on load, while FSStorage errors out as soon as it tries to do anything. So sourmash scripts check fails on ZipStorage load, while FSStorage only fails on fastmultigather. I'll write a test.

[ctb]

The real difference is that ZipStorage errors out on load, while FSStorage errors out as soon as it tries to do anything. So sourmash scripts check fails on ZipStorage load, while FSStorage only fails on fastmultigather. I'll write a test.

Even more fun -

• when using rocksdb built from ZipStorage, if the zip file is not available, fastmultigather will fail to open the rocksdb.
• when using rocksdb built from FSStorage, if the files are not available it looks like fastmultigather produces an empty CSV file.

See commit 310df3d for an example test.

[ctb]

You can also build the index with internal sigs, but I don’t think it’s the default

from slack - luiz says:

hmm, there was a an internal/external divide in the initial implementation, it might not have survived future refactors and the internal (save sigs in rocksdb) is not working anymore. you can see a leftover here:
https://github.com/sourmash-bio/sourmash/blob/e3a4d4bcf150d2d58c36b046c35ed61cf9835686/src/core/src/index/revindex/disk_revindex.rs#L486
in any case, with collections the internal/external thing shouldn't happen anyway: the manifest has the info, and the SIGS column family from above can be used as the storage (with internal_location as key, and the sig as value).

[ctb]

Note:

• feat: RocksDB storage and self-contained RevIndex with internal storage sourmash#3250

[ctb]

@bluegenes could you take a look at this and see if it all makes sense to you? It's ready for merge.

[bluegenes]

makes sense to me!