[MRG] add sqlite3 implementations for Index, CollectionManifest, and LCA_Database

.github/workflows/python.yml

@@ -1,3 +1,4 @@
+# note: to invalidate caches, adjust the pip-v? and tox-v? numbers below.
 name: Python tests
 
 on:
@@ -35,9 +36,9 @@ jobs:
         uses: actions/cache@v2
         with:
           path: ${{ steps.pip-cache.outputs.dir }}
-          key: ${{ runner.os }}-pip-${{ hashFiles('**/setup.py') }}
+          key: ${{ runner.os }}-pip-v2-${{ hashFiles('**/setup.cfg') }}
           restore-keys: |
-            ${{ runner.os }}-pip-
+            ${{ runner.os }}-pip-v2-
 
       - name: Install dependencies
         run: |
@@ -64,9 +65,9 @@ jobs:
         uses: actions/cache@v2
         with:
           path: .tox/
-          key: ${{ runner.os }}-tox-${{ hashFiles('**/setup.py') }}
+          key: ${{ runner.os }}-tox-v2-${{ hashFiles('**/setup.cfg') }}
           restore-keys: |
-            ${{ runner.os }}-tox-
+            ${{ runner.os }}-tox-v2-
 
       - name: Test with tox
         run: tox

doc/command-line.md

@@ -727,7 +727,7 @@ database. It can be used to combine multiple taxonomies into a single file,
 as well as change formats between CSV and sqlite3.
 
 The following command will take in two taxonomy files and combine them into
-a single taxonomy sqlite database.
+a single taxonomy SQLite database.
 
 ```
 sourmash tax prepare --taxonomy file1.csv file2.csv -o tax.db
@@ -931,6 +931,15 @@ As of sourmash 4.2.0, `lca index` supports `--picklist`, to
 can be used to index a subset of a large collection, or to
 exclude a few signatures from an index being built from a large collection.
 
+As of sourmash 4.4.0, `lca index` can produce an _on disk_ LCA
+database using SQLite. To prepare such a database, use
+`sourmash lca index ... -F sql`.
+
+All sourmash commands work with either type of LCA database (the
+default JSON database, and the SQLite version). SQLite databases are
+larger than JSON databases on disk but are typically much faster
+to load and search, and use much less memory.
+
 ### `sourmash lca rankinfo` - examine an LCA database
 
 The `sourmash lca rankinfo` command displays k-mer specificity
@@ -1399,6 +1408,14 @@ iterating over the signatures in the input file. This can be slow for
 large collections. Use `--no-rebuild-manifest` to load an existing
 manifest if it is available.
 
+As of sourmash 4.4.0, `sig manifest` can produce a manifest in a fast
+on-disk format (a SQLite database). SQLite manifests can be _much_
+faster when working with very large collections of signatures.
+To produce a SQLite manifest, use `sourmash sig manifest ... -F sql`.
+
+All sourmash commands that work with manifests will accept both
+CSV and SQLite manifest files.
+
 ### `sourmash signature check` - compare picklists and manifests
 
 Compare picklists and manifests across databases, and optionally output matches
@@ -1452,7 +1469,7 @@ Briefly,
 
 None of these commands currently support searching, comparing, or indexing
 signatures with multiple ksizes or moltypes at the same time; you need
-to pick the ksize and moltype to use for your search. Where possible,
+to pick the ksize and moltype to use for your query. Where possible,
 scaled values will be made compatible.
 
 ### Selecting signatures 
@@ -1549,9 +1566,10 @@ In addition to `sig extract`, the following commands support
 ### Storing (and searching) signatures
 
 Backing up a little, there are many ways to store and search
-signatures. `sourmash` supports storing and loading signatures from JSON
-files, directories, lists of files, Zip files, and indexed databases.
-These can all be used interchangeably for sourmash operations.
+signatures. `sourmash` supports storing and loading signatures from
+JSON files, directories, lists of files, Zip files, custom indexed
+databases, and SQLite databases.  These can all be used
+interchangeably for most sourmash operations.
 
 The simplest is one signature in a single JSON file. You can also put
 many signatures in a single JSON file, either by building them that
@@ -1567,7 +1585,7 @@ signatures from zip files.  You can create a compressed collection of
 signatures using `zip -r collection.zip *.sig` and then specify
 `collections.zip` on the command line.
 
-### Saving signatures, more generally
+### Choosing signature output formats
 
 (sourmash v4.1 and later)
 
@@ -1583,6 +1601,7 @@ This behavior is triggered by the requested output filename --
 * to save to gzipped JSON signature files, use `.sig.gz`;
 * to save to a Zip file collection, use `.zip`;
 * to save signature files to a directory, use a name ending in `/`; the directory will be created if it doesn't exist;
+* to save to a SQLite database, use `.sqldb` (as of sourmash v4.4.0).
 
 If none of these file extensions is detected, output will be written
 in the JSON `.sig` format, either to the provided output filename or
@@ -1614,17 +1633,23 @@ Indexed databases can make searching signatures much faster. SBT
 databases are low memory and disk-intensive databases that allow for
 fast searches using a tree structure, while LCA databases are higher
 memory and (after a potentially significant load time) are quite fast.
+SQLite databases (new in sourmash v4.4.0) are typically larger on disk
+than SBTs and LCAs, but in turn are fast to load and support very low
+memory search.
 
 (LCA databases also directly permit taxonomic searches using `sourmash lca`
 functions.)
 
 Commands that take multiple signatures or collections of signatures
-will also work with databases.
-
-One limitation of indexed databases is that both SBT and LCA database
-can only contain one "type" of signature (one ksize/one moltype at one
-scaled value). If the database signature type is incompatible with the
-other signatures, sourmash will complain appropriately.
+will also work with indexed databases.
+
+One limitation of indexed databases is that they are all restricted in
+to certain kinds of signatures. Both SBT and LCA databases can only
+contain one "type" of signature (one ksize/one moltype at one scaled
+value). SQLite databases can contain multiple ksizes and moltypes, but
+only at one scaled value. If the database signature type is
+incompatible with the other signatures, sourmash will complain
+appropriately.
 
 In contrast, signature files, zip collections, and directory
 hierarchies can contain many different types of signatures, and

setup.cfg

@@ -43,6 +43,7 @@ install_requires =
     scipy
     deprecation>=2.0.6
     cachetools>=4,<5
+    bitstring>=3.1.9,<4
 python_requires = >=3.8
 
 [bdist_wheel]

src/sourmash/cli/lca/index.py

@@ -59,6 +59,12 @@ def subparser(subparsers):
         '--fail-on-missing-taxonomy', action='store_true',
         help='fail quickly if taxonomy is not available for an identifier',
     )
+    subparser.add_argument(
+        '-F', '--database-format',
+        help="format of output database; default is 'json')",
+        default='json',
+        choices=['json', 'sql'],
+    )
 
     add_ksize_arg(subparser, 31)
     add_moltype_args(subparser)

src/sourmash/cli/search.py

@@ -56,6 +56,10 @@ def subparser(subparsers):
         '-q', '--quiet', action='store_true',
         help='suppress non-error output'
     )
+    subparser.add_argument(
+        '-d', '--debug', action='store_true',
+        help='output debug information'
+    )
     subparser.add_argument(
         '--threshold', metavar='T', default=0.08, type=float,
         help='minimum threshold for reporting matches; default=0.08'

src/sourmash/cli/sig/check.py

@@ -55,6 +55,13 @@ def subparser(subparsers):
         help='do not require a manifest; generate dynamically if needed',
         action='store_true'
     )
+    subparser.add_argument(
+        '-F', '--manifest-format',
+        help="format of manifest output file; default is 'csv')",
+        default='csv',
+        choices=['csv', 'sql'],
+    )
+
     add_ksize_arg(subparser, 31)
     add_moltype_args(subparser)
     add_pattern_args(subparser)

src/sourmash/cli/sig/manifest.py

@@ -40,7 +40,12 @@ def subparser(subparsers):
         '--no-rebuild-manifest', help='use existing manifest if available',
         action='store_true'
     )
-
+    subparser.add_argument(
+        '-F', '--manifest-format',
+        help="format of manifest output file; default is 'csv')",
+        default='csv',
+        choices=['csv', 'sql'],
+    )
 
 def main(args):
     import sourmash

src/sourmash/commands.py

@@ -441,7 +441,7 @@ def search(args):
     from .search import (search_databases_with_flat_query,
                          search_databases_with_abund_query)
 
-    set_quiet(args.quiet)
+    set_quiet(args.quiet, args.debug)
     moltype = sourmash_args.calculate_moltype(args)
     picklist = sourmash_args.load_picklist(args)
     pattern_search = sourmash_args.load_include_exclude_db_patterns(args)

src/sourmash/index/__init__.py

@@ -868,6 +868,15 @@ class MultiIndex(Index):
     Note: this is an in-memory collection, and does not do lazy loading:
     all signatures are loaded upon instantiation and kept in memory.
 
+    There are a variety of loading functions:
+    * `load` takes a list of already-loaded Index objects,
+      together with a list of their locations.
+    * `load_from_directory` traverses a directory to load files within.
+    * `load_from_path` takes an arbitrary pathname and tries to load it
+      as a directory, or as a .sig file.
+    * `load_from_pathlist` takes a text file full of pathnames and tries
+      to load them all.
+
     Concrete class; signatures held in memory; builds and uses manifests.
     """
     def __init__(self, manifest, parent, *, prepend_location=False):
@@ -1212,8 +1221,7 @@ def load(cls, location, *, prefix=None):
         if not os.path.isfile(location):
             raise ValueError(f"provided manifest location '{location}' is not a file")
 
-        with open(location, newline='') as fp:
-            m = CollectionManifest.load_from_csv(fp)
+        m = CollectionManifest.load_from_filename(location)
 
         if prefix is None:
             prefix = os.path.dirname(location)
@@ -1245,20 +1253,12 @@ def _signatures_with_internal(self):
         manifest in this class.
         """
         # collect all internal locations
-        iloc_to_rows = defaultdict(list)
-        for row in self.manifest.rows:
-            iloc = row['internal_location']
-            iloc_to_rows[iloc].append(row)
-
-        # iterate over internal locations, selecting relevant sigs
-        for iloc, iloc_rows in iloc_to_rows.items():
-            # prepend with prefix?
+        picklist = self.manifest.to_picklist()
+        for iloc in self.manifest.locations():
+            # prepend location with prefix?
             if not iloc.startswith('/') and self.prefix:
                 iloc = os.path.join(self.prefix, iloc)
 
-            sub_mf = CollectionManifest(iloc_rows)
-            picklist = sub_mf.to_picklist()
-
             idx = sourmash.load_file_as_index(iloc)
             idx = idx.select(picklist=picklist)
             for ss in idx.signatures():