[EXP] Lazy signature loading classes that use manifests #1619

ctb · 2021-06-21T18:47:45Z

This PR extends manifest functionality:

creates a new class DirectoryIndex that makes use of manifests.
creates a new class LazyMultiIndex that provisionally supports full lazy loading of signatures from indexes using manifests.

See #1096 (comment) for originating motivation.

There is no particular urgency in moving towards making this merge-ready until we have more use cases. Things to consider:

refactor DirectoryIndex to use FSStorage underneath;
does the new _directoryindex_load cause any potential problems?
maybe split out the changes to src/sourmash/sig/__main__.py into their own PR;
consider splitting LazyMultiIndex from DirectoryIndex; I'm still not sure if the latter is actually useful!
write more tests, too.

codecov-commenter · 2021-06-21T19:22:40Z

Codecov Report

Merging #1619 (e4be24a) into latest (515a06d) will increase coverage by 7.76%.
The diff coverage is 77.17%.

@@            Coverage Diff             @@
##           latest    #1619      +/-   ##
==========================================
+ Coverage   82.74%   90.51%   +7.76%     
==========================================
  Files         122       92      -30     
  Lines       13200     9085    -4115     
  Branches     1780     1800      +20     
==========================================
- Hits        10922     8223    -2699     
+ Misses       2014      597    -1417     
- Partials      264      265       +1

Flag	Coverage Δ
python	`90.51% <77.17%> (-0.15%)`	⬇️
rust	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/sourmash/index/__init__.py	`93.68% <75.60%> (-2.77%)`	⬇️
src/sourmash/sourmash_args.py	`93.42% <87.50%> (-0.12%)`	⬇️
src/sourmash/sig/__main__.py	`92.95% <100.00%> (ø)`
src/core/tests/test.rs
src/core/src/ffi/minhash.rs
src/core/src/index/search.rs
src/core/src/index/linear.rs
src/core/src/index/mod.rs
src/core/src/storage.rs
src/core/src/ffi/index/revindex.rs
... and 23 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 515a06d...e4be24a. Read the comment docs.

@ctb

commit 0f03178 Author: C. Titus Brown <[email protected]> Date: Sun Jul 4 13:56:06 2021 -0700 add check for parent & internal locations in MultiIndex commit 5e6d26f Author: C. Titus Brown <[email protected]> Date: Sun Jul 4 13:40:32 2021 -0700 add test for MultiIndex re filenames commit d6257a6 Author: C. Titus Brown <[email protected]> Date: Sun Jul 4 13:31:24 2021 -0700 emit full location of .sig files under directories commit f4b75fe Author: C. Titus Brown <[email protected]> Date: Sun Jul 4 13:30:49 2021 -0700 update describe output test for full filename stuff commit 12c3424 Author: C. Titus Brown <[email protected]> Date: Sun Jul 4 13:23:07 2021 -0700 slight refactor of SignatureLoadingProgress commit 5b02dce Author: C. Titus Brown <[email protected]> Date: Sun Jul 4 08:02:17 2021 -0700 more comment commit 445f131 Author: C. Titus Brown <[email protected]> Date: Sun Jul 4 07:55:49 2021 -0700 update docstrings commit 267895f Author: C. Titus Brown <[email protected]> Date: Sun Jul 4 07:55:40 2021 -0700 add test for DirectoryIndex commit 512bbf2 Author: C. Titus Brown <[email protected]> Date: Sun Jul 4 07:21:14 2021 -0700 add directory load w/manifest into _load_databases commit 7c4949f Author: C. Titus Brown <[email protected]> Date: Sun Jul 4 07:03:42 2021 -0700 add DirectoryIndex commit 9726be9 Author: C. Titus Brown <[email protected]> Date: Sun Jul 4 06:34:47 2021 -0700 rename 'locations' to 'index_locations' for ManifestOfManifests commit aeaf63c Author: C. Titus Brown <[email protected]> Date: Sat Jul 3 16:00:30 2021 -0700 simplify database load list commit 8d1f9a7 Author: C. Titus Brown <[email protected]> Date: Sat Jul 3 15:54:17 2021 -0700 update _signatures_with_internal commit c24772d Author: C. Titus Brown <[email protected]> Date: Sat Jul 3 15:32:24 2021 -0700 retain parent origin for MultiIndex directory loading commit fa3a071 Author: C. Titus Brown <[email protected]> Date: Sat Jul 3 15:15:39 2021 -0700 disable failing tests for now commit 6d51ae4 Author: C. Titus Brown <[email protected]> Date: Sat Jul 3 15:15:34 2021 -0700 revert relpath behavior commit 5c3f5fc Author: C. Titus Brown <[email protected]> Date: Sat Jul 3 12:16:49 2021 -0700 make manifest internal_location relative for MultiIndex commit f7e755f Merge: cb1594f 21f5e63 Author: C. Titus Brown <[email protected]> Date: Fri Jul 2 05:40:40 2021 -0700 Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/manifest_lazy_sigfile commit cb1594f Author: C. Titus Brown <[email protected]> Date: Sun Jun 27 08:52:21 2021 -0700 add manifest_of_manifests commit 3a25095 Author: C. Titus Brown <[email protected]> Date: Sun Jun 27 08:04:31 2021 -0700 change from sigfile to lca file for no manifests :) commit cc3b643 Author: C. Titus Brown <[email protected]> Date: Sun Jun 27 07:58:24 2021 -0700 some basic tests of the lazy loading indices commit 61c1b24 Author: C. Titus Brown <[email protected]> Date: Sun Jun 27 07:38:31 2021 -0700 doc strings, comments, etc commit 02c485d Author: C. Titus Brown <[email protected]> Date: Sat Jun 26 09:24:44 2021 -0700 restore LazyLoadedSigfile, add comments/docstrings commit 75ba72b Author: C. Titus Brown <[email protected]> Date: Sat Jun 26 09:12:52 2021 -0700 upgrade debugging output on _load_database commit cc44288 Merge: a60438e e386085 Author: C. Titus Brown <[email protected]> Date: Sat Jun 26 09:03:09 2021 -0700 Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/manifest_lazy_sigfile commit a60438e Author: C. Titus Brown <[email protected]> Date: Thu Jun 24 12:39:17 2021 -0700 fix manifest code commit b633db1 Merge: 6e17c1a 9dbd8b5 Author: C. Titus Brown <[email protected]> Date: Thu Jun 24 11:27:23 2021 -0700 Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/manifest_lazy_sigfile commit 6e17c1a Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 05:39:08 2021 -0700 re-add LazyMultiIndex here commit da381a1 Merge: 0a982cb 99199ee Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 05:35:01 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/manifest_lazy_sigfile commit 0a982cb Merge: 6ebec9c 9bb6a9b Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 05:33:43 2021 -0700 Merge branch 'add/picklist_manifests_sbt' into add/manifest_lazy_sigfile commit 99199ee Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 05:32:52 2021 -0700 move manifest stuff to manifest.py commit 0adee52 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 05:28:54 2021 -0700 remove print commit 83e387e Merge: fe83b68 9bb6a9b Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 18:15:48 2021 -0700 Merge branch 'add/picklist_manifests_sbt' into add/picklist_zf_manifests commit fe83b68 Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 18:07:23 2021 -0700 revert collection to multiindex commit 60a6eec Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 14:08:34 2021 -0700 change LoadedCollection back over to MultiIndex; remove LazyMultiIndex commit 6ebec9c Merge: 287cb7b c3f1a3d Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:53:58 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/manifest_lazy_sigfile commit 9bb6a9b Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:53:40 2021 -0700 fix header writing commit 7486871 Merge: 4221fc9 c3f1a3d Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:53:03 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/picklist_manifests_sbt commit c3f1a3d Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:42:38 2021 -0700 reverse order of adding to seen set commit 71b81ed Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:41:33 2021 -0700 add docstring commit ed5fb7a Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:40:43 2021 -0700 rename matches_siginfo to matches_manifest_row commit 2756e7d Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:28:37 2021 -0700 add save/load test commit ba2e53c Merge: c243b0e c04f137 Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 11:12:07 2021 -0700 Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist_zf_manifests commit c243b0e Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 11:10:20 2021 -0700 add manifest tests commit e301645 Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 10:07:35 2021 -0700 add a test for sig manifest commit e315c90 Merge: d95813e 0814bcc Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 09:58:29 2021 -0700 Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist_zf_manifests commit 287cb7b Author: C. Titus Brown <[email protected]> Date: Sat Jun 19 17:47:04 2021 -0700 a lazy signature loading class, using manifests commit d95813e Author: C. Titus Brown <[email protected]> Date: Sat Jun 19 13:52:50 2021 -0700 add manifest versions commit 4221fc9 Merge: d4a9a2e 31018df Author: C. Titus Brown <[email protected]> Date: Fri Jun 18 05:47:28 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/picklist_manifests_sbt commit 31018df Merge: 9e46ff8 74de59a Author: C. Titus Brown <[email protected]> Date: Fri Jun 18 05:44:38 2021 -0700 Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist_zf_manifests commit d4a9a2e Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 14:01:48 2021 -0700 fix test for manifests commit 2da0085 Merge: a7e153a 9e46ff8 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 14:01:27 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/picklist_manifests_sbt commit 9e46ff8 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 13:55:21 2021 -0700 cleanup of comments etc. commit e1e367a Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 13:49:21 2021 -0700 remove @ctb comments commit 5cad5ff Merge: 54ea3f9 8812142 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 12:22:46 2021 -0700 Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests commit 8812142 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 12:20:45 2021 -0700 further attempt to fix test commit 54ea3f9 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 12:17:27 2021 -0700 only match picklist at end of 'select' commit 122d043 Merge: f697ec4 de6f3c4 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 11:40:39 2021 -0700 Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests commit de6f3c4 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 11:38:35 2021 -0700 remove order dependence from test commit f697ec4 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 09:31:18 2021 -0700 fix coltypes commit 7937292 Merge: bba101c 4d156e9 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 09:24:38 2021 -0700 Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests commit 4d156e9 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 09:13:52 2021 -0700 add docs commit ab286cf Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 08:50:57 2021 -0700 remove debugging print commit c965648 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 08:44:32 2021 -0700 add a test for using prefetch CSV as picklist commit ca6ea4f Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 08:34:08 2021 -0700 add picklist test that checks indexing-and-then-search == index commit bba101c Merge: 39abe57 ba5c8bc Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 08:13:05 2021 -0700 Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests commit ba5c8bc Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 07:47:13 2021 -0700 block multiple picklists on SBTs and LCAs, for now commit a074127 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 07:38:45 2021 -0700 add picklists to lca index commit a0335a3 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 07:32:10 2021 -0700 add picklists to sourmash compare commit c0e5781 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 07:25:59 2021 -0700 add picklists to prefetch commit 7a30b20 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 07:12:17 2021 -0700 add picklists and tests for search, gather, index commit ced72d2 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 06:25:01 2021 -0700 add picklist args throughout, eek. commit 984a557 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 06:14:23 2021 -0700 fix space commit fddf141 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 06:14:09 2021 -0700 move picklist reporting into sourmash_args commit b3c6bb9 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 06:09:40 2021 -0700 move picklist.py from sourmash.sig into sourmash commit 21ce4b7 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 15:41:51 2021 -0700 fix tests for new SignaturePicklist commit b8f4bb8 Merge: 8e5fb8d b787b75 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 15:40:10 2021 -0700 Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist_selectors commit 8e5fb8d Merge: 5ac4671 04c209c Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 12:05:31 2021 -0700 Merge branch 'add/picklist' into add/picklist_selectors commit 04c209c Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 11:27:21 2021 -0700 remove comment commit 14b87d4 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 11:18:32 2021 -0700 trap errors and be nice to users commit 4f8e20c Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 11:16:49 2021 -0700 cover untested code with tests commit 8f65f22 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 10:17:50 2021 -0700 test with --md5 selector commit 9d60e32 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 10:15:06 2021 -0700 documentation commit 3d23d87 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 09:39:59 2021 -0700 add --picklist-require-all &c commit 14a88a7 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 09:34:43 2021 -0700 verify output commit 207a813 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 09:31:22 2021 -0700 more picklist tests commit 9b50748 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 09:19:40 2021 -0700 fix tests :) commit aaa4548 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 09:18:08 2021 -0700 update comments, constructor, etc. commit a7e153a Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 07:22:39 2021 -0700 fix tests commit c356842 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 06:12:24 2021 -0700 done, I think? commit 75dc079 Merge: 1dd8170 39abe57 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 05:25:38 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/picklist_manifests_sbt commit 39abe57 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 05:23:42 2021 -0700 CSV output function commit 1dd8170 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 05:20:35 2021 -0700 add manifests to SBTs commit 72d8497 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 04:23:48 2021 -0700 move manifest stuff into manifest class commit a4057e6 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 10:33:09 2021 -0700 create LazyMultiIndex commit 730a717 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 09:40:45 2021 -0700 more cleanup and docs commit 230c793 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 09:28:09 2021 -0700 cleanup and simplification of ZipFile stuff commit 8a8c3b2 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 08:54:23 2021 -0700 shift signature metadata matching from manifests over to picklist commit ab0fc0e Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 08:45:06 2021 -0700 misc cleanup commit c3b6fc0 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 08:36:08 2021 -0700 more cleanup commit 509eb45 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 08:23:41 2021 -0700 remove MultiIndex commit af5eb86 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 08:09:53 2021 -0700 fix test names for new LoadedCollection commit c6cb1af Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 08:08:46 2021 -0700 fix all the tests commit 915f847 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 07:48:35 2021 -0700 cleanup/simplification of LoadedCollection commit be9ef77 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 07:41:48 2021 -0700 create LoadedCollection to replace MultiIndex non-lazy loading commit 3c0c9cf Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 07:17:08 2021 -0700 try making manifests obligatory for MultiIndex commit 23c1531 Merge: 67a9be1 5ac4671 Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 13:35:20 2021 -0700 Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests commit 5ac4671 Merge: a88b66d 031522c Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 13:35:04 2021 -0700 Merge branch 'add/picklist' into add/picklist_selectors commit 031522c Merge: 3c05f95 ff75ec0 Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 13:34:37 2021 -0700 Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist commit 67a9be1 Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 11:46:07 2021 -0700 more comment commit 1d7e0cf Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 11:41:22 2021 -0700 update comment about picklist.found commit 2f2269b Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 11:33:56 2021 -0700 work through manifests for MultiIndex commit cb8e28d Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 06:34:34 2021 -0700 get started adding manifests to MultiIndex commit 01d33fc Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 06:21:05 2021 -0700 provide 'select' more generically on manifests commit 17b9576 Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 06:12:39 2021 -0700 build out a manifest class a bit commit b2547f3 Author: C. Titus Brown <[email protected]> Date: Sun Jun 13 20:20:28 2021 -0700 add missing manifest CLI file commit 14a5ee1 Author: C. Titus Brown <[email protected]> Date: Sun Jun 13 08:59:00 2021 -0700 hacky but functional manifest support commit 6593a42 Author: C. Titus Brown <[email protected]> Date: Sun Jun 13 08:37:05 2021 -0700 try out manifests commit e205e64 Author: C. Titus Brown <[email protected]> Date: Sun Jun 13 07:39:47 2021 -0700 special case md5 prefixes, for prefetch commit b57b2b3 Author: C. Titus Brown <[email protected]> Date: Sun Jun 13 07:23:46 2021 -0700 support special picklist interactions with zipfile collections commit a88b66d Author: C. Titus Brown <[email protected]> Date: Sun Jun 13 06:32:38 2021 -0700 factor out picklist checks to 'passes_all_picklists' fn commit 54407a3 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 11:16:02 2021 -0700 test 'Index.find' on picklists for SBTs and LCAs commit 03cc61b Merge: de6fc06 3c05f95 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 10:48:59 2021 -0700 Merge branch 'add/picklist' into add/picklist_selectors commit 3c05f95 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 10:48:15 2021 -0700 split column_type out of SignaturePicklist a bit commit 1bdf88e Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 10:28:23 2021 -0700 split pickfile out a little bit commit de6fc06 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 10:48:30 2021 -0700 picklist tests for .signatures() methods on Index classes commit def1933 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 10:48:15 2021 -0700 split column_type out of SignaturePicklist a bit commit a817843 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 10:28:23 2021 -0700 split pickfile out a little bit commit b1fc982 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 10:01:56 2021 -0700 add picklists to selectors commit 74f31f5 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 09:21:58 2021 -0700 track found etc commit 505b04f Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 09:11:24 2021 -0700 basic tests for picklist functionality commit 3ecfb48 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 08:55:28 2021 -0700 integrate picklists into sourmash sig extract commit bb794ec Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 08:38:24 2021 -0700 initial picklist implementation commit 3a583a9 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 07:36:48 2021 -0700 clean up sourmash.sig submodule commit 66b0599 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 07:23:13 2021 -0700 cleanup flakes errors commit 0997834 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 07:17:44 2021 -0700 various cleanups of sourmash_args

* port docstrings and MultiIndex improvements over from #1619 * remove unused import * add tests for MultiIndex manifests, now that they work * comment and parent cleanup

…o add/manifest_lazy_sigfile

…o add/lazy_load_index

…o add/manifest_lazy_sigfile

@ctb

… to use `ZipStorage` (#1598) * port docstrings and MultiIndex improvements over from #1619 * remove unused import * add tests for MultiIndex manifests, now that they work * comment and parent cleanup * Squashed commit of the following: commit 3268907 Author: C. Titus Brown <[email protected]> Date: Fri Jul 2 05:38:56 2021 -0700 remove leftover merge code commit 3b53de9 Merge: 0f7dc81 21f5e63 Author: C. Titus Brown <[email protected]> Date: Fri Jul 2 05:35:54 2021 -0700 Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/zipfile_use_storage commit 0f7dc81 Author: C. Titus Brown <[email protected]> Date: Fri Jun 25 10:45:34 2021 -0700 fix error message commit fc0c6fe Merge: 65646fb a5a52b1 Author: C. Titus Brown <[email protected]> Date: Fri Jun 25 10:17:33 2021 -0700 Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/zipfile_use_storage commit 65646fb Author: C. Titus Brown <[email protected]> Date: Thu Jun 24 11:35:55 2021 -0700 fix merge commit dec537a Merge: c039fd6 9dbd8b5 Author: C. Titus Brown <[email protected]> Date: Thu Jun 24 11:31:47 2021 -0700 Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/zipfile_use_storage commit c039fd6 Merge: 89fad20 8cc96cd Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 17:48:50 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/zipfile_use_storage commit 8cc96cd Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 15:43:19 2021 -0700 fix tests for a CLEAN test-data/prot/ directory commit 5e49336 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 10:32:49 2021 -0700 update docstring commit 2438d90 Merge: 873592d 0ff54e7 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 10:34:23 2021 -0700 Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/picklist_zf_manifests commit 89fad20 Merge: c6a8ad7 0ff54e7 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 10:33:13 2021 -0700 Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/zipfile_use_storage commit c6a8ad7 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 10:32:49 2021 -0700 update docstring commit bd753d2 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 09:04:56 2021 -0700 fix a few more things commit 86ac7ad Merge: 41438a6 873592d Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 09:03:35 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/zipfile_use_storage commit 873592d Merge: b6d5547 1992de9 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 08:48:31 2021 -0700 Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/picklist_zf_manifests commit b6d5547 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 08:47:02 2021 -0700 add test for multiple selects commit 8ebac0d Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 08:37:25 2021 -0700 remove print statements commit 701878b Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 08:36:45 2021 -0700 update test files to have manifest, update tests commit faad6ee Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 08:17:40 2021 -0700 don't test manifest content commit 44aba07 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 08:08:58 2021 -0700 more refactor zipfile select commit 61ce0f2 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 07:37:53 2021 -0700 refactor zipfile select commit 5879ff2 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 07:37:28 2021 -0700 check compatibility in MinHash.intersection_and_union commit e1c44a6 Merge: 6c1f9da d473199 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 07:14:38 2021 -0700 Merge branch 'latest' of http://github.com/sourmash-bio/sourmash into add/picklist_zf_manifests commit 6c1f9da Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 07:12:58 2021 -0700 more manifest testing for zipfiles commit 1b2cf73 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 07:04:47 2021 -0700 add use_manifest fixture, refactor manifest loading commit 38ec792 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 06:50:13 2021 -0700 add sig manifest tests for other file types commit 6905d40 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 06:42:51 2021 -0700 update sig manifest to error when manifests cannot be generated commit fa47667 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 06:19:51 2021 -0700 rename signatures_with_internal to _signatures_with_internal commit 096b141 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 05:46:32 2021 -0700 add manifests to default zip collection output commit 99199ee Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 05:32:52 2021 -0700 move manifest stuff to manifest.py commit 0adee52 Author: C. Titus Brown <[email protected]> Date: Wed Jun 23 05:28:54 2021 -0700 remove print commit 83e387e Merge: fe83b68 9bb6a9b Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 18:15:48 2021 -0700 Merge branch 'add/picklist_manifests_sbt' into add/picklist_zf_manifests commit fe83b68 Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 18:07:23 2021 -0700 revert collection to multiindex commit 60a6eec Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 14:08:34 2021 -0700 change LoadedCollection back over to MultiIndex; remove LazyMultiIndex commit 9bb6a9b Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:53:40 2021 -0700 fix header writing commit 7486871 Merge: 4221fc9 c3f1a3d Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:53:03 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/picklist_manifests_sbt commit 41438a6 Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:52:32 2021 -0700 fix header writing commit 4026855 Merge: 6b18439 c3f1a3d Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:48:33 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/zipfile_use_storage commit c3f1a3d Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:42:38 2021 -0700 reverse order of adding to seen set commit 71b81ed Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:41:33 2021 -0700 add docstring commit ed5fb7a Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:40:43 2021 -0700 rename matches_siginfo to matches_manifest_row commit 2756e7d Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 12:28:37 2021 -0700 add save/load test commit ba2e53c Merge: c243b0e c04f137 Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 11:12:07 2021 -0700 Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist_zf_manifests commit c243b0e Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 11:10:20 2021 -0700 add manifest tests commit e301645 Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 10:07:35 2021 -0700 add a test for sig manifest commit e315c90 Merge: d95813e 0814bcc Author: C. Titus Brown <[email protected]> Date: Tue Jun 22 09:58:29 2021 -0700 Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist_zf_manifests commit d95813e Author: C. Titus Brown <[email protected]> Date: Sat Jun 19 13:52:50 2021 -0700 add manifest versions commit 4221fc9 Merge: d4a9a2e 31018df Author: C. Titus Brown <[email protected]> Date: Fri Jun 18 05:47:28 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/picklist_manifests_sbt commit 6b18439 Merge: 9ff0eab 31018df Author: C. Titus Brown <[email protected]> Date: Fri Jun 18 05:46:39 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/zipfile_use_storage commit 31018df Merge: 9e46ff8 74de59a Author: C. Titus Brown <[email protected]> Date: Fri Jun 18 05:44:38 2021 -0700 Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist_zf_manifests commit 9ff0eab Merge: 9c530a5 9e46ff8 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 14:02:08 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/zipfile_use_storage commit d4a9a2e Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 14:01:48 2021 -0700 fix test for manifests commit 2da0085 Merge: a7e153a 9e46ff8 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 14:01:27 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/picklist_manifests_sbt commit 9e46ff8 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 13:55:21 2021 -0700 cleanup of comments etc. commit e1e367a Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 13:49:21 2021 -0700 remove @ctb comments commit 5cad5ff Merge: 54ea3f9 8812142 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 12:22:46 2021 -0700 Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests commit 8812142 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 12:20:45 2021 -0700 further attempt to fix test commit 54ea3f9 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 12:17:27 2021 -0700 only match picklist at end of 'select' commit 122d043 Merge: f697ec4 de6f3c4 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 11:40:39 2021 -0700 Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests commit de6f3c4 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 11:38:35 2021 -0700 remove order dependence from test commit f697ec4 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 09:31:18 2021 -0700 fix coltypes commit 7937292 Merge: bba101c 4d156e9 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 09:24:38 2021 -0700 Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests commit 4d156e9 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 09:13:52 2021 -0700 add docs commit ab286cf Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 08:50:57 2021 -0700 remove debugging print commit c965648 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 08:44:32 2021 -0700 add a test for using prefetch CSV as picklist commit ca6ea4f Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 08:34:08 2021 -0700 add picklist test that checks indexing-and-then-search == index commit bba101c Merge: 39abe57 ba5c8bc Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 08:13:05 2021 -0700 Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests commit ba5c8bc Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 07:47:13 2021 -0700 block multiple picklists on SBTs and LCAs, for now commit a074127 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 07:38:45 2021 -0700 add picklists to lca index commit a0335a3 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 07:32:10 2021 -0700 add picklists to sourmash compare commit c0e5781 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 07:25:59 2021 -0700 add picklists to prefetch commit 7a30b20 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 07:12:17 2021 -0700 add picklists and tests for search, gather, index commit ced72d2 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 06:25:01 2021 -0700 add picklist args throughout, eek. commit 984a557 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 06:14:23 2021 -0700 fix space commit fddf141 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 06:14:09 2021 -0700 move picklist reporting into sourmash_args commit b3c6bb9 Author: C. Titus Brown <[email protected]> Date: Thu Jun 17 06:09:40 2021 -0700 move picklist.py from sourmash.sig into sourmash commit 21ce4b7 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 15:41:51 2021 -0700 fix tests for new SignaturePicklist commit b8f4bb8 Merge: 8e5fb8d b787b75 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 15:40:10 2021 -0700 Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist_selectors commit 8e5fb8d Merge: 5ac4671 04c209c Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 12:05:31 2021 -0700 Merge branch 'add/picklist' into add/picklist_selectors commit 04c209c Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 11:27:21 2021 -0700 remove comment commit 14b87d4 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 11:18:32 2021 -0700 trap errors and be nice to users commit 4f8e20c Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 11:16:49 2021 -0700 cover untested code with tests commit 8f65f22 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 10:17:50 2021 -0700 test with --md5 selector commit 9d60e32 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 10:15:06 2021 -0700 documentation commit 3d23d87 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 09:39:59 2021 -0700 add --picklist-require-all &c commit 14a88a7 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 09:34:43 2021 -0700 verify output commit 207a813 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 09:31:22 2021 -0700 more picklist tests commit 9b50748 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 09:19:40 2021 -0700 fix tests :) commit aaa4548 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 09:18:08 2021 -0700 update comments, constructor, etc. commit a7e153a Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 07:22:39 2021 -0700 fix tests commit 9c530a5 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 07:12:46 2021 -0700 add comment about Storage encapsulation commit 48fd900 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 07:07:17 2021 -0700 all tests pass, w00t commit d6a48c1 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 07:01:34 2021 -0700 refactor ZipFileLinearIndex to use ZipStorage underneath commit 5a185bb Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 06:45:42 2021 -0700 change internal zipfile writing to use ZipStorage commit c356842 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 06:12:24 2021 -0700 done, I think? commit 75dc079 Merge: 1dd8170 39abe57 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 05:25:38 2021 -0700 Merge branch 'add/picklist_zf_manifests' into add/picklist_manifests_sbt commit 39abe57 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 05:23:42 2021 -0700 CSV output function commit 1dd8170 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 05:20:35 2021 -0700 add manifests to SBTs commit 72d8497 Author: C. Titus Brown <[email protected]> Date: Wed Jun 16 04:23:48 2021 -0700 move manifest stuff into manifest class commit a4057e6 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 10:33:09 2021 -0700 create LazyMultiIndex commit 730a717 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 09:40:45 2021 -0700 more cleanup and docs commit 230c793 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 09:28:09 2021 -0700 cleanup and simplification of ZipFile stuff commit 8a8c3b2 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 08:54:23 2021 -0700 shift signature metadata matching from manifests over to picklist commit ab0fc0e Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 08:45:06 2021 -0700 misc cleanup commit c3b6fc0 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 08:36:08 2021 -0700 more cleanup commit 509eb45 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 08:23:41 2021 -0700 remove MultiIndex commit af5eb86 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 08:09:53 2021 -0700 fix test names for new LoadedCollection commit c6cb1af Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 08:08:46 2021 -0700 fix all the tests commit 915f847 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 07:48:35 2021 -0700 cleanup/simplification of LoadedCollection commit be9ef77 Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 07:41:48 2021 -0700 create LoadedCollection to replace MultiIndex non-lazy loading commit 3c0c9cf Author: C. Titus Brown <[email protected]> Date: Tue Jun 15 07:17:08 2021 -0700 try making manifests obligatory for MultiIndex commit 23c1531 Merge: 67a9be1 5ac4671 Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 13:35:20 2021 -0700 Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests commit 5ac4671 Merge: a88b66d 031522c Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 13:35:04 2021 -0700 Merge branch 'add/picklist' into add/picklist_selectors commit 031522c Merge: 3c05f95 ff75ec0 Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 13:34:37 2021 -0700 Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist commit 67a9be1 Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 11:46:07 2021 -0700 more comment commit 1d7e0cf Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 11:41:22 2021 -0700 update comment about picklist.found commit 2f2269b Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 11:33:56 2021 -0700 work through manifests for MultiIndex commit cb8e28d Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 06:34:34 2021 -0700 get started adding manifests to MultiIndex commit 01d33fc Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 06:21:05 2021 -0700 provide 'select' more generically on manifests commit 17b9576 Author: C. Titus Brown <[email protected]> Date: Mon Jun 14 06:12:39 2021 -0700 build out a manifest class a bit commit b2547f3 Author: C. Titus Brown <[email protected]> Date: Sun Jun 13 20:20:28 2021 -0700 add missing manifest CLI file commit 14a5ee1 Author: C. Titus Brown <[email protected]> Date: Sun Jun 13 08:59:00 2021 -0700 hacky but functional manifest support commit 6593a42 Author: C. Titus Brown <[email protected]> Date: Sun Jun 13 08:37:05 2021 -0700 try out manifests commit e205e64 Author: C. Titus Brown <[email protected]> Date: Sun Jun 13 07:39:47 2021 -0700 special case md5 prefixes, for prefetch commit b57b2b3 Author: C. Titus Brown <[email protected]> Date: Sun Jun 13 07:23:46 2021 -0700 support special picklist interactions with zipfile collections commit a88b66d Author: C. Titus Brown <[email protected]> Date: Sun Jun 13 06:32:38 2021 -0700 factor out picklist checks to 'passes_all_picklists' fn commit 54407a3 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 11:16:02 2021 -0700 test 'Index.find' on picklists for SBTs and LCAs commit 03cc61b Merge: de6fc06 3c05f95 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 10:48:59 2021 -0700 Merge branch 'add/picklist' into add/picklist_selectors commit 3c05f95 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 10:48:15 2021 -0700 split column_type out of SignaturePicklist a bit commit 1bdf88e Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 10:28:23 2021 -0700 split pickfile out a little bit commit de6fc06 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 10:48:30 2021 -0700 picklist tests for .signatures() methods on Index classes commit def1933 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 10:48:15 2021 -0700 split column_type out of SignaturePicklist a bit commit a817843 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 10:28:23 2021 -0700 split pickfile out a little bit commit b1fc982 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 10:01:56 2021 -0700 add picklists to selectors commit 74f31f5 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 09:21:58 2021 -0700 track found etc commit 505b04f Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 09:11:24 2021 -0700 basic tests for picklist functionality commit 3ecfb48 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 08:55:28 2021 -0700 integrate picklists into sourmash sig extract commit bb794ec Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 08:38:24 2021 -0700 initial picklist implementation commit 3a583a9 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 07:36:48 2021 -0700 clean up sourmash.sig submodule commit 66b0599 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 07:23:13 2021 -0700 cleanup flakes errors commit 0997834 Author: C. Titus Brown <[email protected]> Date: Sat Jun 12 07:17:44 2021 -0700 various cleanups of sourmash_args * fix test by raising proper error * update comments * add relevant tests * fix tests and new, exciting bug * add tests for hand-created zipfile with, and without, manifests * remove outdated comment

…o add/manifest_lazy_sigfile

…o add/manifest_lazy_sigfile_upd

ctb · 2022-03-25T13:40:01Z

note, DirectoryIndex in this PR is completely superseded by StandaloneManifestIndex in #1891.

LazyMultiIndex might still be useful, so I'll leave this PR open for a bit.

ctb mentioned this pull request Jun 21, 2021

Supporting on-demand loading of signatures from file lists in search, gather, and prefetch #1096

Closed

ctb changed the title ~~[EXP] a lazy signature loading class, using manifests~~ [EXP] Lazy signature loading classes that use manifests Jun 23, 2021

Base automatically changed from add/picklist_zf_manifests to latest June 24, 2021 18:07

This was referenced Jun 26, 2021

support low-memory search of many on-disk signatures #1641

Closed

[EXP] a simple JSON RPC client for sourmash search #1644

Open

rework database construction and release process to use manifests #1652

Closed

ctb added 5 commits July 5, 2021 05:53

port docstrings and MultiIndex improvements over from #1619

839d406

remove unused import

528fb3b

add tests for MultiIndex manifests, now that they work

fe850a9

comment and parent cleanup

1c49650

ctb force-pushed the add/manifest_lazy_sigfile branch from 0f03178 to 79c0b86 Compare July 5, 2021 13:21

ctb changed the base branch from latest to update/multiindex July 5, 2021 13:22

ctb added 2 commits July 5, 2021 06:24

re-enable tests

8aaa7ac

add update_manifest to ManifestOfManifests

bf486da

Base automatically changed from update/multiindex to latest July 5, 2021 20:26

ctb added 12 commits July 5, 2021 15:00

Merge branch 'latest' of https://github.com/sourmash-bio/sourmash int…

e7f9a89

…o add/manifest_lazy_sigfile

Merge branch 'latest' of https://github.com/sourmash-bio/sourmash int…

eb7934e

…o add/manifest_lazy_sigfile

add comment

2920b0f

do a last round of picklist filtering

d7a98dc

add sig cat --from-file

f88bbcf

add sig cat --from-file

1c23f63

improve output a bit

0599dde

Merge branch 'add/sig_cat_fromfile' into add/manifest_lazy_sigfile

2d7aef6

more output improvements

174e2b2

Merge branch 'add/sig_cat_fromfile' into add/manifest_lazy_sigfile

44f3ea3

add clear to EOL

ae10e28

more update print

eda9d10

ctb added 10 commits July 10, 2021 06:31

do more filtering on sqlite side

86a1936

remove MoM

ef6d426

remove MoM, cleanup

2e412ec

remove unneeded comment

894929a

fix bool; add find to LazyLoadedIndex

55929c8

update comment, remove print

c426476

support picklist type 'manifest' and 'gather'

7c704d6

minor cleanup, add picklist type 'prefetch'

24dc66a

minor cleanup

3d41bba

port LazyLoadedIndex over from another branch

3c6b933

ctb mentioned this pull request Jul 11, 2021

[MRG] implement a lazy/on-demand Index loading class #1661

Merged

1 task

ctb added 8 commits July 11, 2021 12:23

remove reference to LazyMultiIndex

3b7a0a3

mo tests

8c8acbb

Merge branch 'latest' of https://github.com/sourmash-bio/sourmash int…

bebb81b

…o add/lazy_load_index

add test to cover load w/o manifest

95898b1

one moar test

d18cc38

revise load so that create_manifest is no longer done.

062ecc0

Merge branch 'add/lazy_load_index' into add/manifest_lazy_sigfile

6c777ce

Merge branch 'latest' of https://github.com/sourmash-bio/sourmash int…

58153b5

…o add/manifest_lazy_sigfile

ctb added 5 commits October 27, 2021 05:31

Merge branch 'latest' of https://github.com/sourmash-bio/sourmash int…

7253060

…o add/manifest_lazy_sigfile

remove meta coltype stuff (merged in #1660)

5560924

Merge branch 'latest' of https://github.com/sourmash-bio/sourmash int…

4bd0336

…o add/manifest_lazy_sigfile_upd

fix a few things

c8f7447

fix DirectoryIndex to support sourmash sig manifest

e4be24a

ctb mentioned this pull request Mar 22, 2022

[MRG] Add StandaloneManifestIndex class for direct loading of manifest CSVs #1891

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[EXP] Lazy signature loading classes that use manifests #1619

[EXP] Lazy signature loading classes that use manifests #1619

ctb commented Jun 21, 2021 •

edited

Loading

codecov-commenter commented Jun 21, 2021 •

edited by codecov bot

Loading

ctb commented Mar 25, 2022

[EXP] Lazy signature loading classes that use manifests #1619

Are you sure you want to change the base?

[EXP] Lazy signature loading classes that use manifests #1619

Conversation

ctb commented Jun 21, 2021 • edited Loading

codecov-commenter commented Jun 21, 2021 • edited by codecov bot Loading

Codecov Report

ctb commented Mar 25, 2022

ctb commented Jun 21, 2021 •

edited

Loading

codecov-commenter commented Jun 21, 2021 •

edited by codecov bot

Loading