parallelizing large database search with snakemake

[ctb]

see https://github.com/ctb/2021-sourmash-greymake2 -- README in sum,

2021-sourmash-greymake2

parallelize containment searches of large sourmash databases using
manifests, picklists, and snakemake.

Briefly, this code -

• spits a database manifest into 25 batches and saves the batches into CSV files
• uses the CSV files as picklists to search the database in parallel with prefetch
• combines the resulting prefetch output into a single picklist and then uses that to search the database again, to generate the final output

ref #1664 which did something similar with manifests-of-manifests, but in a more general (and more complex) way.

[ctb]

7 minutes for all GTDB with greymake2 and 4 threads,

        Command being timed: "snakemake -j 4"
        User time (seconds): 1496.79
        System time (seconds): 21.72
        Percent of CPU this job got: 359%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 7:02.35
        Maximum resident set size (kbytes): 775924

vs 21 minutes with normal ol' sourmash (single-threaded).

        Command being timed: "sourmash gather 63.fa.sig gtdb-rs202.genomic.k31.zip -o matches2.csv"
        User time (seconds): 1238.62
        System time (seconds): 7.19
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 20:49.51
        Maximum resident set size (kbytes): 780244

Interesting to note that max memory is the same; that's probably some combination of manifest + matching signatures, although I'm not sure why you wouldn't get num_threads x manifest memory 🤔

[ctb]

after #1891 is merged, we can update this to the manifests directly, w/o picklists.