can we turn picklists + collections into manifests?

[ctb]

along the theme of cool ways to subset collections with manifests, @AnneliektH is converting fastgather results into manifests here. This is presumably because mgmanysearch doesn't support picklists (for which fastgather output could then be used), but does supports standalone manifests.

so the question du jour is: do we have a standard way to go from picklists + collections => standalone manifest?

I think sourmash sig check might do it: docs. I will check and then recommend it to annie if so :).

it looks like sourmash sig collect does not, however: docs. That was my first guess.

related:

• how to support lazy/streaming load with standard sourmash functions: standalone manifests #3023
• MRG: rework the manifest documentation; do misc cleanup #3027

[ctb]

sig check seems to work for this!

Build a bunch of individual .sig.zip files:

for i in *.fa
do
   sourmash sketch dna $i -o $i.sig.zip --name-from-first
done

create a picklist for some of them in ident.list

ident
CP001472.1
CP001941.1
CP001071.1
AE000782.1
NC_003272.1

then run sig check --picklist ... -m mf.csv to create a standalone manifest of just the sketches that match to the picklist:

sourmash sig check *.sig.zip --picklist ident.list:ident:ident  -m mf.csv

and

sourmash sig summarize mf.csv

shows just five sketches in the standalone manifest mf.csv, which looks like this:

# SOURMASH-MANIFEST-VERSION: 1.0
internal_location,md5,md5short,ksize,moltype,num,scaled,n_hashes,with_abundance,name,filename
0.fa.sig.zip,324074c7287ed934af4fd0a6a459aa30,324074c7,31,DNA,0,1000,4168,False,"CP001472.1 Acidobacterium capsulatum ATCC 51196, complete genome",0.fa
1.fa.sig.zip,c11126d0591db94cd3d1c8568499375f,c11126d0,31,DNA,0,1000,1478,False,"CP001941.1 Aciduliprofundum boonei T469, complete genome",1.fa
2.fa.sig.zip,f3a90d4e5528864a5bcc8434b0d0c3b1,f3a90d4e,31,DNA,0,1000,2701,False,"CP001071.1 Akkermansia muciniphila ATCC BAA-835, complete genome",2.fa
3.fa.sig.zip,cee0a3fb7b00990a22be02b2b0a78418,cee0a3fb,31,DNA,0,1000,2143,False,"AE000782.1 Archaeoglobus fulgidus DSM 4304, complete genome",3.fa
35.fa.sig.zip,264cfdad44548ad96c4a24b6a514a877,264cfdad,31,DNA,0,1000,7295,False,"NC_003272.1 Nostoc sp. PCC 7120 DNA, complete genome",35.fa

[ctb]

@AnneliektH maybe worth giving it a try ;)

[AnneliektH]

Ok so this would work perfectly, but I did something wrong I think while creating all signatures initially.
I have a zip file that contains all signatures in sourmash/sig_files/signatures_concat/allMAGs.zip, where all zips come from sourmash/sig_files/MAGs.
For most of them , the match_name in a fastgather is only the MAG name, which works (e.g. "SRR8960976_MAG07"), but for some, the match_name is a whole file path to a MAG (eg. "../atlas/MAGs/genomes/all_fasta/fastafiles/AtH2023_SRR8960918_MAG18.fasta") and for those it does not work.
I don't know what i did wrong building signature files that for some I get this and not for others..

Just rebuild all signatures?

[ctb]

that's up to you 😆 - depends on what's easiest. we have sig rename to rename sketches, or you can go back and rebuild them with sourmash sketch (and provide --name this time), or use manysketch from the branchwater plugin to quickly rebuild them with nicer names, or whatever!

you can also ignore all of that and use the md5sum in the picklist, instead of the name/ident, which should also work.

[AnneliektH]

Ok so its almost working..
snakefile here: https://github.com/AnneliektH/2023-swine-sra/blob/main/sourmash/Snakefile_mg

Snakefile makes the manifests (in /group/ctbrowngrp2/scratch/annie/2023-swine-sra/sourmash/manifests/MAGs), where there is a manifest for each clustering treshold of genomes.
Running all of this from the folder 'sourmash', folders described below are subfolders in that folder.

When running mgmanysearch, I get an error, which I think has to do with the file paths of the queries (aka the manifests): Within the manifest, the internal location is correctly stated as "sig_files/signatures_concat/MAGs2.zip"
The manifests themselves are saved in the folder manifests/MAGs/
Now when running mgmanysearch, the error is "ValueError: Error while reading signatures from 'manifests/MAGs/sig_files/signatures_concat/MAGs2.zip"

Seems like it pastes the manifest location in front of the query location, which isnt where the files at

[ctb]

Seems like it pastes the manifest location in front of the query location, which isnt where the files at

Spent time figuring this all out (see #3053 for a demonstration), and I think that the mgmanysearch behavior here is correct and that sig check is creating manifests incorrectly - see breakdown & discussion here, #3008 (comment). I'm fixing the behavior over in #3054.

[ctb]

(once #3054 is merged, solution will be to use sourmash sig check --relpath)