You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
right now, sourmash sketch fromfile doesn't readily support building signatures in parallel, because we don't have a good way to save the signatures to disk in a threadsafe way from multiple processes, per #1911.
so, for example, you can't have two processes writing to the same collection format.
HOWEVER, using the collection manifest support (e.g. as used for wort-genomes #1965) we could definitely do the following:
write signatures to different .zip files, one for each sketch fromfile process
build a combined manifest across all the different zip files that loads from each zip file
We'd have to do some more testing to make sure it works, but we could implement this all in one go, where the SQL manifest is created up front and then added to by each process, too. Not sure it's necessary.
I'd probably implement this in snakemake, to support multi-node parallelism. But it might need some additional support.
The text was updated successfully, but these errors were encountered:
right now,
sourmash sketch fromfile
doesn't readily support building signatures in parallel, because we don't have a good way to save the signatures to disk in a threadsafe way from multiple processes, per #1911.so, for example, you can't have two processes writing to the same collection format.
HOWEVER, using the collection manifest support (e.g. as used for wort-genomes #1965) we could definitely do the following:
sketch fromfile
processWe'd have to do some more testing to make sure it works, but we could implement this all in one go, where the SQL manifest is created up front and then added to by each process, too. Not sure it's necessary.
I'd probably implement this in snakemake, to support multi-node parallelism. But it might need some additional support.
The text was updated successfully, but these errors were encountered: