what can we use for multithreaded/multiprocess safe signature writing?

[ctb]

per #1671 (comment), we don't have a good way to write signatures to a single file from multiple processes. Such a thing would be nice, but would probably entail some kind of locking...

This is a challenge for things like sourmash sketch fromfile where we would like to have parallel signature sketching but would need to make the output single-threaded.

[ctb]

keywords: parallel output

[ctb]

idea in #2033: write to many different files, use a single manifest to point at the different files.

[ctb]

@dkoslicki asked about multithreaded sketching on matrix chat; I responded:

dug into this a bit more, the real problem is here: #1911 - we don't have a good way to write sketches to a single file from multiple processes.

@luizirber added -

But we can set a specific thread to write and still calculate the sigs in parallel. I did a quick skim in sourmash sketch fromfile and the place that need to change is https://github.com/sourmash-bio/sourmash/blob/b1ddabcb05d3455affa862df33b039348c437d61/src/sourmash/command_sketch.py#L301L351

either going multiprocessing or rewriting this function in Rust can achieve parallelism. Easier if the order of the sketches added is not important (and I don't think it is? manifests are the source of order), but even if it needs to be in the same order it is doable.

[ctb]

see plugin https://github.com/sourmash-bio/pyo3_branchwater, manysketch command, which writes to zip files.