Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adjust documentation for signature outputs to talk about memory usage and streaming #1531

Closed
ctb opened this issue May 17, 2021 · 1 comment · Fixed by #2184
Closed

adjust documentation for signature outputs to talk about memory usage and streaming #1531

ctb opened this issue May 17, 2021 · 1 comment · Fixed by #2184
Labels
doc documentation content or issues

Comments

@ctb
Copy link
Contributor

ctb commented May 17, 2021

In #1493, we added support for saving signatures to more than just .sig or .sig.gz files. A slightly hidden consequence of this is that

(a) zipfile and directory saving can happen progressively with e.g. prefetch, so that signatures will be output "as you go" and not held in memory.
(b) that's not true of the .sig and .sig.gz formats, which require that the signatures be held in memory.

This should be mentioned somewhere in the docs!

@ctb ctb added the doc documentation content or issues label May 17, 2021
@ctb
Copy link
Contributor Author

ctb commented Sep 30, 2023

added to command-line.md in #2184 commit fb45b6c:

Note that when outputting large collections of signatures, some save
formats require holding all the sketches in memory until they can be
written out, and others can save progressively. This can affect memory
usage! Currently .sig and .sig.gz formats are held in memory,
while .zip, directory outputs, and .sqldb formats write progressively
to disk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc documentation content or issues
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant