-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
e3ac0e5
commit aeeee93
Showing
4 changed files
with
24 additions
and
22 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,24 +1,3 @@ | ||
# Snakemake storage plugin: fs | ||
|
||
A Snakemake storage plugin that reads and writes from a locally mounted filesystem using rsync. | ||
This is particularly useful when running Snakemake on an NFS. | ||
Complex parallel IO patterns can slow down NFS quite substantially. | ||
The following Snakemake CLI flags allow to avoid such patterns by instructing Snakemake | ||
to copy any input to a fast local scatch disk and copying output files back to NFS at the end of a job. | ||
|
||
```bash | ||
snakemake --default-storage-provider fs --shared-fs-usage persistence software-deployment sources source-cache --local-storage-prefix /local/work/$USER | ||
``` | ||
|
||
with `/local/work/$USER` being the path to the local (non NFS) scratch dir. | ||
Alternatively, these options can be persisted in a profile: | ||
|
||
```yaml | ||
default-storage-provider: fs | ||
local-storage-prefix: /local/work/$USER | ||
shared-fs-usage: | ||
- persistence | ||
- software-deployment | ||
- sources | ||
- source-cache | ||
``` | ||
See the [snakemake plugin catalog](https://snakemake.github.io/snakemake-plugin-catalog) for documentation. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
The following Snakemake CLI flags allow to avoid harmful IO patterns on shared network filesystems by instructing Snakemake | ||
to copy any input to a fast local scatch disk and copying output files back to the network filesystem at the end of a job. | ||
|
||
```bash | ||
snakemake --default-storage-provider fs --shared-fs-usage persistence software-deployment sources source-cache --local-storage-prefix /local/work/$USER | ||
``` | ||
|
||
with `/local/work/$USER` being the path to the local (non-shared) scratch dir. | ||
Alternatively, these options can be persisted in a profile: | ||
|
||
```yaml | ||
default-storage-provider: fs | ||
local-storage-prefix: /local/work/$USER | ||
shared-fs-usage: | ||
- persistence | ||
- software-deployment | ||
- sources | ||
- source-cache | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
A Snakemake storage plugin that reads and writes from a locally mounted filesystem using rsync. | ||
This is particularly useful when running Snakemake on an NFS as complex parallel IO patterns can slow down NFS quite substantially. | ||
See "Further information" for an example configuration in such a scenario. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -4,6 +4,7 @@ version = "0.1.3" | |
description = " A Snakemake storage plugin that reads and writes from a locally mounted filesystem using rsync" | ||
authors = ["Johannes Koester <[email protected]>"] | ||
readme = "README.md" | ||
repository = "https://github.com/snakemake/snakemake-storage-plugin-fs" | ||
|
||
[tool.poetry.dependencies] | ||
python = "^3.11" | ||
|