Skip to content

Commit

Permalink
docs: refactor docs
Browse files Browse the repository at this point in the history
  • Loading branch information
johanneskoester committed Dec 8, 2023
1 parent e3ac0e5 commit aeeee93
Show file tree
Hide file tree
Showing 4 changed files with 24 additions and 22 deletions.
23 changes: 1 addition & 22 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,3 @@
# Snakemake storage plugin: fs

A Snakemake storage plugin that reads and writes from a locally mounted filesystem using rsync.
This is particularly useful when running Snakemake on an NFS.
Complex parallel IO patterns can slow down NFS quite substantially.
The following Snakemake CLI flags allow to avoid such patterns by instructing Snakemake
to copy any input to a fast local scatch disk and copying output files back to NFS at the end of a job.

```bash
snakemake --default-storage-provider fs --shared-fs-usage persistence software-deployment sources source-cache --local-storage-prefix /local/work/$USER
```

with `/local/work/$USER` being the path to the local (non NFS) scratch dir.
Alternatively, these options can be persisted in a profile:

```yaml
default-storage-provider: fs
local-storage-prefix: /local/work/$USER
shared-fs-usage:
- persistence
- software-deployment
- sources
- source-cache
```
See the [snakemake plugin catalog](https://snakemake.github.io/snakemake-plugin-catalog) for documentation.
19 changes: 19 additions & 0 deletions docs/further.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
The following Snakemake CLI flags allow to avoid harmful IO patterns on shared network filesystems by instructing Snakemake
to copy any input to a fast local scatch disk and copying output files back to the network filesystem at the end of a job.

```bash
snakemake --default-storage-provider fs --shared-fs-usage persistence software-deployment sources source-cache --local-storage-prefix /local/work/$USER
```

with `/local/work/$USER` being the path to the local (non-shared) scratch dir.
Alternatively, these options can be persisted in a profile:

```yaml
default-storage-provider: fs
local-storage-prefix: /local/work/$USER
shared-fs-usage:
- persistence
- software-deployment
- sources
- source-cache
```
3 changes: 3 additions & 0 deletions docs/intro.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
A Snakemake storage plugin that reads and writes from a locally mounted filesystem using rsync.
This is particularly useful when running Snakemake on an NFS as complex parallel IO patterns can slow down NFS quite substantially.
See "Further information" for an example configuration in such a scenario.
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ version = "0.1.3"
description = " A Snakemake storage plugin that reads and writes from a locally mounted filesystem using rsync"
authors = ["Johannes Koester <[email protected]>"]
readme = "README.md"
repository = "https://github.com/snakemake/snakemake-storage-plugin-fs"

[tool.poetry.dependencies]
python = "^3.11"
Expand Down

0 comments on commit aeeee93

Please sign in to comment.