From 0d3dd52d7a34d0f03d586d26b3f198a2161a939c Mon Sep 17 00:00:00 2001 From: jvfe Date: Thu, 24 Oct 2024 13:01:12 -0300 Subject: [PATCH 1/7] tutorials: Add cross-org tutorial --- .../cross_organisational_subworkflows.md | 57 +++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md diff --git a/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md b/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md new file mode 100644 index 0000000000..29c490f1a8 --- /dev/null +++ b/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md @@ -0,0 +1,57 @@ +--- +title: 'Using subworkflows with modules from multiple organisations' +subtitle: Guidance on how to use nf-core/tools with subworkflows that use modules from more than one organisation. +--- + +In order to use cross-organisational subworkflows, you can specify individual remotes to be used for each module within the `meta.yml` +file in that subworkflow. + +For example, let's say you have a subworkflow in your own organisation's `modules` repository that uses FastQC and fastp. +But, instead of using the fastp module from your own organisation, you want to use nf-core's fastp module. This is the case for the `fastq_trim_fastp_fastqc` subworkflow in the [nf-core-test](https://github.com/nf-core-test/modules/) organization. + +In order to have this subworkflow, with a FastQC module from your organisation and fastp from nf-core, you'd define the +components section of this subworkflow's `meta.yml` file as such: + + +```yaml +name: "fastq_trim_fastp_fastqc" +description: Read QC, fastp trimming and read qc +keywords: + - qc + - quality_control + - adapters + - trimming + - fastq +components: + - fastqc + - fastp: + git_remote: https://github.com/nf-core/modules.git +``` + +By using the structure above, `nf-core/tools` knows that it should install FastQC from the same repository that the subworkflow is located +in, but fastp should come from the `nf-core/modules` remote. + +:::note +Be aware that to install subworkflows from organisations other than nf-core you must +specify `--git-remote` in the `nf-core subworkflows` command. +::: + +If you're using cross-organisational subworkflows in your repository, be aware that you must also specify a different JSON schema +to lint the `meta.yml` files for your subworkflows. +To do this, just change the `components` section type to allow both objects and strings. + +```json +"components": { + "type": "array", + "description": "Modules and subworkflows used in the subworkflow", + "items": { + "type": ["object", "string"] + }, + "minItems": 0 +}, +``` + +:::tip +To get the full content for the schema, see the corresponding +file in the [nf-core-test](https://github.com/nf-core-test/modules/) modules repository. +::: From 42b84a76b33a4182cbe8e20f53db7e749e83cb60 Mon Sep 17 00:00:00 2001 From: jvfe Date: Thu, 24 Oct 2024 13:20:56 -0300 Subject: [PATCH 2/7] tools: Add note about cross-org subwfs in install --- .../src/content/docs/nf-core-tools/subworkflows/install.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/sites/docs/src/content/docs/nf-core-tools/subworkflows/install.md b/sites/docs/src/content/docs/nf-core-tools/subworkflows/install.md index c6f3b34775..5ed2fec2d9 100644 --- a/sites/docs/src/content/docs/nf-core-tools/subworkflows/install.md +++ b/sites/docs/src/content/docs/nf-core-tools/subworkflows/install.md @@ -24,3 +24,9 @@ There are four additional flags that you can use when installing a subworkflow: - `--force`: Overwrite a previously installed version of the subworkflow. - `--prompt`: Select the subworkflow version using a cli prompt. - `--sha `: Install the subworkflow at a specific commit. + +:::note +You can use the `meta.yml` file to specify different remotes for the components +in your subworkflow. This is useful when you want to use modules from organisations +different from the one where the subworkflow is located. For more information, [see our tutorial](/docs/tutorials/external_usage/cross_organisational_subworkflows). +::: From eade522ac8b7c9f9706ce88ffc377f9a2636767c Mon Sep 17 00:00:00 2001 From: jvfe Date: Thu, 24 Oct 2024 15:38:42 -0300 Subject: [PATCH 3/7] fix: Use correct quotes in yaml metadata --- .../external_usage/cross_organisational_subworkflows.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md b/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md index 29c490f1a8..0add0c110b 100644 --- a/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md +++ b/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md @@ -12,9 +12,8 @@ But, instead of using the fastp module from your own organisation, you want to u In order to have this subworkflow, with a FastQC module from your organisation and fastp from nf-core, you'd define the components section of this subworkflow's `meta.yml` file as such: - ```yaml -name: "fastq_trim_fastp_fastqc" +name: 'fastq_trim_fastp_fastqc' description: Read QC, fastp trimming and read qc keywords: - qc From 8128927cabfb02b9cbcd635f5c605878a161096c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Cavalcante?= Date: Fri, 25 Oct 2024 07:03:40 -0300 Subject: [PATCH 4/7] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Matthias Hörtenhuber --- .../external_usage/cross_organisational_subworkflows.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md b/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md index 0add0c110b..f00cbecf5f 100644 --- a/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md +++ b/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md @@ -12,7 +12,7 @@ But, instead of using the fastp module from your own organisation, you want to u In order to have this subworkflow, with a FastQC module from your organisation and fastp from nf-core, you'd define the components section of this subworkflow's `meta.yml` file as such: -```yaml +```yaml title="meta.yml" {11-12} name: 'fastq_trim_fastp_fastqc' description: Read QC, fastp trimming and read qc keywords: @@ -32,7 +32,7 @@ in, but fastp should come from the `nf-core/modules` remote. :::note Be aware that to install subworkflows from organisations other than nf-core you must -specify `--git-remote` in the `nf-core subworkflows` command. +specify `--git-remote` in the `nf-core subworkflows` command, e.g. `nf-core subworkflows install fastq_trim_fastp_fastqc --git-remote $ORG_NAME{:bash}` ::: If you're using cross-organisational subworkflows in your repository, be aware that you must also specify a different JSON schema From 58b068cd73913fd8256a9fd40ed65acd1df54dd8 Mon Sep 17 00:00:00 2001 From: jvfe Date: Sun, 1 Dec 2024 10:53:19 -0300 Subject: [PATCH 5/7] docs: Link to the actual schema file --- .../external_usage/cross_organisational_subworkflows.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md b/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md index f00cbecf5f..3828e67971 100644 --- a/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md +++ b/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md @@ -52,5 +52,5 @@ To do this, just change the `components` section type to allow both objects and :::tip To get the full content for the schema, see the corresponding -file in the [nf-core-test](https://github.com/nf-core-test/modules/) modules repository. +file in the [nf-core-test](https://github.com/nf-core-test/modules/blob/main/subworkflows/yaml-schema.json) modules repository. ::: From a24f1a99b55810bbc7391c657211c358a7e461bb Mon Sep 17 00:00:00 2001 From: jvfe Date: Sun, 1 Dec 2024 10:55:50 -0300 Subject: [PATCH 6/7] Link to correct subwf --- .../external_usage/cross_organisational_subworkflows.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md b/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md index 3828e67971..33dcd0581a 100644 --- a/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md +++ b/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md @@ -7,7 +7,7 @@ In order to use cross-organisational subworkflows, you can specify individual re file in that subworkflow. For example, let's say you have a subworkflow in your own organisation's `modules` repository that uses FastQC and fastp. -But, instead of using the fastp module from your own organisation, you want to use nf-core's fastp module. This is the case for the `fastq_trim_fastp_fastqc` subworkflow in the [nf-core-test](https://github.com/nf-core-test/modules/) organization. +But, instead of using the fastp module from your own organisation, you want to use nf-core's fastp module. This is the case for the `fastq_trim_fastp_fastqc` subworkflow in the [nf-core-test](https://github.com/nf-core-test/modules/tree/main/subworkflows/nf-core-test/fastq_trim_fastp_fastqc) organization. In order to have this subworkflow, with a FastQC module from your organisation and fastp from nf-core, you'd define the components section of this subworkflow's `meta.yml` file as such: From 1bda3ecf52109cc73b18a06e023b6939f8c986fa Mon Sep 17 00:00:00 2001 From: jvfe Date: Sun, 1 Dec 2024 10:58:18 -0300 Subject: [PATCH 7/7] Use title and highlight line in schema chunk --- .../external_usage/cross_organisational_subworkflows.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md b/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md index 33dcd0581a..b6eb09296f 100644 --- a/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md +++ b/sites/docs/src/content/docs/tutorials/external_usage/cross_organisational_subworkflows.md @@ -39,7 +39,7 @@ If you're using cross-organisational subworkflows in your repository, be aware t to lint the `meta.yml` files for your subworkflows. To do this, just change the `components` section type to allow both objects and strings. -```json +```json title="yaml-schema.json" {5} "components": { "type": "array", "description": "Modules and subworkflows used in the subworkflow",