Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DSL2: Tweaks to modules config to avoid configuration overwrites #996

Merged
merged 14 commits into from
Jun 16, 2023
Merged

DSL2: Tweaks to modules config to avoid configuration overwrites #996

merged 14 commits into from
Jun 16, 2023

Conversation

TCLamnidis
Copy link
Collaborator

@TCLamnidis TCLamnidis commented Jun 9, 2023 •
edited
Loading

Still pending (potentially):

  • Should BAM input and bam_filtering include some lane information for input BAMs? Maybe merge across lanes as first step if multiple exist, then send to bam filtering? (it's a pretty niche use case anyhow)

Updates:

  • Change all meta.id calls to meta.sample_id calls.
  • More specific tags for each process (from ${meta.id} to ${meta.sample_id}_${meta.library_id}_L${meta.lane} (lane only when it applies).
  • Add meta.genomic_region to process tags for deduplication as ${meta.reference}:${meta.genomic_region}

This PR also enforces some formatting changes to the modules.conf, that will be required from now on:

  • Modules that are used multiple times across various SWFs (e.g. SAMTOOLS_*) MUST be given unique informative names, to avoid configuration "leakage" into unintended parts of the pipeline.
  • All module configurations SHOULD avoid using meta.id as a tag/prefix if possible. Use the minimal unique combination of meta.sample_id, meta.library_id and meta.lane instead.

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
    • If you've added a new tool - add to the software_versions process and a regex to scrape_software_versions.py
    • If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-core/eager/tree/master/.github/CONTRIBUTING.md)
    • If necessary, also make a PR on the nf-core/eager branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core lint .).
  • Ensure the test suite passes (nextflow run . -profile test,docker).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@github-actions
Copy link

github-actions bot commented Jun 9, 2023 •
edited
Loading

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 1abd118

+| ✅ 158 tests passed       |+
!| ❗  19 tests had warnings |!

❗ Test warnings:

  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in nextflow.config: Specify your pipeline's command line flags
  • pipeline_todos - TODO string in README.md: Include a figure that guides the user through the major workflow steps. Many nf-core
  • pipeline_todos - TODO string in README.md: Fill in short bullet-pointed list of the default steps in the pipeline
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your prefered methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
  • pipeline_todos - TODO string in eager.nf: Add all file path parameters for the pipeline to the list below
  • pipeline_todos - TODO string in WorkflowMain.groovy: Add Zenodo DOI for pipeline after first release
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
  • pipeline_todos - TODO string in test_humanbam.config: Specify the paths to your test data on nf-core/test-datasets
  • pipeline_todos - TODO string in test_humanbam.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
  • pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in test.config: Specify the paths to your test data on nf-core/test-datasets
  • pipeline_todos - TODO string in test.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in ci.yml: You can customise CI pipeline run tests as required
  • pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
  • schema_description - No description provided in schema for parameter: skip_damage_calculation

✅ Tests passed:

Run details

  • nf-core/tools version 2.8
  • Run at 2023-06-16 13:47:32

@TCLamnidis TCLamnidis requested review from jfy133 and a team and removed request for jfy133 June 9, 2023 13:23
@scarlhoff scarlhoff mentioned this pull request Jun 12, 2023
Closed
11 tasks
jfy133
jfy133 approved these changes Jun 16, 2023
View reviewed changes
Copy link
Member

@jfy133 jfy133 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, left a few comments but mostly minor so I gave preemptive approval.

While you are doing formatting things, could you also standardise the order of each withName scope so the order of the entries are consistent, e.g.:

  • tag
  • ext.args
  • prefix
  • publishDir

If not already, and whatever order makes sense to you and yougo with should become also standard within the pipeline

conf/modules.config Show resolved Hide resolved
conf/modules.config Outdated Show resolved Hide resolved
conf/modules.config Outdated Show resolved Hide resolved
conf/modules.config Outdated Show resolved Hide resolved
conf/modules.config Show resolved Hide resolved
conf/modules.config Outdated Show resolved Hide resolved
conf/modules.config Outdated Show resolved Hide resolved
conf/modules.config Outdated Show resolved Hide resolved
conf/modules.config Outdated Show resolved Hide resolved
TCLamnidis
TCLamnidis commented Jun 16, 2023
View reviewed changes
conf/modules.config Outdated Show resolved Hide resolved
TCLamnidis
TCLamnidis commented Jun 16, 2023
View reviewed changes
conf/modules.config Outdated Show resolved Hide resolved
TCLamnidis
TCLamnidis commented Jun 16, 2023
View reviewed changes
conf/modules.config Outdated Show resolved Hide resolved
TCLamnidis
TCLamnidis commented Jun 16, 2023
View reviewed changes
conf/modules.config Outdated Show resolved Hide resolved
TCLamnidis
TCLamnidis commented Jun 16, 2023
View reviewed changes
conf/modules.config Outdated Show resolved Hide resolved
@TCLamnidis
Copy link
Collaborator Author

Agreed that we should stick to

  1. tag
  2. ext.args*
  3. ext.prefix
  4. publishDir
    I homogenised what is there.
    @aidaanva what should the tag be for HOST_REMOVAL? Also, how will that work with multiple references? 😬

@jfy133
Copy link
Member

jfy133 commented Jun 16, 2023

LGTM!

aidaanva
aidaanva reviewed Jun 16, 2023
View reviewed changes
conf/modules.config Outdated Show resolved Hide resolved
aidaanva
aidaanva reviewed Jun 16, 2023
View reviewed changes
conf/modules.config Outdated Show resolved Hide resolved
TCLamnidis
TCLamnidis commented Jun 16, 2023
View reviewed changes
conf/modules.config Outdated Show resolved Hide resolved
@TCLamnidis @aidaanva
Apply suggestions from code review
bf7fd39 
Co-authored-by: Aida Andrades Valtueña <[email protected]>
aidaanva
aidaanva requested changes Jun 16, 2023
View reviewed changes
Copy link
Contributor

@aidaanva aidaanva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a couple of comments

@TCLamnidis TCLamnidis merged commit 8828f24 into dev Jun 16, 2023
@TCLamnidis TCLamnidis deleted the dsl2-fix-damage-manipulation branch June 16, 2023 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jfy133 jfy133 jfy133 approved these changes

@aidaanva aidaanva aidaanva approved these changes

Assignees
No one assigned
Labels
DSL2
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@TCLamnidis @jfy133 @aidaanva