Dp24 organellar #35

DLBPointon · 2023-10-13T13:58:51Z

Adds blast for organelles

…t and parse inside script

DLBPointon · 2023-10-13T13:59:28Z

Currently relies on PR #34 to be merged first

github-actions · 2023-10-13T14:04:42Z

Python linting (`black`) is failing

To keep the code consistent with lots of contributors, we run automated code consistency checks.
To fix this CI test, please run:

Install black: pip install black
Fix formatting errors in your pipeline: black .

Once you push these changes the test should pass, and you can hide this comment 👍

We highly recommend setting up Black in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help!

Thanks again for your contribution!

github-actions · 2023-10-13T14:06:38Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 1f3d386

+| ✅ 125 tests passed       |+
#| ❔  21 tests were ignored |#
!| ❗  21 tests had warnings |!

❗ Test warnings:

files_exist - File not found: assets/multiqc_config.yml
readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
pipeline_todos - TODO string in README.md: TODO nf-core:
pipeline_todos - TODO string in README.md: Include a figure that guides the user through the major workflow steps. Many nf-core
pipeline_todos - TODO string in README.md: Fill in short bullet-pointed list of the default steps in the pipeline
pipeline_todos - TODO string in README.md: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
pipeline_todos - TODO string in README.md: update the following command to include all required parameters for a minimal example
pipeline_todos - TODO string in README.md: If applicable, make list of people who have also contributed
pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
pipeline_todos - TODO string in nextflow.config: Specify your pipeline's command line flags
pipeline_todos - TODO string in ci.yml: You can customise CI pipeline run tests as required
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
pipeline_todos - TODO string in base.config: Check the defaults for all processes
pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
pipeline_todos - TODO string in output.md: Write this documentation describing your workflow's output
pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
pipeline_todos - TODO string in WorkflowMain.groovy: Add Zenodo DOI for pipeline after first release
system_exit - System.exit in WorkflowAscc.groovy: System.exit(1) [line 18]

❔ Tests ignored:

files_exist - File is ignored: CODE_OF_CONDUCT.md
files_exist - File is ignored: assets/nf-core-ascc_logo_light.png
files_exist - File is ignored: docs/images/nf-core-ascc_logo_light.png
files_exist - File is ignored: docs/images/nf-core-ascc_logo_dark.png
files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: conf/igenomes.config
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/nf-core-ascc_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-ascc_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-ascc_logo_dark.png
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/ascc/ascc/.github/workflows/awstest.yml
multiqc_config - 'assets/multiqc_config.yml' not found

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: conf/base.config
files_exist - File found: lib/WorkflowAscc.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-ascc_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 1.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
files_unchanged - pyproject.toml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (179 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/yaml'
schema_params - Schema matched params returned from nextflow config
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.10
Run at 2023-11-06 10:07:02

DLBPointon · 2023-10-19T15:09:52Z

Local testing has revealed an error, with a slightly larger data set causing an NCBI C++ error.

yumisims · 2023-10-20T09:32:19Z

bin/filter_barcode_blast_results.py

looks fine for now, but better to consider a better way to write.

I will leave that to Eerik, he understands what is supposed to be happening here.

yumisims · 2023-10-20T09:33:57Z

could Eerik to have a check on this workflow?

DLBPointon · 2023-10-20T14:31:54Z

could Eerik to have a check on this workflow?

That's fine, I just wanted to chase down this error with BLASTN first.
I'll send the error on Slack, hopefully someone understands it a bit more.

yumisims · 2023-10-27T08:52:14Z

I have PR to fix the blastn nf core module, could use it here instead of creating a new module, currently the PR is waiting for nf-core to fix conda problem then merge. nf-core/modules#4245 (comment)

DLBPointon · 2023-10-27T09:33:59Z

I have PR to fix the blastn nf core module, could use it here instead of creating a new module, currently the PR is waiting for nf-core to fix conda problem then merge. nf-core/modules#4245 (comment)

PR waiting on above fix by nf-core + nf-test

…eblastdb ones

bin/organelle_contamination_recommendation.py

eeaunin · 2023-10-20T14:58:39Z

modules/local/filter_comments.nf

+            'docker.io/ubuntu:20.04' }"
+
+    input:
+    tuple val(meta), path(busco)


What's this file used for? ASCC doesn't currently use BUSCO, so I'm confused about what this script does

ASCC can run the BTK-BUSCO SnakeMake pipeline but its own scripts don't use BUSCO

eeaunin · 2023-11-06T13:34:13Z

conf/modules.config

            path: { "${params.outdir}/pipeline_info" },
            mode: params.publish_dir_mode,
            pattern: '*_versions.yml'
        ]
    }

    withName: KRAKEN2_KRAKEN2 {
-        ext.args = { "--report-zero-counts --use-names --memory-mapping" }
+        ext.args    = { "--report-zero-counts --use-names --memory-mapping" }


In the https://github.com/sanger-tol/cobiontcheck/tree/develop version I'm using Kraken2 without memory mapping

eeaunin · 2023-11-06T13:38:56Z

modules/local/get_lineage_for_top.nf

@@ -9,16 +9,16 @@ process GET_LINEAGE_FOR_TOP {

    input:
    tuple val(meta), path(tophits)
-    path( ncbi_taxonomy_path )
+    path( accessions_folder )


Renaming ncbi_taxonomy_path to accessions_folder is a good change. Perhaps it would be even clearer what this folder contains if it was called accession2taxid_folder?

DLBPointon added 5 commits October 9, 2023 14:26

Adding organellar blast subworkflow

27b3fe0

Merge branch 'dev' into dp24_organellar

7565797

Updates

2abac20

Merge branch 'dp24_barcodes' into dp24_organellar

fcf213e

Completing organelle blast, modified python script to accept arrayLis…

0497a94

…t and parse inside script

DLBPointon added enhancement New feature or request Release 1 labels Oct 13, 2023

DLBPointon requested review from yumisims and weaglesBio October 13, 2023 13:58

DLBPointon self-assigned this Oct 13, 2023

Black linting

e79e96f

DLBPointon added 7 commits October 13, 2023 16:23

Uncommenting BLAST for testing

a33877a

Merge branch 'dev' into dp24_organellar

014d413

Updates

8b2ec57

Updates for Organella Blast output checking

f613cac

Local update for diamond blast

d5e7525

Updating use of variables

8055b7b

Updated to filter out empties

245465a

DLBPointon added the WIP Work In Progress label Oct 19, 2023

yumisims reviewed Oct 20, 2023

View reviewed changes

yumisims requested review from eeaunin and removed request for weaglesBio October 20, 2023 09:33

DLBPointon added 2 commits October 26, 2023 15:09

Fixes for blast subworkflows

729d338

Added a blast module that does not rely on makeblastdb

702399c

Fixes to better allow running on github

85ab1cc

DLBPointon removed the WIP Work In Progress label Oct 26, 2023

DLBPointon added 6 commits November 1, 2023 14:25

Updating modules

4294edf

Updating modules and patches

b5909cf

Updating pipeline to reflect changes

2010d7f

Generalising the Blast module for FULL databases as well as local mak…

73639d7

…eblastdb ones

Prettier linting

3b928bf

Adding tracedir to schema

a5bf4a5

yumisims reviewed Nov 2, 2023

View reviewed changes

bin/organelle_contamination_recommendation.py Show resolved Hide resolved

DLBPointon added 2 commits November 6, 2023 10:04

Update script based on recomendation

87bc604

Black Linting

1f3d386

eeaunin approved these changes Nov 6, 2023

View reviewed changes

yumisims merged commit 5196d4d into dev Nov 6, 2023
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dp24 organellar #35

Dp24 organellar #35

DLBPointon commented Oct 13, 2023

DLBPointon commented Oct 13, 2023

github-actions bot commented Oct 13, 2023

github-actions bot commented Oct 13, 2023 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

DLBPointon commented Oct 19, 2023

yumisims Oct 20, 2023

DLBPointon Oct 20, 2023

yumisims commented Oct 20, 2023

DLBPointon commented Oct 20, 2023

yumisims commented Oct 27, 2023

DLBPointon commented Oct 27, 2023

eeaunin Oct 20, 2023

eeaunin Nov 6, 2023

eeaunin Nov 6, 2023

eeaunin Nov 6, 2023

Dp24 organellar #35

Dp24 organellar #35

Conversation

DLBPointon commented Oct 13, 2023

DLBPointon commented Oct 13, 2023

github-actions bot commented Oct 13, 2023

Python linting (black) is failing

github-actions bot commented Oct 13, 2023 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

DLBPointon commented Oct 19, 2023

yumisims Oct 20, 2023

Choose a reason for hiding this comment

DLBPointon Oct 20, 2023

Choose a reason for hiding this comment

yumisims commented Oct 20, 2023

DLBPointon commented Oct 20, 2023

yumisims commented Oct 27, 2023

DLBPointon commented Oct 27, 2023

eeaunin Oct 20, 2023

Choose a reason for hiding this comment

eeaunin Nov 6, 2023

Choose a reason for hiding this comment

eeaunin Nov 6, 2023

Choose a reason for hiding this comment

eeaunin Nov 6, 2023

Choose a reason for hiding this comment

Python linting (`black`) is failing

github-actions bot commented Oct 13, 2023 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️