ASCC multi-assembly re-write #65

DLBPointon · 2024-11-13T11:45:41Z

Unsure whether this will actually work, GitHub seems to thing the merge will be simple though.

Re-write adds:

Eeriks work on documentation
Support for multi-assembly runs using the samplesheet
organellar and genomic workflows
- separate include/exclude flags for each, defaults to genomic workflow include "ALL" if not specified.

TODO:

Not right now, but we should continue to look at using Mahesh's pipeline nesting.

github-actions · 2024-11-13T11:46:01Z

Python linting (`black`) is failing

To keep the code consistent with lots of contributors, we run automated code consistency checks.
To fix this CI test, please run:

Install black: pip install black
Fix formatting errors in your pipeline: black .

Once you push these changes the test should pass, and you can hide this comment 👍

We highly recommend setting up Black in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help!

Thanks again for your contribution!

github-actions · 2024-11-13T14:43:25Z

`nf-core pipelines lint` overall result: Failed ❌

Posted for pipeline commit a4e87aa

+| ✅ 156 tests passed       |+
#| ❔  24 tests were ignored |#
!| ❗  20 tests had warnings |!
-| ❌   4 tests failed       |-

❌ Test failures:

nextflow_config - Config variable (incorrectly) found: params.max_cpus
nextflow_config - Config variable (incorrectly) found: params.max_memory
nextflow_config - Config variable (incorrectly) found: params.max_time
files_unchanged - .gitignore does not match the template

❗ Test warnings:

files_exist - File not found: conf/igenomes_ignored.config
nextflow_config - nf-validation has been detected in the pipeline. Please migrate to nf-schema: https://nextflow-io.github.io/nf-schema/latest/migration_guide/
readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
pipeline_todos - TODO string in nextflow.config: Specify your pipeline's command line flags
pipeline_todos - TODO string in README.md: update the following command to include all required parameters for a minimal example
pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
pipeline_todos - TODO string in test.config: Specify the paths to your test data on nf-core/test-datasets
pipeline_todos - TODO string in test.config: Give any required params for the test so that command line flags are not needed
pipeline_todos - TODO string in base.config: Check the defaults for all processes
pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
pipeline_todos - TODO string in ci.yml: You can customise CI pipeline run tests as required
nfcore_yml - nf-core version in .nf-core.yml is not set to the latest version. Should be 3.0.2 but was 2.14.1

❔ Tests ignored:

files_exist - File is ignored: CODE_OF_CONDUCT.md
files_exist - File is ignored: assets/nf-core-ascc_logo_light.png
files_exist - File is ignored: docs/images/nf-core-ascc_logo_light.png
files_exist - File is ignored: docs/images/nf-core-ascc_logo_dark.png
files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: conf/igenomes.config
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting_comment.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/email_template.html
files_unchanged - File ignored due to lint config: assets/nf-core-ascc_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-ascc_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-ascc_logo_dark.png
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/ascc/ascc/.github/workflows/awstest.yml

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-ascc_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowAscc.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-validation plugin
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 1.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .prettierignore matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
modules_config - conf/modules.config found and not ignored.
modules_config - SANGER_TOL_BTK found in conf/modules.config and Nextflow scripts.
modules_config - AUTOFILTER_AND_CHECK_ASSEMBLY found in conf/modules.config and Nextflow scripts.
modules_config - AUTOFILTER_AND_CHECK_ASSEMBLY found in conf/modules.config and Nextflow scripts.
modules_config - ASCC_MERGE_TABLES found in conf/modules.config and Nextflow scripts.
modules_config - FILTER_FASTA found in conf/modules.config and Nextflow scripts.
modules_config - SEQKIT_SLIDING found in conf/modules.config and Nextflow scripts.
modules_config - BLAST_MAKEBLASTDB found in conf/modules.config and Nextflow scripts.
modules_config - DIAMOND_BLASTX found in conf/modules.config and Nextflow scripts.
modules_config - SAMTOOLS_DEPTH found in conf/modules.config and Nextflow scripts.
modules_config - SAMTOOLS_SORT found in conf/modules.config and Nextflow scripts.
modules_config - KRAKEN2_KRAKEN2 found in conf/modules.config and Nextflow scripts.
modules_config - FCS_FCSADAPTOR_PROK found in conf/modules.config and Nextflow scripts.
modules_config - FCS_FCSADAPTOR_EUK found in conf/modules.config and Nextflow scripts.
modules_config - FCS_FCSADAPTOR_EUK found in conf/modules.config and Nextflow scripts.
modules_config - SED_SED found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_ALIGN_SE found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_ALIGN_ILLUMINA found in conf/modules.config and Nextflow scripts.
modules_config - NCBITOOLS_VECSCREEN found in conf/modules.config and Nextflow scripts.
modules_config - FILTER_VECSCREEN_RESULTS found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline

Run details

nf-core/tools version 3.0.2
Run at 2024-11-14 13:25:42

…ents

…naconda

…github_test.yaml

DLBPointon added 8 commits November 12, 2024 15:43

Updates as part of Version 1 rewrite

4403b94

Updates as part of Version 1 rewrite

4589922

Updates as part of Version 1 rewrite

e2dfd3a

Updates from rewrite to main repo

79f36bf

Updates from rewite

fd29a3d

Updating test case and CICD

1366293

Updates from Eeriks docs branch

9bd0beb

Roll back a couple of things

0a8d370

DLBPointon added enhancement New feature or request Release 1 WIP Work In Progress labels Nov 13, 2024

DLBPointon self-assigned this Nov 13, 2024

DLBPointon added 2 commits November 13, 2024 11:50

Formatting updates

2c810e9

Newline

6c49224

DLBPointon mentioned this pull request Nov 13, 2024

Steps update #60

Open

This was linked to issues Nov 13, 2024

Steps update #60

Open

Running the pipeline with primary assembly and haplotigs FASTA files at once #53

Open

DLBPointon mentioned this pull request Nov 13, 2024

Running the pipeline with primary assembly and haplotigs FASTA files at once #53

Open

DLBPointon linked an issue Nov 13, 2024 that may be closed by this pull request

V1.1.0 - [REFACTOR] - Major refactor #44

Open

DLBPointon mentioned this pull request Nov 13, 2024

V1.1.0 - [REFACTOR] - Major refactor #44

Open

DLBPointon added 3 commits November 13, 2024 14:30

Updates for CICD - addition of the modules.json

68f7521

Updates for CICD

c009601

Updates for CICD

fdd1b02

DLBPointon added 5 commits November 13, 2024 15:00

Updates for CICD

91cd465

Adding a test function to parse nextflow version and resouce requirem…

735c78a

…ents

Adding a test function to parse nextflow version and resouce requirem…

834ceb0

…ents

Adding a test function to parse nextflow version and resouce requirem…

cb574c7

…ents

Adding a test function to parse nextflow version and resouce requirem…

dff1b55

…ents

DLBPointon added 30 commits November 21, 2024 14:46

Fixing CI data

048671a

Fixing CI data

2334287

Fixing CI data

7aac083

Updating biopython+numpy images built with anaconda to biopython wo a…

302c18e

…naconda

Linting YML was broken

a28e957

Pre Commit linting

88d7423

Pre Commit linting

4f717cd

Pre Commit linting

6cc000f

Pre Commit linting

35703e4

Pre Commit linting

62552c6

Re-add pipeline download

03f6ead

Re-add pipeline download

7e5f64d

Forgot to point it to the github_testing folder

f5f2e92

Forgot to point it to the github_testing folder

c7dd1a9

Updating notes in the pipeline and removed the BTK_YAML arg from the …

5dd28da

…github_test.yaml

Using realpath so we don't have to pass the bam around like a wally

3d24a42

Updates to pipeline

3b8d923

Uncomment out the organellar workflow

0fb7342

Lint

d2dbd4a

revert blast makeblastdb to original

0e6d799

Updated to fix organellar BTK issue

8e2ec15

Removing local busco dataset call

b05a4a6

Updates

e35a762

Updating setup-nextflow from 1 to 2

aa241a9

Updates

ca770c9

Adding functionality for gzipped input

f432b42

Adding functionality for gzipped input

fa7ebf2

Updates

8afa3ec

Citation for sanger-tol-btk

4d3f395

Note for further updates

4335b1e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ASCC multi-assembly re-write #65

ASCC multi-assembly re-write #65

DLBPointon commented Nov 13, 2024

github-actions bot commented Nov 13, 2024

github-actions bot commented Nov 13, 2024 •

edited

Loading

❌ Test failures:

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

ASCC multi-assembly re-write #65

Are you sure you want to change the base?

ASCC multi-assembly re-write #65

Conversation

DLBPointon commented Nov 13, 2024

github-actions bot commented Nov 13, 2024

Python linting (black) is failing

github-actions bot commented Nov 13, 2024 • edited Loading

nf-core pipelines lint overall result: Failed ❌

❌ Test failures:

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

Python linting (`black`) is failing

github-actions bot commented Nov 13, 2024 •

edited

Loading

`nf-core pipelines lint` overall result: Failed ❌