update dev

[priyanka-surana]

• New modules structure
• Some input/outputs were changed and resulting in downstream modifications
• Added temporary gunzip functionality
• Issues Update bed_filter #47, update bedtools_bamtobed config settings #48, N50 from GoaT not always available #49, Update create_table to deal with optional inputs #50, Remove bedtools_sort #51
• Added chromosome and organelle information

PR checklist

• This comment contains a description of changes (with reason).
• If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the nf-core/genomenote branch on the nf-core/test-datasets repository.
• Make sure your code lints (nf-core lint).
• Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
• Usage Documentation in docs/usage.md is updated.
• Output Documentation in docs/output.md is updated.
• CHANGELOG.md is updated.
• README.md is updated (including new tool citations and authors/contributors).

[priyanka-surana]

Both unit tests and full tests successful locally. The final output table for a full run looks as below:

##Assembly_Information
Accession,GCA_934047225.1
Organism_Name,Ypsolopha sequella
ToL_ID,ilYpsSequ2
Taxon_ID,1870436
Assembly_Name,ilYpsSequ2.1
Assembly_Level,Chromosome
Life_Stage,adult
Tissue,WHOLE ORGANISM
Sex,NOT COLLECTED
##Assembly_Statistics
Total_Sequence,866863759
Chromosomes,30
Scaffolds,150
Scaffold_N50,28919749
Contigs,228
Contig_N50,20841000
GC_Percent,36
##Chromosome,Length,GC_Percent
1,34252299,35.5
...
29,16431280,36
Z,51978369,35.5
##Organelle,Length,GC_Percent
Mitochondrion,15291,19.5
##BUSCO
Lineage,lepidoptera_odb10
Summary,"C:98.0%[S:96.8%,D:1.2%],F:0.5%,M:1.5%,n:5286"
##MerquryFK,ilYpsSequ2
QV,65.2
Completeness,100.00

[priyanka-surana]

@muffato Once we figure out the correct sort command, I will make the changes.

[priyanka-surana]

bed sort is fixed and unit testing was completed successfully with only -k4.

[muffato]

(adding @BethYates so that she can comment on the output of the pipeline and how data are retrieved / computed)

[BethYates]

Both unit tests and full tests successful locally. The final output table for a full run looks as below:

##Assembly_Information
Accession,GCA_934047225.1
Organism_Name,Ypsolopha sequella
ToL_ID,ilYpsSequ2
Taxon_ID,1870436
Assembly_Name,ilYpsSequ2.1
Assembly_Level,Chromosome
Life_Stage,adult
Tissue,WHOLE ORGANISM
Sex,NOT COLLECTED
##Assembly_Statistics
Total_Sequence,866863759
Chromosomes,30
Scaffolds,150
Scaffold_N50,28919749
Contigs,228
Contig_N50,20841000
GC_Percent,36
##Chromosome,Length,GC_Percent
1,34252299,35.5
...
29,16431280,36
Z,51978369,35.5
##Organelle,Length,GC_Percent
Mitochondrion,15291,19.5
##BUSCO
Lineage,lepidoptera_odb10
Summary,"C:98.0%[S:96.8%,D:1.2%],F:0.5%,M:1.5%,n:5286"
##MerquryFK,ilYpsSequ2
QV,65.2
Completeness,100.00

I think this has most of the information I expect to be able to get from the genome afterparty pipeline. There is also some information I wasn't expecting to be able to source from here but can certainly use for the genome note. The only thing I'm not seeing (other than BlobTool Kit data obviously) is the transcript mappability value, is this something this pipeline will produce?

[priyanka-surana]

I think this has most of the information I expect to be able to get from the genome afterparty pipeline. There is also some information I wasn't expecting to be able to source from here but can certainly use for the genome note. The only thing I'm not seeing (other than BlobTool Kit data obviously) is the transcript mappability value, is this something this pipeline will produce?

@muffato and I had a discussion on the topic of mappability and it is not going to be a number but rather a track on the contact maps. Yes that will be done for the genome using a different subworkflow #23. That would be genome mappability though not transcript mappability.

Transcript mappability requires RNAseq data from what I understand and we don't have enough of those yet to implement those in our pipeline, but eventually the same/similar workflow as above can be used to generate that.

Does that make sense?

[muffato]

This could undergo a small rewrite like the other for+for+if line

[priyanka-surana]

I think the most natural would be to have a single table-creation script that has some mandatory inputs (take genome_summary json and sequence_summary json) and some optional ones (merqury stats, and I would also make the busco json optional in this subworkflow). The summarytable.nf module would be updated accordingly.
It'd be like a funnel, summarising all the inputs that are present. This way, there would be a single output file for this subworkflow, no confusion about which of summary and table is the right output.
Also, retrospectively, it would guarantee that this file has the same format throughout (e.g. w.r.t. the line returns)

Moved back to a single table module which accepts optional inputs. Testing with all inputs and without merqury works. Cannot find an example for without busco.

[muffato]

The script is great, thanks !

[muffato]

For the record, create_table.nf was renamed to createtable.nf to adhere to the naming conventions.