Refactor module

.travis.yml

@@ -226,7 +226,6 @@ before_script:
     - chmod +x scripts/travis/wgbs_harness.sh
     - chmod +x scripts/travis/docs_harness.sh
     - chmod +x scripts/travis/pylint_harness.sh
-    - ls tests/data/
     - git status
 
     - export PATH="${HOME}/bin:$PATH"

docs/adr.rst

@@ -125,11 +125,21 @@ Other changes include:
 2018-06-27 - Remove reads marked as duplicate by BioBamBam
 ----------------------------------------------------------
 
-BioBamBam only marks reads as duplicate, but does not remove the after. The Tool has been updated to remove the flagged duplicates using samtools with the parameter `-F 1024`. This matches the pipeline used within the `Blueprints project <http://dcc.blueprint-epigenome.eu/#/md/chip_seq_grch37>`_.
+BioBamBam only marks reads as duplicate, but does not remove them after. The Tool has been updated to remove the flagged duplicates using samtools with the parameter `-F 1024`. This matches the pipeline used within the `Blueprints project <http://dcc.blueprint-epigenome.eu/#/md/chip_seq_grch37>`_.
 
 Also performed some tidying of the code to annotate issues that had been highlighted by pylint.
 
 
+2018-07-05 - Refactoring of repo to avoid naming collisions
+-----------------------------------------------------------
+
+The original repo had the tools and tests in a directory that was in the root of the repo. This is problematic when there is sharing of the code as there are collisions in the name space. This has meant that all of the code needs to be moved into a new subdirectory (mg_process_fastq).
+
+There is also the movement of the workflow classes out of the pipeline scripts and into their own module in mg_process_fastq. This means that tworkflows that have been written by others can be more easily imported.
+
+This change will probably necessitate incrementing the major release number to 1.0.0 as this is not a backwards compatible change and will require changes in other repos that rely on mg-process-fastq. As a result this will need to be part of a planned release with other developers.
+
+
 2018-07-11 - Changes FASTQ splitter file management
 ---------------------------------------------------
 
@@ -227,3 +237,9 @@ Ability to handle multiple input single/paired end data and background data file
 ----------------------------------------------------------
 
 The output wig files from BS Seeker2 are now converted to BigWig files by default rather than returning wig files. This is so that they are easier to visualise on the JBrowse interface.
+
+
+2018-10-24 - ChIP-seq Pipeline to use BWA MEM
+---------------------------------------------
+
+Changed the default aligner for the ChIP-seq pipeline from BWA ALN to BWA MEM for speed improvements and better handling of short read data.

docs/common.rst

@@ -16,28 +16,28 @@
 Utility Functions
 =================
 
-.. automodule:: tool
+.. automodule:: mg_process_fastq.tool
 
    Common Functions
    ----------------
 
    The following functions are ones that have been used across multiple tools for
    transformations of the data when requried.
 
-   .. autoclass:: tool.common.cd
+   .. autoclass:: mg_process_fastq.tool.common.cd
       :members:
 
    Alignment Utilities
    -------------------
-   .. autoclass:: tool.aligner_utils.alignerUtils
+   .. autoclass:: mg_process_fastq.tool.aligner_utils.alignerUtils
       :members:
 
    Bam Utilities
    -------------
-   .. autoclass:: tool.bam_utils.bamUtils
+   .. autoclass:: mg_process_fastq.tool.bam_utils.bamUtils
       :members:
 
-   .. autoclass:: tool.bam_utils.bamUtilsTask
+   .. autoclass:: mg_process_fastq.tool.bam_utils.bamUtilsTask
       :members:
 
    FASTQ Functions
@@ -52,7 +52,7 @@ Utility Functions
    The following functions are to provide easy access for iterating through entries
    within a FASTQ file(s) both single and paired.
 
-   .. autoclass:: tool.fastqreader.fastqreader
+   .. autoclass:: mg_process_fastq.tool.fastqreader.fastqreader
       :members:
 
    Splitting
@@ -62,13 +62,13 @@ Utility Functions
    chunks for parallel processing. It is able to work on single and paired end
    files.
 
-   .. autoclass:: tool.fastq_splitter.fastq_splitter
+   .. autoclass:: mg_process_fastq.tool.fastq_splitter.fastq_splitter
       :members:
 
    Entry Functions
    ^^^^^^^^^^^^^^^
 
    The following functions allow for manipulating FASTQ files.
 
-   .. autoclass:: tool.fastq_utils.fastqUtils
+   .. autoclass:: mg_process_fastq.tool.fastq_utils.fastqUtils
       :members:

docs/conf.py

@@ -39,7 +39,7 @@
 # Required to us the dummy_pycompss calls
 sys._run_from_cmdl = True
 
-import tool
+import mg_process_fastq.tool
 
 from mock import Mock as MagicMock
 

docs/index.rst

@@ -29,6 +29,7 @@ MuG - FASTQ Pipelines's documentation!
    full_installation
    pipelines
    tools
+   workflow
    common
    settingUpTravis
    docker

docs/pipelines.rst

@@ -72,13 +72,6 @@ Download and index genome files
             --out_metadata tests/json/output_genome_indexer.json
 
 
-   Methods
-   =======
-
-   .. autoclass:: process_genome.process_genome
-      :members:
-
-
 BioBamBam Alignment Filtering
 -----------------------------
 .. automodule:: process_biobambam
@@ -133,12 +126,6 @@ BioBamBam Alignment Filtering
             --out_metadata tests/json/output_biobambam.json
 
 
-   Methods
-   =======
-   .. autoclass:: process_biobambam.process_biobambam
-      :members:
-
-
 Bowtie2 Alignment
 -----------------
 .. automodule:: process_align_bowtie
@@ -207,12 +194,6 @@ Bowtie2 Alignment
             --out_metadata tests/json/output_bowtie2_paired.json
 
 
-   Methods
-   =======
-   .. autoclass:: process_align_bowtie.process_bowtie
-      :members:
-
-
 BSgenome Builder
 ----------------
 .. automodule:: process_bsgenome
@@ -272,12 +253,6 @@ BSgenome Builder
             --out_metadata tests/json/output_bsgenome.json
 
 
-   Methods
-   =======
-   .. autoclass:: process_bsgenome.process_bsgenome
-      :members:
-
-
 BS Seeker2 Indexer
 ------------------
 .. automodule:: process_bs_seeker_index
@@ -331,12 +306,6 @@ BS Seeker2 Indexer
             --out_metadata tests/json/output_wgbs_index.json
 
 
-   Methods
-   =======
-   .. autoclass:: process_bs_seeker_index.process_bs_seeker_index
-      :members:
-
-
 BS Seeker2 Aligner
 ------------------
 .. automodule:: process_bs_seeker_aligner
@@ -393,12 +362,6 @@ BS Seeker2 Aligner
             --out_metadata tests/json/output_wgbs_align.json
 
 
-   Methods
-   =======
-   .. autoclass:: process_bs_seeker_aligner.process_bs_seeker_aligner
-      :members:
-
-
 BiSulphate Sequencing Filter
 ----------------------------
 .. automodule:: process_bs_seeker_filter
@@ -455,12 +418,6 @@ BiSulphate Sequencing Filter
             --out_metadata tests/json/output_metadata.json
 
 
-   Methods
-   =======
-   .. autoclass:: process_bs_seeker_filter.process_bsFilter
-      :members:
-
-
 BS Seeker2 Methylation Peak Caller
 ----------------------------------
 .. automodule:: process_bs_seeker_peak_caller
@@ -520,12 +477,6 @@ BS Seeker2 Methylation Peak Caller
             --out_metadata tests/json/output_wgbs_peak_caller.json
 
 
-   Methods
-   =======
-   .. autoclass:: process_bs_seeker_peak_caller.process_bs_seeker_peak_caller
-      :members:
-
-
 BWA Alignment - bwa aln
 -----------------------
 .. automodule:: process_align_bwa
@@ -594,12 +545,6 @@ BWA Alignment - bwa aln
             --out_metadata tests/json/output_bwa_aln_paired.json
 
 
-   Methods
-   =======
-   .. autoclass:: process_align_bwa.process_bwa
-      :members:
-
-
 BWA Alignment - bwa mem
 -----------------------
 .. automodule:: process_align_bwa_mem
@@ -668,12 +613,6 @@ BWA Alignment - bwa mem
             --out_metadata tests/json/output_bwa_mem_paired.json
 
 
-   Methods
-   =======
-   .. autoclass:: process_align_bwa_mem.process_bwa_mem
-      :members:
-
-
 ChIP-Seq Analysis
 -----------------
 .. automodule:: process_chipseq
@@ -729,12 +668,6 @@ ChIP-Seq Analysis
             --out_metadata tests/json/output_chipseq.json
 
 
-   Methods
-   =======
-   .. autoclass:: process_chipseq.process_chipseq
-      :members:
-
-
 iDamID-Seq Analysis
 -------------------
 .. automodule:: process_damidseq
@@ -788,12 +721,6 @@ iDamID-Seq Analysis
             --in_metadata tests/json/input_idamidseq.json \
             --out_metadata tests/json/output_idamidseq.json
 
-
-   Methods
-   =======
-   .. autoclass:: process_damidseq.process_damidseq
-      :members:
-
 iNPS
 ----
 .. automodule:: process_iNPS
@@ -848,12 +775,6 @@ iNPS
              --out_metadata tests/json/output_iNPS.json
 
 
-   Methods
-   =======
-   .. autoclass:: process_iNPS.process_iNPS
-      :members:
-
-
 MACS2 Analysis
 --------------
 .. automodule:: process_macs2
@@ -921,12 +842,6 @@ MACS2 Analysis
             --out_metadata tests/json/output_macs2_bgd.json
 
 
-   Methods
-   =======
-   .. autoclass:: process_macs2.process_macs2
-      :members:
-
-
 Mnase-Seq Analysis
 ------------------
 .. automodule:: process_mnaseseq
@@ -981,12 +896,6 @@ Mnase-Seq Analysis
             --out_metadata tests/json/output_mnaseseq.json
 
 
-   Methods
-   =======
-   .. autoclass:: process_mnaseseq.process_mnaseseq
-      :members:
-
-
 RNA-Seq Analysis
 ----------------
 .. automodule:: process_rnaseq
@@ -1039,11 +948,6 @@ RNA-Seq Analysis
             --in_metadata tests/json/input_rnaseq.json \
             --out_metadata tests/json/output_rnaseq.json
 
-   Methods
-   =======
-   .. autoclass:: process_rnaseq.process_rnaseq
-      :members:
-
 
 TrimGalore
 ----------
@@ -1101,12 +1005,6 @@ TrimGalore
             --out_metadata tests/json/output_trimgalore.json
 
 
-   Methods
-   =======
-   .. autoclass:: process_trim_galore.process_trim_galore
-      :members:
-
-
 Whole Genome BiSulphate Sequencing Analysis
 -------------------------------------------
 .. automodule:: process_wgbs
@@ -1165,12 +1063,6 @@ Whole Genome BiSulphate Sequencing Analysis
             --in_metadata tests/json/input_wgbs.json \
             --out_metadata tests/json/output_wgbs.json
 
-
-   Methods
-   =======
-   .. autoclass:: process_wgbs.process_wgbs
-      :members:
-
 Hi-C Analysis
 -------------
 
@@ -1271,8 +1163,3 @@ Hi-C Analysis
             --normalized 1             \
             --tag Human.SRR1658573     \
             --window_type frag
-
-   Methods
-   =======
-   .. autoclass:: process_hic.process_hic
-      :members: