TheiaValidate_PHB: new features and new Docker image from TheiaValidate repository

[sage-wright]

theiagen/theiavalidate#1 partially

closes #281

🛠️ Changes Being Made

This PR moves the entirety of the TheiaValidate code into its own repository. It also implements a few new features:

• a table outputting validation-criteria failures only
• a new input that can translate different column names between tables to enable comparison

Impacted Workflows/Tasks

• TheiaValidate_PHB
• the task compare_two_tsvs in task_validate.wdl is now task theiavalidate

🧠 Context and Rationale

The Python code was exceeding our limit of 100 lines in a task file so we decided to transfer all the code into its own repository. Also, some changes and enhancements had been requested highly and so were implemented as well.

📋 Workflow/Task Steps

Inputs

Renamed inputs:

• table1 -> table1_name
• table2 -> table2_name

New outputs:

• Boolean debug_output - set to true for additional outputs
• na_values - provide a comma-separated list of values to be used as "NAs"
• column_translation_tsv - a TSV file that matches columns of different names to each other; it "translates" column names between tables

Outputs

Renamed outputs:

• validation_report -> theiavalidate_report
• validation_status -> theiavalidate_status
• input_table1 -> theiavalidate_filtered_table1
• input_table2 -> theiavalidate_filtered_table2
• validation_differences_table -> theiavalidate_exact_differences
• theiavalidate_version -> `theiavalidate_wf_version

New outputs:

• theiavalidate_version - version of TheiaValidate Python script in Docker
• theiavalidate_criteria_differences - same as exact_differences but only for differences that fail to meet the validation criteria

Impacted Outputs

Several old TheiaValidate outputs have been deprecated, but are still accessible under a different name (see above).

🧪 Testing

Locally

Terra

See: https://app.terra.bio/#workspaces/theiagen-validations/Theiagen_Wright_PHBG_Sandbox/job_history/cbb69629-f456-4e17-91ad-946397a39161
(the failure was expected)

Scenarios for Reviewer to Test

• compare validation outputs to outputs generated previously
• ensure values that passed validation criteria are not present in the validation criteria differences output

🔬 Quality checks

Pull Request (PR) checklist:

• Include a description of what is in this pull request in this message.
• The workflow/task has been tested locally and on Terra
• The CI/CD has been adjusted and tests are passing
• Everything follows the style guide

[michellescribner]

Tested successfully: https://app.terra.bio/#workspaces/theiagen-validations/Theiagen_Scribner_Sandbox/job_history/db64e855-159d-4d60-96e1-bb3820006b70
Tested on 40 samples that were run with 2 different GAMBIT databases. 2 samples also had small assembly length differences.

• workflow completed successfully
• differences in exact differences file matched previous results
• differences that passed validation criteria were not listed in the validation criteria differences file
  Looks great! Happy to approve pending merge conflict resolution

[michellescribner]

Great job Sage!

[kevinlibuit]

Oh, great call in adding a column_translation_tsv input! Do you have an example of this option utilized on Terra?

[michellescribner]

@kevinlibuit @sage-wright I did not test this option but I can do that now

[michellescribner]

Tested: https://app.terra.bio/#workspaces/theiagen-validations/Theiagen_Scribner_Sandbox/job_history/45a03c16-8e84-46af-ab8b-759041b85426
Looks good though you can't have only one row in column_translation_tsv - will break the workflow. This is described in TheiaValidate repo