disable call caching for various workflows

[kapsakcj]

Need to test in Terra, but wanted to at least get the draft PR in place.

Testing is done in Terra, ready for review and further testing. I would recommend that the reviewer re-launches workflows on the dev branch cjk-disable-call-cache for workflows that have been run previously. If you don't have previous workflows, launch the workflow twice. Ensure the call cache box IS CHECKED (default behavior) and inspect job manager to ensure cache was not used for the updated WDL tasks

Closes #250 #235

🛠️ Changes Being Made

Added this code to a few different task files:

  meta {
    # added so that call caching is always turned off
    volatile: true
  }

So that call caching is disabled when running these workflows in Terra. In my opinion these are workflows where the expected behavior is to always run from the beginning, regardless of previous attempts.

The benefit of knowing the output files are fresh & not having to troubleshoot after a user accidentally forgets to disable the call caching feature in Terra (which is on by default) far outweigh the benefit of saving on compute time.

Impacted Workflows/Tasks

• basespace_fetch task and workflow
• transfer_column_content task and workflow
• and ncbi_datasets task which impacts assembly_fetch workflow and Snippy_streamline workflow
  • enabled multi-threading for transfer_column_content task: added this flag gsutil -m
• theiavalidate workflow tasks, export_two_tsvs and compare_two_tsvs,

🧠 Context and Rationale

See above for rationale

📋 Workflow/Task Steps

N/A

Inputs

N/A

Outputs

N/A

Impacted Outputs

🧪 Testing

Locally

Did not test locally, change is most relevant to Terra environment. Although these files pass miniwdl check syntax check.

Terra

Will add test workflows later

Scenarios for Reviewer to Test

Would be good to re-launch workflows that have been run in the past. Then inspect the Job Manager in Terra and ensure call caching is off for the specified task.

🔬 Quality checks

Pull Request (PR) checklist:

• Include a description of what is in this pull request in this message.
• The workflow/task has been tested locally and on Terra
• The CI/CD has been adjusted and tests are passing
• Everything follows the style guide

[kapsakcj]

Perhaps also add to export_taxon_tables task? specifically to tasks/utilities/task_broad_terra_tools.wdl

Is this another task that should always run fresh every time? I think yes to prevent from previous workflow outputs from being kept, despite the workflow being re-run.

[kapsakcj]

⚠️ Need to ensure that the call-caching box IS CHECKED at all times during these tests

workflow tests:

• basespace_fetch 1st attempt: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/workflows/theiagen-validations/BaseSpace_Fetch_PHB
  • good, cache was not used
• basesapce_fetch 2nd attempt: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/1038b283-a8d6-4d89-a013-14f667be8708
  • good, cache was not used
• transfer_column_content 1st: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/850aee48-20e9-4109-a8e3-f0a5757edb58
• transfer_column_content 2nd: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/08c4de72-2299-4378-86a6-8689ff881e66
  • good, cache was not used
• assembly_fetch (mainly need to test ncbi datasets task, so just supply an accession) 1st: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/2428d6d4-735a-4517-bad4-5f2380afaaed
• assembly_fetch 2nd: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/b0fb76a6-c37b-42b4-b864-568f45abd543
  • Good, cache was not used
• theiavalidate 1st: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/f264ade4-4eb5-4baa-b353-1346c2e6752b
  • good, cache was not used for either tasks in workflow
• theiavalidate 2nd: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/b2fb608d-451d-4ab4-9827-640b5e36bd18
  • good, cache was not used
• snippy_streamline 1st: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/d4d0650c-38f5-461b-b559-f3f75bcbe7b8
  • good, cache was not used
• snippy_streamline 2nd: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/12d13f9c-d425-40e0-a7d5-4d31d84bd277
  • Good, cache was not used for ncbi datasets task

[sage-wright]

⭐