Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

switch default docker image for version_capture to GAR-hosted image; CI change to micromamba #198

Merged
merged 12 commits into from
Oct 2, 2023

Conversation

kapsakcj
Copy link
Contributor

@kapsakcj kapsakcj commented Sep 28, 2023

🛠️ Changes Being Made

Updated /tasks/task_versioning.wdl which is used in many of the WDL workflows in PHB. This is a small but impactful change.

  • exposed docker image as an optional input - just in case the user ever needs to change the docker (hopefully never)
  • set default to GAR-hosted docker image us-docker.pkg.dev/general-theiagen/ubuntu/ubuntu:jammy-20230816
    • Original image is ubuntu:jammy-20230816 downloaded from dockerhub on 2023-09-28

Updated CI:

  • CI now uses micromamba to install miniwdl, cromwell, pytest and other dependencies for CI setup. This was done in attempt to save space on the CI runner hard drive. (it helped a little bit)
  • added a few rm -rf ... commands to cleanup unnecessary stuff that is pre-installed on the CI runners.
  • added tests/config/environment.yml for more discrete control over the CI environment instead of installing on the fly with conda create ...
  • updated md5sums for theiaprok workflows

🧠 Context and Rationale

Doesn't happen often, but occaisionally we see errors in Terra related to pulling docker images from dockerhub. It is said to be one of the least reliable docker image repo's as per the Broad developers.

This would update nearly all WDL workflows to pull from Theiagen's google artifact registry instead of dockerhub, in an attempt to reduce or eliminate the transient errors in Terra.

📋 Workflow/Task Steps

N/A

Inputs

N/A

Outputs

🧪 Testing

Locally

The WDL task ran successfully with miniwdl:

$ miniwdl run -v tasks/task_versioning.wdl 
2023-09-28 18:01:45.539 miniwdl-run read configuration file :: path: "/home/curtis_kapsak/.config/miniwdl.cfg"
2023-09-28 18:01:46.072 wdl.t:version_capture task setup :: name: "version_capture", source: "tasks/task_versioning.wdl", line: 3, column: 1, dir: "/home/curtis_kapsak/github/public_health_bioinformatics/20230928_180145_version_capture", thread: 139949331470144
2023-09-28 18:01:46.074 miniwdl-run.CallCache call cache miss :: cache_file: "/home/curtis_kapsak/.cache/miniwdl/version_capture/fskwfl44k5x66o66bxxrwqlcgcrucu4t/iqjw7i2vwntyuekgvulpp2det2kpwt6c.json"
2023-09-28 18:01:46.280 wdl.t:version_capture docker swarm resources :: workers: 1, max_cpus: 8, max_mem_bytes: 33648046080, total_cpus: 8, total_mem_bytes: 33648046080
2023-09-28 18:01:46.281 wdl.t:version_capture eval :: name: "timezone", value: null
2023-09-28 18:01:46.281 wdl.t:version_capture eval :: name: "docker", value: "us-docker.pkg.dev/general-theiagen/ubuntu/ubuntu:jammy-20230816"
2023-09-28 18:01:46.282 wdl.t:version_capture effective runtime :: docker: "us-docker.pkg.dev/general-theiagen/ubuntu/ubuntu:jammy-20230816", cpu: 1, memory_reservation: 1000000000
2023-09-28 18:01:46.282 wdl.t:version_capture ignored runtime settings :: keys: ["disks", "dx_instance_type"]
2023-09-28 18:01:46.295 wdl.t:version_capture docker image :: tag: "us-docker.pkg.dev/general-theiagen/ubuntu/ubuntu:jammy-20230816", id: "sha256:c6b84b685f35f1a5d63661f5d4aa662ad9b7ee4f4b8c394c022f25023c907b65", RepoDigest: "ubuntu@sha256:aabed3296a3d45cede1dc866a24476c4d7e093aa806263c27ddaadbdce3c1054"
2023-09-28 18:01:47.604 wdl.t:version_capture docker task complete :: service: "86h1keh8mf", task: "brnrcdji2o", node: "is3ehuh1yr", message: "finished"
2023-09-28 18:01:47.605 wdl.t:version_capture docker task exit :: state: "complete", exit_code: 0
2023-09-28 18:01:48.129 wdl.t:version_capture output :: name: "date", value: "2023-09-28"
2023-09-28 18:01:48.130 wdl.t:version_capture output :: name: "phb_version", value: "PHB v1.1.0"
2023-09-28 18:01:48.133 wdl.t:version_capture done
2023-09-28 18:01:48.137 miniwdl-run.CallCache call cache insert :: cache_file: "/home/curtis_kapsak/.cache/miniwdl/version_capture/fskwfl44k5x66o66bxxrwqlcgcrucu4t/iqjw7i2vwntyuekgvulpp2det2kpwt6c.json"
{
  "dir": "/home/curtis_kapsak/github/public_health_bioinformatics/20230928_180145_version_capture",
  "outputs": {
    "version_capture.date": "2023-09-28",
    "version_capture.phb_version": "PHB v1.1.0"
  }
}

Terra

TheiaCov_FASTA ran successfully on Terra: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/1cdd9c35-ba44-4855-9a19-2c928b946dd9

🔬 Quality checks

Pull Request (PR) checklist:

  • Include a description of what is in this pull request in this message.
  • The workflow/task has been tested locally and on Terra
  • The CI/CD has been adjusted and tests are passing
  • Everything follows the style guide

…ut; set default to GAR-hosted docker image. Original image is ubuntu:jammy-20230816 downloaded from dockerhub. tested fine with miniwdl
@kapsakcj kapsakcj changed the title version_capture task changes: switch default docker image for version_capture to GAR-hosted image Sep 28, 2023
@frankambrosio3 frankambrosio3 marked this pull request as ready for review October 2, 2023 13:37
Copy link
Contributor

@frankambrosio3 frankambrosio3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🌟

@frankambrosio3 frankambrosio3 merged commit 4f1c834 into main Oct 2, 2023
15 checks passed
@cimendes cimendes deleted the cjk-update-version-capture-docker branch October 2, 2023 13:40
@kapsakcj kapsakcj changed the title switch default docker image for version_capture to GAR-hosted image switch default docker image for version_capture to GAR-hosted image; CI change to micromamba Oct 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants