Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use pathogen-repo-build in GH Action workflows #1119

Merged
merged 1 commit into from
Jun 27, 2024

Conversation

joverlee521
Copy link
Contributor

@joverlee521 joverlee521 commented Jun 26, 2024

Description of proposed changes

Updates all GH Action workflows for running nextstrain build to use the shared pathogen-repo-build workflow.

Resolves #1113

Testing

TODO

Run times are safely under 12h so we should be good to switch to short-lived sessions.
I'm currently holding off on removing the secrets because it would break @victorlin 's testing for #1106

  • remove AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from repo secrets
  • Delete nextstrain-ncov-runner IAM user and associated policies

Updates all GH Action workflows for running `nextstrain build` to use
the shared `pathogen-repo-build` workflow.

Resolves #1113
@joverlee521
Copy link
Contributor Author

Docs failure unrelated to changes in this PR, it's the same issue as nextstrain/docs.nextstrain.org#209

@joverlee521
Copy link
Contributor Author

joverlee521 commented Jun 26, 2024

The rebuild-country workflow exited with an error that is unrelated to the GH Action workflow changes.

I can replicate the error locally
$ nextstrain build --envdir ~/Repos/env.d/aws/ . results/nextstrain_country_6m/auspice_config.json --profile nextstrain_profiles/nextstrain-country 
Using profile nextstrain_profiles/nextstrain-country for setting default command line arguments.
Config file defaults/parameters.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 8
Rules claiming more threads will be scaled down.
Provided resources: concurrent_deploys=2
Conda environments: ignored
Job stats:
job               count
--------------  -------
auspice_config        1
total                 1

Cannot send slack message as the config does not define a channel and/or token.
Select jobs to execute...

[Wed Jun 26 17:47:36 2024]
Job 0: Making a custom auspice config.
Reason: Missing output files: results/nextstrain_country_6m/auspice_config.json

Config file defaults/parameters.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 8
Rules claiming more threads will be scaled down.
Provided resources: concurrent_deploys=2
Conda environments: ignored
Select jobs to execute...
[Wed Jun 26 17:47:37 2024]
Error in rule auspice_config:
    jobid: 0
    output: results/nextstrain_country_6m/auspice_config.json

RuleException:
ValueError in file /nextstrain/build/workflow/snakemake_rules/export_for_nextstrain.smk, line 146:
too many values to unpack (expected 2)
  File "/nextstrain/build/workflow/snakemake_rules/export_for_nextstrain.smk", line 146, in __rule_auspice_config
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

I don't think this profile ever worked as intended, since the build names have three parts delimited by underscores (e.g. nextstrain_country_1m), while our auspice config rule only expects two parts:

if "_" in build_name:
build_region, build_timespan = build_name.split("_")

Also worth noting that I cannot find any history of the rebuild-country GH Action workflow being used. I only see my one trial run in the run history

$ gh run list --workflow rebuild-country.yml
STATUS  TITLE                                   WORKFLOW                 BRANCH                   EVENT              ID          ELAPSED   AGE               
X       Rebuild country-specific phylogenet...  Rebuild country-spec...  use-pathogen-repo-build  workflow_dispatch  9671643994  1h28m21s  about 17 hours ago

My plan is to disable the rebuild-country GH Action workflow for now and fix it (or remove it) in a separate PR. Tracking in #1120

@joverlee521 joverlee521 marked this pull request as ready for review June 26, 2024 19:29
@joverlee521 joverlee521 requested a review from a team June 26, 2024 19:29
@joverlee521 joverlee521 merged commit 0336d0e into master Jun 27, 2024
58 of 61 checks passed
@joverlee521 joverlee521 deleted the use-pathogen-repo-build branch June 27, 2024 16:35
@joverlee521 joverlee521 self-assigned this Jun 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Port builds to pathogen-repo-build
2 participants