Output AWS Batch job ID from pathogen-repo-build worklfow

[joverlee521]

If downstream jobs/workflows require the AWS Batch job ID of the build, we need to add an output to the pathogen-repo-build workflow. This idea was prompted by nextstrain/seasonal-flu#159. I'm not able to think of any other use case that would require the AWS Batch job ID output...

I don't think it's too much work to add/maintain the output but will revisit this another time.

[jameshadfield]

This seems very similar to nextstrain/zika#52 where we added a job to the workflow as a workaround because we couldn't pass information from one job to another. But artifacts (for files) or outputs (for strings) should let us do this, right?

Or have I misunderstood and the technical difficulty relates to how the job which calls nextstrain build would itself get this information out of the build?

[joverlee521]

Ah, this is a technical bit in the GitHub Action workflow. Now that I'm looking at it again, we already store the AWS Batch job ID as an output from the run-build job.

.github/.github/workflows/pathogen-repo-build.yaml.in

Lines 281 to 282 in d15f4f9

	outputs:
	AWS_BATCH_JOB_ID: ${{ env.AWS_BATCH_JOB_ID }}

We just need to make it an output of the reusable workflow.

[jameshadfield]

Right, and its value is obtained by parsing the build output:

echo "AWS_BATCH_JOB_ID=$(sed -nE 's/.+AWS Batch Job ID\:.+ ([-a-f0-9]+)$/\1/p' < "$NEXTSTRAIN_BUILD_LOG")" | tee -a "$GITHUB_ENV"

This approach is what I'm suggesting we aim for with regards to surfacing relevant information from the reusable workflow. In other words, we design some structured text pattern which (ingest/pathogen) workflows can print and the reusable workflow will parse this from the logs and surface via an output. For instance, we could use this to know whether there were "new data" or "no new data" in an ingest.

[tsibley]

In other words, we design some structured text pattern which (ingest/pathogen) workflows can print and the reusable workflow will parse this from the logs and surface via an output.

Why not produce a file instead of pattern matching on log output?

[jameshadfield]

Why not produce a file instead of pattern matching on log output?

👍 I should have said "design some structure" -- a file would be just fine.