[RFC] hub dashboard orchestration

[zkamvar]

This is still a draft, but it's most of the way there.

This is dependent on #3 and effectively is an RFC for a decision that was
already made and implemented, but it would be good to have a discussion around
this since there is still some uncertainty.

[bsweger]

We will host the App's webhook code on a free service like glitch.io

Assuming that this note about not relying solely on GitHub workflows is related to the quoted text from the above list of decisions...

I'm still unclear on why we need to host webhook code on Glitch, etc. instead of having a manually-triggered action in the hub-specific repos (e.g., variant-nowcast-hub-dashboard) that use the GitHub api to trigger a workflow/repository dispatch event in the central dashboard "control room"

Is there a technical and/or code-reuse reason we haven't tried on-demand updates w/o writing our own intervening API?

If not, it's appealing to remove the extra moving part (i.e., Glitch/server). Asking hub admins to manually run a GitHub workflow to update the dashboard doesn't seem overly-burdensome, especially for a first iteration of dashboards.

[bsweger]

Adding that I also saw this note above:

The admins of at least one hubverse project has expressed a desire to not manage
any more workflows.

If there's a way to include a generalizable workflow in the dashboard template (i.e., admins don't have to make changes to it) or if there's a way to share the hub-specific version of the repo, that doesn't seem terrible.

In any case, I'd like to make sure we're making a good tradeoff between user desire for "no more workflows" and the added complexity introduced by another service into the dashboard update process.

[zkamvar]

If there's a way to include a generalizable workflow in the dashboard template (i.e., admins don't have to make changes to it).

As soon as a workflow is introduced, it's out of our control. We can do our darnedest to make sure admins don't have to make changes to it, but I have yet to ever find an example where that's the case.

if there's a way to share the hub-specific version of the repo, that doesn't seem terrible.

I'm not sure I understand this. Can you elaborate?

[bsweger]

Repeating my question from a Zoom call: our synchronous discussion was helpful for understanding how the webhook/API component of this decision improves usability for hub admins while maintaining a good security posture.

That said, a lot of us have pushed back against running and maintaining an API/web service, as it adds a non-trivial moving part to our setup.

As expected, everyone has their own .02 about where to land on the hub admin usability <--> hub dev maintainability spectrum. I'm wondering how much (if at all) the choice of host/language for the API impacts that perspective.

[zkamvar]

I'm wondering how much (if at all) the choice of host/language for the API impacts that perspective.

I suspect that a language for the API that is more aligned with the lab languages would make the app option more appealing (depending on how robust the abstraction is). Most of the challenge with the app code is dealing with GitHub's API: https://github.com/hubverse-org/hub-dashboard-control-room/blob/60fc0660a96f054123d702c7dbe4b2108545f73f/app/index.js#L4-L24

[bsweger]

A way this could work ... GitHub actions in the hub dashboard repos could authenticate to a vault app via its GitHub OIDC provider and get the GitHub app token (HasiCorp Vault or AWS for example). OIDC is how the AWS Upload action is able to push data to a hub's S3 bucket.

As with getting hubs onto the Hubverse-hosted AWS cloud, there would need to be an onboarding process for hub dashboards. Essentially, we'd have to make sure each individual hubverse dashboard repo is allowed to access the vault, either manually or via infrastructure as code (the relevant bit for AWS cloud onboarding)

This might require less ongoing maintenance than running an API, but it's still not trivial.

[zkamvar]

Four things:

1. there would be no need for an app here because the secret is a PAT for the control-room repository that's needed to send a repository dispatch event, not an app secret.
2. we lose control of the PAT as soon as it leaves the vault.
3. every solution that involves sharing a PAT or some sort of secret is effectively re-inventing one aspect of what an app is doing in a much less secure way.
4. there's the ongoing maintenance of rotation of the token for security.

[nickreich]

It seems like some of the conversation here is hinging on security concerns. It's not fully clear to me what the concerns are, or what the implications would be if the token was either misused or fell into "bad actor" hands. What could get borked? What is a "worst case scenario" here?

[zkamvar]

To expand a bit on this, it is difficult not because the process involved is challenging per-se, but it requires folks to be careful about the permissions of the tokens they use. In order to update workflow files, a GitHub token needs both the contents: write and workflows: write contexts.

I've done this before and it's stressful: https://github.com/carpentries/sandpaper/tree/main/inst/workflows#updates

[elray1]

This option comes closest to what I understood as the option we chose yesterday to focus on in a next phase of exploration. But I'm confused by the part of this title that says "invoked by individual hub repos via PAT". I was thinking there was a way this could be done without requiring individual hub admins to maintain a PAT for calling the re-usable workflows. This would involve placing a reduced version of the build.yaml file in each hub dashboard repo, calling the centrally-maintained re-usable workflow(s). I thought that would not require a PAT, but am I missing something?