invalid-publisher error when using reusable workflow

[janosh]

My publish step sits in a resuable workflow .github/workflows/pytest-release.yml.

When setting up trusted publishing, should the workflow name be the file in the actual repo (test.yml in my case) or the reusable workflow file being exectued (pytest-release.yml)? The docs sound like the former which throws

Error: Trusted publishing exchange failure:
Token request failed: the server refused the request for the following reasons:

• invalid-publisher: valid token, but no corresponding publisher

[webknjaz]

Hello, reusable workflows aren't fully supported on the PyPI side at the moment. The only known case when this might work is when the reusable workflows are in the same repository.
This is tracked @ pypi/warehouse#11096 (comment).

[webknjaz]

@woodruffw should we improve the hint that PyPI returns to also suggest checking whether the workflow is reusable?

[woodruffw]

@woodruffw should we improve the hint that PyPI returns to also suggest checking whether the workflow is reusable?

IMO this would be a (small) mismatch: reusable workflows are a GitHub concept not an OIDC one, while these error messages are mostly limited generic to OIDC configuration problems or errors (and not specific publishers).

We could do it, but I think my preference would be for linking the troubleshooting page everywhere instead and having users diagnose from there -- that way we don't introduce individual trusted publisher implementation details and don't give them a potential red herring 🙂

[janosh]

The error message could state that the workflow was not found in the publishing repo itself which currently is a must. Seems generic enough. Other git hosters may have the same concept of reusable workflows.

Does it make sense to wait for reusable workflow support btw? Would be a shame to have to revert legacy publishing everywhere.

[woodruffw]

Other git hosters may have the same concept of reusable workflows.

Sorry, this is a source of confusion: trusted publishers are not necessarily git-based, or even source repositories at all: they might be arbitrary CI or cloud platforms (such as Google Cloud, which we're currently working on supporting).

In general, IMO PyPI should avoid responding with recommendations that it isn't confident in (e.g. the error might be because of a reusable workflow, or it might be something completely different). This unfortunately inconveniences some users, but keeps us from throwing red herrings at others.

[janosh]

Oh, thanks for clearing that up.

Another thing I was uncertain about: Is the need for the workflow file to be in the same repo a security measure?

[woodruffw]

Another thing I was uncertain about: Is the need for the workflow file to be in the same repo a security measure?

Nope! That's purely a technical limitation of the current approach. It's tracked as an item to fix as part of the issue linked above.

[mikegerber]

I stumbled upon this, and having the reusable workflow in the same repo also causes this error, it seems.

[woodruffw]

I stumbled upon this, and having the reusable workflow in the same repo also causes this error, it seems.

Can you share your trusted publisher configuration and the repository + workflow you're attempting to use it with?

[mikegerber]

I stumbled upon this, and having the reusable workflow in the same repo also causes this error, it seems.

Can you share your trusted publisher configuration and the repository + workflow you're attempting to use it with?

I've since somewhat reverted it, but I think this was it:

Trusted publisher configuration (not sure if the second one is necessary):

This was my attempt:
https://github.com/qurator-spk/setuptools_ocrd/tree/5415329d46c78981190b37389b89ab46b97a53c2/.github/workflows

release.yml should use release-github+pypi.yml. I put secrets: inherit into release.yml, should that also go into the (supposedly) reusable workflow release-github+pypi.yml?

It works beautifully non-reusable, see current master branch if necessary. Ideally I'd like to use the workflow in a few more projects where I want to publish to GitHub + PyPI.

[mikegerber]

Could gh-action-pypi-publish give more debugging info - if possible? Eg. the values of Publisher - Repository - Workflow - Environment name that should be used in the Trusted publisher configuration?

[webknjaz]

@mikegerber v1.8.9 has more debug info

[webknjaz]

release.yml should use release-github+pypi.yml. I put secrets: inherit into release.yml, should that also go into the (supposedly) reusable workflow release-github+pypi.yml?

@mikegerber
You don't use a relative in-repo path + you reuse the workflow from whatever master points to, which will not be the same commit most of the time. Maybe that's what prevents it from working.
Try uses: ./.github/workflows/release-github+pypi.yml — this makes sure that both workflows come from the same commit.

[virtuald]

PR #186 adds a warning to tell users that reusable workflows don't work.

... why don't they work? Is there a manual workaround using "The Manual Way" referenced at https://docs.pypi.org/trusted-publishers/using-a-publisher/ ?

[webknjaz]

There's no workaround. It has to be implemented in Warehouse.

[woodruffw]

To add to what @webknjaz said: the limitations with reusable workflows are currently documented here: https://docs.pypi.org/trusted-publishers/troubleshooting/#reusable-workflows-on-github

Adding support for reusable workflows is on my list of things to do, but there's no current timeline for it.

[woodruffw]

This keeps snaring people despite documentation, unfortunately, so maybe it's time to add a check to the action for it 😅

@webknjaz How would you feel about a check similar to the current third-party PR one? In effect, "fail if the underlying action event is workflow_call."

[webknjaz]

@woodruffw maybe not fail hard but show a big explainer for this specific case? I don't remember if workflow_call is exposed as an event. I think it's the event of the calling workflow that is visible to us. Also, some people managed to get it to work when the reusable workflow is within the same repo and all the secrets are inherited, it seems. I don't want to break them. But if the upload fails, showing something that is impossible to miss would be a good idea. For some reason I thought we already had that warning in the error messages calling out the reusable workflows special case explicitly but it seems like it's not there.

[woodruffw]

Makes sense! I'll try and set aside some time to add that warning.

[janosh]

Adding support for reusable workflows is on my list of things to do, but there's no current timeline for it.

would you accept a bounty for this feature? happy to use GH Sponsors

[woodruffw]

would you accept a bounty for this feature? happy to use GH Sponsors

This is a very thoughtful offer, thank you! Unfortunately, I can't really personally take bounties for this kind of work -- OSS engineering is my full time job, and I get paid a salary for it. I might do this eventually as a side project, but to do it professionally I'll need one of my clients (or a new client) to fund it.

(I am happy to discuss that further if you'd like, but I also completely understand if that kind of funding isn't what you're interested in.)