Automate signed pkg build for macOS App Store submission

[danryu]

This PR adds automation to create a signed pkg (installer) file for direct submission to the macOS App Store, and then submits that signed installer to Apple App Store Connect (macOS Testflight) to allow it to be selected for App Store release.

CHANGELOG: Build: adds macOS signed pkg build automation

Context: automates building of signed pkg file for macOS App Store

Does this change need documentation? What needs to be documented and how?

Required:

1. In Apple Developer Account, create the following resources in in https://developer.apple.com/account/resources/certificates/list

• Certificates:

  • Mac Installer Distribution
  • Mac App Distribution

• Identifier:

  • app ID (bundleID)

2. Add the certs to Github Secrets as per https://docs.github.com/en/actions/deployment/deploying-xcode-applications/installing-an-apple-certificate-on-macos-runners-for-xcode-development

Status of this Pull Request

What is missing until this pull request can be merged?

Checklist

• I've verified that this Pull Request follows the general code principles
• I tested my code and it does what I want
• My code follows the style guide
• I waited some time after this Pull Request was opened and all GitHub checks completed without errors.
• I've filled all the content above

[ann0see]

Looks very interesting ;-). I think only @emlynmac can really comment here.

[emlynmac]

I think we need to make this work along side the ad-hoc signing.
The changes here will break the existing signing set up.
Ad hoc requires:

• Signing
• Notarizing
• Stapling the package when notarization complete

App Store distribution requires

• Signing (with a different certificate)
• Packaging
• Installer signing (with an installer certificate)
• Validation
• Upload

I'd like to see the App Store target work along side the existing ad-hoc signing steps.

[danryu]

@emlynmac @ann0see I've updated to reflect your comments.
Let me know what you think of the cert naming so far.
Now doing some testing and looking at validate+upload with altool.

[ann0see]

Good to hear!
@emlynmac Should test it on his repo

[danryu]

Good to hear! @emlynmac Should test it on his repo

Yes sure, and from the build checks above it looks like it works transparently when signing deps are not satisfied.

As I noted in the iOS PR #2625 the build now attempts to validate and upload to App Store using altool - and may fail when eg re-attempting the same version upload.

[ann0see]

@emlynmac any news?

[danryu]

Updated with the necessary logic to validate and upload the signed macOS "pkg" installer to the Mac App Store when the necessary conditions are met (I thought I had already done this!)

[ann0see]

Probably it's close to get in. It of course raises the questions when/if we deploy a cert in our repo.

[danryu]

Probably it's close to get in. It of course raises the questions when/if we deploy a cert in our repo.

The cert stuff is obviously most easily handled by whoever already has the App Store Connect account for Jamulus. I don't think the certs are any more sensitive than the existing auth details you are already using for dmg file Notarization (appstore-connect-username,appstore-connect-password ).

[ann0see]

Fair point. Emlyn owns the cert for now and not many people have push access to his repo.

[danryu]

When/if he gets round to it Emlyn will have to create the 2 new certificates and follow the guides as mentioned in the description. Apple doesn't make this easy enough IMO - too much downloading certificates, importing to keychain manager, exporting as p12 then base64 encoding - for each cert. It's a pain.

[ann0see]

@danryu could you please rebase this PR?

I think fixing the app store alongside signing isn't an unsolvable issue (check if app store cert is present --> if not skip modifications)

[danryu]

@ann0see Yes, can do. I'll need a few days at least to review things.

[ann0see]

@danryu any updates?

[danryu]

@danryu any updates?

Sorry, hit a very busy period. I managed to take a look today but the rebase appeared problematic, so I've gone with a merge - hope that's ok. Looks like there might be some style check failures still, at least.

[ann0see]

Ok. Thanks.

Merging makes it a lot more difficult to get right later - but I hope we can figure it out. Fixing the styling should be possible

[danryu]

Coding style checks now fixed

[ann0see]

Great! Thanks. Maybe this gets ready for 3.11.0 (next release, not this one)

[ann0see]

Maybe you're on the wrong local branch? Check what commits git log show.

[softins]

Force push works with git push --force?

Not to my branch, no.

jamulus % git push koord-live macos_sign --force
Everything up-to-date

I haven't attempted this kind of force push before, is this not expected?

It's because your koord-live/koord-app-compat fork of jamulussoftware/jamulus is not up to date with jamulus itself. For example, you still have the master branch, dated 10 Oct 2022. The master branch has been deprecated since then, and replaced with main, which I can't see in your fork.

I see you have a copy of my macos-sign-squash branch in your repo. That does therefore include all the commits up to the recent main, but your repo doesn't know that they are.

I'm happy to help, but that probably needs to be done interactively. What messaging method do you prefer? Do you have Discord? What time zone are you in?

[danryu]

I see you have a copy of my macos-sign-squash branch in your repo. That does therefore include all the commits up to the recent main, but your repo doesn't know that they are.

Thanks, I think that the divergence between branches makes this a bit too much overhead to be worthwhile. That was a temporary branch at the time and I'd rather just dump it at this point.

I just have a single commit to make - if you can give me permissions to push to your branch softins:macos-sign-squash then I can update there.
I know we'll need to re-open #3309 - sorry, I thought this was gonna be easier.

[ann0see]

Just copy and paste the changes of the commit here.

You can use a git patch https://devconnected.com/how-to-create-and-apply-git-patch-files/

[danryu]

Thanks, I couldn't get git format-patch to produce anything, so I used git diff directly.

logic_separate.patch

[danryu]

@softins I also pushed the update to https://github.com/koord-live/koord-app-compat/tree/appstore2 if you just want to pull that?

FWIW, what steps would be required to update my fork? I pushed the upstream main branch but that didn't help. Just curious :)

[softins]

@danryu thanks! I have a busy day today (UK time), but will hopefully get to it later. I'll also answer your question about updating too.

[softins]

@danryu I linked your repo as an additional remote, and I found your differ branch which was based off my macos-sign-squash with just your recent changes for optional app store submission. Those changes look good, thanks! I've updated my branch to include that commit too.

So we can either merge my branch via #3309, or your could reset your macos_sign branch to point to differ and force push it to this PR. To do that, you would do this:

# switch to the macos_sign branch
git checkout macos_sign
# keep a pointer to the old branch for reference (does not switch to it)
git branch macos_sign_old
# reset the macos_sign branch pointer to point to the commit at the head of differ
git reset --hard differ
# update the remote github branch with force
git push --force

I see you now have the up to date main branch now too. Excellent!

[danryu]

@softins Thanks, I went with your instructions (I hadn't thought to do the git reset previously!)

Unfortunately the macOS build breaks at the dmg creation stage - possibly related to this (intermittent?) error?
actions/runner-images#7522

PS I didn't make any changes to this part, so I think my changes are working correctly (thus far)

[softins]

@softins Thanks, I went with your instructions (I hadn't thought to do the git reset previously!)

Great, all looks good to me.

Unfortunately the macOS build breaks at the dmg creation stage - possibly related to this (intermittent?) error? actions/runner-images#7522

Yes, hdiutil: create failed - Resource busy crops up from time to time. Re-running the workflows normally works. I'll try that.

PS I didn't make any changes to this part, so I think my changes are working correctly (thus far)

Cool. I'll set this PR as "Ready for review" instead of "Draft" and will close my #3309 in favour of this one.

[softins]

Yes, hdiutil: create failed - Resource busy crops up from time to time. Re-running the workflows normally works. I'll try that.

Yes, re-running succeeded for all platforms.

[softins]

I think this is ok now. Let's get it merged and deal with anything that might occur when we try signing and/or submitting.

[softins]

This PR doesn't appear to allow pushes from project maintainers (I tried), so we rely on @danryu applying suggested changes.

[danryu]

Thanks, I've applied the changes as suggested (I think the for loop was a leftover from an earlier iteration, so it was good to remove that!)
Hopefully we're all done now :)

[ann0see]

Before us merging, please squash the commits into one: e.g via git rebase -i. Then select to squash all commits in the editor.

[pljones]

I'd like all the "Outdated" but still not "Resolved" comments marked resolved, if they have been, too, please.

Changes were done as requested

[ann0see]

Ok. Squash merged. Not ideal, but the PR ID shows up.

Thanks all for waiting this long!

[ann0see]

Note: For the certificate IDs, check security find-identity -v to get the correct hash.

[ann0see]

Concerning the altool upload to app store connect, I didn't look too much into it/tested it, but I belive we could have something like this as starting point:

upload_pkg_to_appstore_connect() {
    # Upload pkg build to App Store connect. See https://help.apple.com/asc/appsaltool/#/apdATD1E53-D1E1A1303-D1E53A1126 for more information
    # Validate if pkg meets minimum App Store requirements
    xcrun altool --validate-app -f "${ARTIFACT_PATH}" \
        -u "${NOTARIZATION_USERNAME}" \
        -p "${NOTARIZATION_PASSWORD}"
    
    # Upload binary package to App Store connect
    xcrun altool --upload-package "${ARTIFACT_PATH}" \
        --apple-id "${NOTARIZATION_USERNAME}" \
        --bundle-id "${NOTARIZATION_BUNDLE_ID}" \
        --bundle-short-version-string "${JAMULUS_BUILD_VERSION}" \
        --bundle-version "${JAMULUS_BUILD_VERSION}" \
        --team-id "${APPLE_TEAM_ID}" \
        -u "${NOTARIZATION_USERNAME}" \
        -p "${NOTARIZATION_PASSWORD}"

}