Rework release CI/CD for efficiency #409
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Brooks Townsend <[email protected]> added Cross.toml Signed-off-by: Brooks Townsend <[email protected]> corrected the artifact name Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]> debug check paths Signed-off-by: Brooks Townsend <[email protected]> dll not exe Signed-off-by: Brooks Townsend <[email protected]> WINDOWSS Signed-off-by: Brooks Townsend <[email protected]> giving a full build a try Signed-off-by: Brooks Townsend <[email protected]> renamed needs sections Signed-off-by: Brooks Townsend <[email protected]> fix small mv error, test docker build too Signed-off-by: Brooks Townsend <[email protected]> correct windows target, copy into dockerfile Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]> redid some paths Signed-off-by: Brooks Townsend <[email protected]> added built Signed-off-by: Brooks Townsend <[email protected]> updated mix env to release_prod Signed-off-by: Brooks Townsend <[email protected]> added release prod file Signed-off-by: Brooks Townsend <[email protected]> removed executable for windows, ls for debugging Signed-off-by: Brooks Townsend <[email protected]> MIX ENVIRONMENT WINDOWS GAH Signed-off-by: Brooks Townsend <[email protected]> corrected docker path Signed-off-by: Brooks Townsend <[email protected]> see if anything else is broken Signed-off-by: Brooks Townsend <[email protected]> compile esbuild in release_prod, correct mix env in docker Signed-off-by: Brooks Townsend <[email protected]> added file debug Signed-off-by: Brooks Townsend <[email protected]> updated deps, use macos-11 Signed-off-by: Brooks Townsend <[email protected]> use release prod for docker Signed-off-by: Brooks Townsend <[email protected]> create the directory first just in case Signed-off-by: Brooks Townsend <[email protected]> create dir Signed-off-by: Brooks Townsend <[email protected]> install specific otp/elixir version Signed-off-by: Brooks Townsend <[email protected]> updated dart saas, create RELEASES Signed-off-by: Brooks Townsend <[email protected]> remove unused targets for Rust Signed-off-by: Brooks Townsend <[email protected]> reintroduce aarch64 linux Signed-off-by: Brooks Townsend <[email protected]> don't tar up the release Signed-off-by: Brooks Townsend <[email protected]> upload correct file Signed-off-by: Brooks Townsend <[email protected]> updated distillery Signed-off-by: Brooks Townsend <[email protected]> updated makefiles Signed-off-by: Brooks Townsend <[email protected]> corrected filename Signed-off-by: Brooks Townsend <[email protected]>
brooksmtownsend
force-pushed
the
ci/precompiled-nifs
branch
from
9cb8b28 to
35013f9
Compare
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
Signed-off-by: Brooks Townsend <[email protected]>
brooksmtownsend
requested review from
autodidaddict,
thomastaylor312 and
stevelr
as code owners
Signed-off-by: Brooks Townsend <[email protected]>
thomastaylor312
approved these changes
Jun 8, 2022
| target: | ||
| - x86_64-unknown-linux-gnu | ||
| - aarch64-unknown-linux-gnu | ||
| - x86_64-apple-darwin |
There was a problem hiding this comment.
You can build for an m1 on an x86_mac. Should we have a separate step that runs-on mac and builds the nif for m1 macs?
There was a problem hiding this comment.
NOTE: I don't know if we could build the rest of the elixir stuff as well, but I know we can at least get the nif
There was a problem hiding this comment.
I can definitely build the NIF for m1 mac, even with cross here but I don't have a way to get the erlang runtime system for the m1 mac (I tested and verified that x86-macos.tar.gz that we normally publish doesn't work even with rosetta but that was a quick test)
There was a problem hiding this comment.
We could possibly get the elixir stuff by running the erlang docker image with --platform linux/arm64/v8 and downloading the erts folder... but it's a stretch and would require rearranging all of this to not package the erts with the release
(This is something I really want to do, but I'm not sure if it's best done now since this PR has been in progress for a while)
There was a problem hiding this comment.
Yeah no need to block on this. It is just a nit
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR reworks the release CI/CD for our host_core and wasmcloud_host releases to use precompiled NIFs while still preserving the development experience of using a local cargo install. This is under the
release_prodprofile to avoid confusion. Additionally, as you can see in a mostly successful run on https://github.com/wasmCloud/wasmcloud-otp/actions/runs/2457366553, this cuts the total release time down to around 40 minutes. From tag push to releasing new tarballs is around 15 minutes, and tag push to new docker images (yes, even the aarch64 one) is around 30 minutes.I'm blocked here as there is a specific combination of OTP/Elixir versions that we're forced to use with homebrew (currently investigating ways to manually select a homebrew version) that Distillery isn't quite updated for yet. See issue here: bitwalker/distillery#744. For now, I'm resorting to using ASDF on macos to install erlang/elixir specific versions as my homebrew tap hack didn't end up working.I started this PR with the notion that this would be a good way to upgrade to the self-extracting tarball that distillery offers, but it isn't supported on Windows, and with the general unmaintained feel of the distillery package I would suggest we move to mix release, the officially reported release mechanism, instead. mix release also doesn't support self-extracting tarballs, which is unfortunate, but at the very least changing to the executable tarball format now would require a breaking change if we moved to mix releases.