From e6feef9827551666a8b1e01a9af76963a45de6b9 Mon Sep 17 00:00:00 2001 From: nacho Date: Mon, 12 Feb 2024 10:55:14 +0100 Subject: [PATCH 1/9] improve docs --- rust/README.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/rust/README.md b/rust/README.md index 6fe6b3d..b6b6dcc 100644 --- a/rust/README.md +++ b/rust/README.md @@ -84,7 +84,7 @@ cross: RUN --mount=$EARTHLY_RUST_CARGO_HOME_CACHE --mount=$EARTHLY_RUST_TARGET_CACHE cross build --target $TARGET --release END ``` -## COPY_OUTPUT +## +COPY_OUTPUT This function copies files out of the target cache into the image layers. Use it function when you want to `SAVE ARTIFACT` from the target folder (mounted cache), always trying to minimize the total size of the copied fileset. @@ -182,15 +182,20 @@ all: ## Mount caches and parallelization -This library uses several mount caches per tuple of `{project, os_release}`: +As of today, local Cargo builds cannot run in parallel for a given project, given that the output artifact folder is globally locked for the whole build. +This library overcomes such limitation by using different mount caches for the target folder, one per Earthly target. +While multiple concurrent builds of the same Earthly target would still block, the user now has the choice of creating new Earthly targets instead of reusing the same to increase parallelization. + +Hence, this library uses several mount caches per tuple of `{project, os_release}`: - One cache mount for `$CARGO_HOME`, shared across all target builds without any locking involved. -- A family of locked cache mounts for `$CARGO_TARGET_DIR`. One per target. +- A family of locked cache mounts for `$CARGO_TARGET_DIR`. One per Earthly target. Notice that: -- the previous targets builds might belong to one or multiple Earthly builds. +- the previous targets builds might belong to one or multiple Earthly builds, given that the caches involved are global - builds will only be blocked by concurrent ones of the same target +- Earthly target builds are atomic in the sense that the artifacts returned are guaranteed to be the ones generated by that build. For example, running `earthly +all` in the previous example will: - run all targets (`+lint,+build,+test,+fmt,+check-dependencies`) in parallel without any blocking involved - use a common cache mount for `$CARGO_HOME` -- use one individual `$CARGO_TARGET_DIR` cache mount per target +- use one individual `$CARGO_TARGET_DIR` cache mount per each target From e48408bd31b22f5b9cf532a6ae18f0ff66368393 Mon Sep 17 00:00:00 2001 From: nacho Date: Mon, 12 Feb 2024 10:57:27 +0100 Subject: [PATCH 2/9] improve docs --- rust/README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/rust/README.md b/rust/README.md index b6b6dcc..426d71c 100644 --- a/rust/README.md +++ b/rust/README.md @@ -184,18 +184,18 @@ all: As of today, local Cargo builds cannot run in parallel for a given project, given that the output artifact folder is globally locked for the whole build. This library overcomes such limitation by using different mount caches for the target folder, one per Earthly target. -While multiple concurrent builds of the same Earthly target would still block, the user now has the choice of creating new Earthly targets instead of reusing the same to increase parallelization. +While multiple concurrent builds of the same Earthly target will still block, the user now has the choice of creating new Earthly targets instead of reusing the same to increase parallelization. Hence, this library uses several mount caches per tuple of `{project, os_release}`: - One cache mount for `$CARGO_HOME`, shared across all target builds without any locking involved. - A family of locked cache mounts for `$CARGO_TARGET_DIR`. One per Earthly target. Notice that: -- the previous targets builds might belong to one or multiple Earthly builds, given that the caches involved are global -- builds will only be blocked by concurrent ones of the same target +- the previous targets builds might belong to one or multiple Earthly builds, given that the caches involved are global. +- builds will only be blocked by concurrent ones of the same target. - Earthly target builds are atomic in the sense that the artifacts returned are guaranteed to be the ones generated by that build. For example, running `earthly +all` in the previous example will: -- run all targets (`+lint,+build,+test,+fmt,+check-dependencies`) in parallel without any blocking involved -- use a common cache mount for `$CARGO_HOME` -- use one individual `$CARGO_TARGET_DIR` cache mount per each target +- run all targets (`+lint,+build,+test,+fmt,+check-dependencies`) in parallel without any blocking involved. +- use a common cache mount for `$CARGO_HOME`. +- use one individual `$CARGO_TARGET_DIR` cache mount per each target. From be68f1d2f2b309df54fe719904d13e99baac3069 Mon Sep 17 00:00:00 2001 From: nacho Date: Mon, 12 Feb 2024 16:16:28 +0100 Subject: [PATCH 3/9] fix grammar --- rust/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rust/README.md b/rust/README.md index 426d71c..ceb2757 100644 --- a/rust/README.md +++ b/rust/README.md @@ -198,4 +198,4 @@ Notice that: For example, running `earthly +all` in the previous example will: - run all targets (`+lint,+build,+test,+fmt,+check-dependencies`) in parallel without any blocking involved. - use a common cache mount for `$CARGO_HOME`. -- use one individual `$CARGO_TARGET_DIR` cache mount per each target. +- use one individual `$CARGO_TARGET_DIR` cache mount per Earthly target. From cc8ad5f04b9c0aa798a8e091de9c99ac1c2f7c12 Mon Sep 17 00:00:00 2001 From: nacho Date: Wed, 14 Feb 2024 00:14:07 +0100 Subject: [PATCH 4/9] add target suffix --- rust/Earthfile | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/rust/Earthfile b/rust/Earthfile index d690601..5fd9c50 100644 --- a/rust/Earthfile +++ b/rust/Earthfile @@ -1,6 +1,6 @@ VERSION --global-cache 0.7 - # INIT sets some configuration in the environment (used by following functions), and installs required dependencies. +# Arguments: # - cache_prefix: Overrides cache prefix for cache IDS. Its value is exported to the build environment under the entry: $EARTHLY_CACHE_PREFIX. By default ${EARTHLY_TARGET_PROJECT_NO_TAG}#${OS_RELEASE}#earthly-cargo-cache # - keep_fingerprints (false): Instructs the following +CARGO calls to don't remove the Cargo fingerprints of the source packages. Use only when source packages have been COPYed with --keep-ts option. # - sweep_days (4): +CARGO uses cargo-sweep to clean build artifacts that haven't been accessed for this number of days. @@ -17,7 +17,6 @@ INIT: ENV PATH="$PATH:$CARGO_HOME/bin" END DO +INSTALL_CARGO_SWEEP - RUN mkdir -p /tmp/earthly/cfg # $EARTHLY_CACHE_PREFIX ARG EARTHLY_TARGET_PROJECT_NO_TAG #https://docs.earthly.dev/docs/earthfile/builtin-args @@ -70,15 +69,19 @@ CARGO: # - EARTHLY_RUST_CARGO_HOME_CACHE: Code of the mount cache for the cargo home. # - EARTHLY_RUST_TARGET_CACHE: Code of the mount cache for the target folder. # Notice that in order to run this function, +INIT must be called first. +# Arguments: +# - target_cache_suffix: Optional cache suffix for the target folder cache ID. # Example: # DO rust+SET_CACHE_MOUNTS_ENV # RUN --mount=$EARTHLY_RUST_CARGO_HOME_CACHE --mount=$EARTHLY_RUST_TARGET_CACHE cargo build --release SET_CACHE_MOUNTS_ENV: COMMAND DO +CHECK_INITED + ARG target_cache_suffix + ARG TARGETPLATFORM ARG EARTHLY_TARGET_NAME #https://docs.earthly.dev/docs/earthfile/builtin-args ENV EARTHLY_RUST_CARGO_HOME_CACHE="type=cache,mode=0777,id=$EARTHLY_CACHE_PREFIX#cargo-home,sharing=shared,target=$CARGO_HOME" - ENV EARTHLY_RUST_TARGET_CACHE="type=cache,mode=0777,id=${EARTHLY_CACHE_PREFIX}#target#${EARTHLY_TARGET_NAME},sharing=locked,target=target" + ENV EARTHLY_RUST_TARGET_CACHE="type=cache,mode=0777,id=$EARTHLY_CACHE_PREFIX#target#$EARTHLY_TARGET_NAME#$TARGETPLATFORM#$target_cache_suffix,sharing=locked,target=target" # COPY_OUTPUT copies files out of the target cache into the image layers. # Use this function when you want to SAVE an ARTIFACT from the target folder (mounted cache), always trying to minimize the total size of the copied fileset. From 3990162b6bf69248a4dc27eb3a024a8d3b0bf731 Mon Sep 17 00:00:00 2001 From: nacho Date: Wed, 14 Feb 2024 00:35:07 +0100 Subject: [PATCH 5/9] add target suffix --- rust/README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/rust/README.md b/rust/README.md index ceb2757..f29c6a6 100644 --- a/rust/README.md +++ b/rust/README.md @@ -74,6 +74,11 @@ Sets the following entries in the environment, to be used to mount the cargo cac Notice that in order to run this function, [+INIT](#init) must be called first. +### Arguments + +#### `target_cache_suffix` +Optional cache suffix for the target folder cache ID. + ### Example ```earthfile From d0e4387c79ab8c2311c3addd3a154ef48a9344e5 Mon Sep 17 00:00:00 2001 From: nacho Date: Wed, 14 Feb 2024 17:44:07 +0100 Subject: [PATCH 6/9] add +CROSS and upgrade to 0.8 --- rust/Earthfile | 49 ++++++++++++++++++++++++++++++++++++++++--------- rust/README.md | 49 +++++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 81 insertions(+), 17 deletions(-) diff --git a/rust/Earthfile b/rust/Earthfile index 5fd9c50..30205a2 100644 --- a/rust/Earthfile +++ b/rust/Earthfile @@ -1,11 +1,11 @@ -VERSION --global-cache 0.7 +VERSION 0.8 # INIT sets some configuration in the environment (used by following functions), and installs required dependencies. # Arguments: # - cache_prefix: Overrides cache prefix for cache IDS. Its value is exported to the build environment under the entry: $EARTHLY_CACHE_PREFIX. By default ${EARTHLY_TARGET_PROJECT_NO_TAG}#${OS_RELEASE}#earthly-cargo-cache # - keep_fingerprints (false): Instructs the following +CARGO calls to don't remove the Cargo fingerprints of the source packages. Use only when source packages have been COPYed with --keep-ts option. # - sweep_days (4): +CARGO uses cargo-sweep to clean build artifacts that haven't been accessed for this number of days. INIT: - COMMAND + FUNCTION RUN if [ -n "$EARTHLY_CACHE_PREFIX" ]; then \ echo "+INIT has already been called in this build environment" ; \ exit 1; \ @@ -48,7 +48,7 @@ INIT: # Use this argument when you want to SAVE an ARTIFACT from the target folder (mounted cache), always trying to minimize the total size of the copied fileset. # For example --output="release/[^\./]+" would keep all the files in /target/release that don't have any extension. CARGO: - COMMAND + FUNCTION DO +CHECK_INITED ARG --required args ARG output @@ -75,7 +75,7 @@ CARGO: # DO rust+SET_CACHE_MOUNTS_ENV # RUN --mount=$EARTHLY_RUST_CARGO_HOME_CACHE --mount=$EARTHLY_RUST_TARGET_CACHE cargo build --release SET_CACHE_MOUNTS_ENV: - COMMAND + FUNCTION DO +CHECK_INITED ARG target_cache_suffix ARG TARGETPLATFORM @@ -93,7 +93,7 @@ SET_CACHE_MOUNTS_ENV: # RUN --mount=$EARTHLY_RUST_CARGO_HOME_CACHE --mount=$EARTHLY_RUST_TARGET_CACHE cargo build --release # DO rust+COPY_OUTPUT --output="release/[^\./]+" # Keep all the files in /target/release that don't have any extension. COPY_OUTPUT: - COMMAND + FUNCTION ARG --required output ARG TMP_FOLDER="/tmp/earthly/lib/rust" RUN if [ ! -n "$EARTHLY_RUST_TARGET_CACHE" ]; then \ @@ -109,7 +109,38 @@ COPY_OUTPUT: cd ..; \ fi; RUN mkdir -p target; \ - mv $TMP_FOLDER/* target 2>/dev/null || echo "no files found within ./target matching the provided output regexp" ; + mv $TMP_FOLDER/* target 2>/dev/null || echo "no files found within ./target matching the provided output regexp"; + +# CROSS runs the [cross](https://github.com/cross-rs/cross) command "cross $args --target $target". +# Notice that in order to run this function, +SET_CACHE_MOUNTS_ENV or +CARGO must be called first. +# Arguments: +# - target: Cross [target](https://github.com/cross-rs/cross?tab=readme-ov-file#supported-targets). Required. +# - args: Cross subcommand and its arguments. By default: "build --release" +# - output: Regex matching output artifacts files to be copied to ./target folder in the caller filesystem (image layers). By default: "$target/release/[^\./]+" +# Example: +# DO rust+SET_CACHE_MOUNTS_ENV +# DO rust+CROSS --target aarch64-unknown-linux-gnu + +CROSS: + FUNCTION + ARG --required target + ARG args="build --release" + ARG output="$target/release/[^\./]+" + ARG version=0.2.5 + DO +SET_CACHE_MOUNTS_ENV --target_cache_suffix="$target" + RUN echo "$EARTHLY_RUST_TARGET_CACHE" + COPY (+get-cross/cross --version=$version) /tmp/earthly/cross + WITH DOCKER --pull ghcr.io/cross-rs/$target:$version + RUN --mount=$EARTHLY_RUST_CARGO_HOME_CACHE \ + --mount=$EARTHLY_RUST_TARGET_CACHE \ + /tmp/earthly/cross $args --target=$target; + END + DO +COPY_OUTPUT --output="$output" +get-cross: + FROM alpine:3.18.3 + ARG --required version + RUN wget -nv -O- "https://github.com/cross-rs/cross/releases/download/v${version}/cross-x86_64-unknown-linux-musl.tar.gz" | tar -xzf - -C . + SAVE ARTIFACT cross get-tomljson: FROM alpine:3.18.3 @@ -129,14 +160,14 @@ get-jq: SAVE ARTIFACT jq INSTALL_CARGO_SWEEP: - COMMAND + FUNCTION RUN if [ ! -f $CARGO_HOME/bin/cargo-sweep ]; then \ echo "Installing cargo sweep" ; \ cargo install cargo-sweep --root $CARGO_HOME; \ fi; REMOVE_SOURCE_FINGERPRINTS: - COMMAND + FUNCTION DO +CHECK_INITED COPY +get-tomljson/tomljson /tmp/tomljson COPY +get-jq/jq /tmp/jq @@ -156,7 +187,7 @@ REMOVE_SOURCE_FINGERPRINTS: done; CHECK_INITED: - COMMAND + FUNCTION RUN if [ ! -n "$EARTHLY_CACHE_PREFIX" ]; then \ echo "+INIT has not been called yet in this build environment" ; \ exit 1; \ diff --git a/rust/README.md b/rust/README.md index f29c6a6..a0dd6ed 100644 --- a/rust/README.md +++ b/rust/README.md @@ -6,10 +6,9 @@ Earthly's official collection of Rust [functions](https://docs.earthly.dev/docs/ First, import the library up in your Earthfile: ```earthfile -VERSION --global-cache 0.7 +VERSION 0.8 IMPORT github.com/earthly/lib/rust: AS rust ``` -> :warning: Due to [this issue](https://github.com/earthly/earthly/issues/3490), make sure to enable `--global-cache` in the calling Earthfile, as shown above. ## +INIT @@ -82,13 +81,12 @@ Optional cache suffix for the target folder cache ID. ### Example ```earthfile -cross: +clean-target: ... DO rust+SET_CACHE_MOUNTS_ENV - WITH DOCKER - RUN --mount=$EARTHLY_RUST_CARGO_HOME_CACHE --mount=$EARTHLY_RUST_TARGET_CACHE cross build --target $TARGET --release - END + RUN --mount=$EARTHLY_RUST_TARGET_CACHE rm -rf target ``` + ## +COPY_OUTPUT This function copies files out of the target cache into the image layers. Use it function when you want to `SAVE ARTIFACT` from the target folder (mounted cache), always trying to minimize the total size of the copied fileset. @@ -105,6 +103,34 @@ DO rust+SET_RUST_CACHE_MOUNTS RUN --mount=$EARTHLY_RUST_CARGO_HOME_CACHE --mount=$EARTHLY_RUST_TARGET_CACHE cargo build --release DO rust+COPY_OUTPUT --output="release/[^\./]+" # Keep all the files in /target/release that don't have any extension. ``` +## +CROSS + +Runs the [cross](https://github.com/cross-rs/cross) command: `cross $args --target $target` . + +Notice that: +- This function makes use of `WITH DOCKER`, and hence parallelization might be tricky to achieve ([earthly#3808](https://github.com/earthly/earthly/issues/3808)). +- In order to run this function, [+INIT](#init) must be called first. + +### Arguments + +#### `target` +Cross [target](https://github.com/cross-rs/cross?tab=readme-ov-file#supported-targets). Required. + +#### `args` +Cross subcommand and its arguments. By default: `build --release` + +#### `output` +Regex matching output artifacts files to be copied to `./target` folder in the caller filesystem (image layers). By default: `$target/release/[^\./]+` + +### Example + +```earthfile +cross: + ... + DO rust+SET_CACHE_MOUNTS_ENV + DO rust+CROSS --target aarch64-unknown-linux-gnu + DO rust+COPY_OUTPUT --output="release/[^\./]+" # Keep all the files in /target/release that don't have any extension. +``` ## Complete example @@ -128,7 +154,7 @@ Suppose the following project: The Earthfile would look like: ```earthfile -VERSION --global-cache 0.7 +VERSION 0.8 # Imports the library definition from default branch (in a real case, specify version or commit to guarantee immutability) IMPORT github.com/earthly/lib/rust AS rust @@ -175,7 +201,14 @@ lint: check-dependencies: FROM +source DO rust+CARGO --args="deny --all-features check --deny warnings bans license sources" - + +# cross performs cross compilation +cross: + FROM +source + ARG --required target + DO rust+CROSS --target=$target + SAVE ARTIFACT target/$target AS LOCAL dist/$target + # all runs all other targets in parallel all: BUILD +lint From 0c99b48da2f287930fa0d19ceadcdfd88b927a1d Mon Sep 17 00:00:00 2001 From: nacho Date: Wed, 14 Feb 2024 18:06:15 +0100 Subject: [PATCH 7/9] add +CROSS and upgrade to 0.8 --- rust/Earthfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rust/Earthfile b/rust/Earthfile index 30205a2..ec9639d 100644 --- a/rust/Earthfile +++ b/rust/Earthfile @@ -112,7 +112,7 @@ COPY_OUTPUT: mv $TMP_FOLDER/* target 2>/dev/null || echo "no files found within ./target matching the provided output regexp"; # CROSS runs the [cross](https://github.com/cross-rs/cross) command "cross $args --target $target". -# Notice that in order to run this function, +SET_CACHE_MOUNTS_ENV or +CARGO must be called first. +# Notice that in order to run this function, +INIT must be called first. # Arguments: # - target: Cross [target](https://github.com/cross-rs/cross?tab=readme-ov-file#supported-targets). Required. # - args: Cross subcommand and its arguments. By default: "build --release" From 042c8d192362b22615aa96f6f62e52c5228410ad Mon Sep 17 00:00:00 2001 From: nacho Date: Wed, 14 Feb 2024 18:06:54 +0100 Subject: [PATCH 8/9] add +CROSS and upgrade to 0.8 --- rust/Earthfile | 1 + 1 file changed, 1 insertion(+) diff --git a/rust/Earthfile b/rust/Earthfile index ec9639d..f72d294 100644 --- a/rust/Earthfile +++ b/rust/Earthfile @@ -136,6 +136,7 @@ CROSS: /tmp/earthly/cross $args --target=$target; END DO +COPY_OUTPUT --output="$output" + get-cross: FROM alpine:3.18.3 ARG --required version From 996045f222e87c4dbdde0883cc2c6461a1694232 Mon Sep 17 00:00:00 2001 From: nacho Date: Wed, 14 Feb 2024 18:08:33 +0100 Subject: [PATCH 9/9] add +CROSS and upgrade to 0.8 --- rust/Earthfile | 1 - 1 file changed, 1 deletion(-) diff --git a/rust/Earthfile b/rust/Earthfile index f72d294..6b5d641 100644 --- a/rust/Earthfile +++ b/rust/Earthfile @@ -120,7 +120,6 @@ COPY_OUTPUT: # Example: # DO rust+SET_CACHE_MOUNTS_ENV # DO rust+CROSS --target aarch64-unknown-linux-gnu - CROSS: FUNCTION ARG --required target