Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ci/cache refactor and ci versions update #436

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Ci/cache refactor and ci versions update #436

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3a647bc
ci: [~] cache refactor (memory leak) & ci full upgrade
eruizalo Mar 12, 2024
31f04b5
ci: [~] replace actions/cache by coursier/cache-actions
eruizalo Mar 12, 2024
af81780
ci: [-] remove cleaning paths no longer exist
eruizalo Mar 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions .github/actions/build_and_coverage/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,17 @@ runs:
steps:

# SETUP & CACHE
- uses: coursier/[email protected]
- uses: coursier/[email protected]
id: coursier-cache
with:
jvm: adopt:8
apps: sbt scala scalafmt # https://github.com/coursier/apps
ignoreJob: 'true'
ignoreMatrix: 'true'

- uses: coursier/cache-action@v6
- uses: coursier/[email protected]
if: steps.coursier-cache.outputs.cache-hit-coursier != 'true'
with:
extraKey: "${{ env.SPARK_VERSION }}"
jvm: adopt:8
Copy link
Collaborator Author

@eruizalo eruizalo Mar 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

something is not working fine here
before --> [info] welcome to sbt 1.9.9 (AdoptOpenJDK Java 1.8.0_292)
after --> [info] welcome to sbt 1.9.9 (Eclipse Adoptium Java 11.0.22)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is using a different JVM but is it affecting something?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought cache won't affect Java, but it seems if cache works Java is not set (we are now skipping the "install" step)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could it be that you exchange the cache step with the prepare step? It's something that I see suspicious

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, it is supposed to execute cache before the install so you avoid this step, that should not be the problem. The problem is that if we match a cache we shouldn't install anything else, I mean this section apps: sbt scala scalafmt. As we don't need to install those apps because they should be provided by the cache, I ignored the step completely, but the step not only install those apps, but it set the java version as well (jvm: adopt:8). Because the whole step is now ignored if there is a cache hit, the java set is ignored as well, so I would probably add a new step only setting the java version if there is a cache hit... but it is kind of weird and I'm trying to determine if cache is needed at all, because our build times are not much better with the cache step

apps: sbt scala scalafmt # https://github.com/coursier/apps

# CHECK CODE FORMAT
- name: Code Format
Expand All @@ -32,7 +35,7 @@ runs:
- name: Test Summary
id: test_summary
if: ${{ success() && steps.test_coverage.conclusion == 'success' || failure() && steps.test_coverage.conclusion == 'failure' }}
uses: test-summary/action@v2.0
uses: test-summary/action@v2.3
with:
paths: "./core/target/test-reports/**/TEST-*.xml"
output: "test-summary.md"
Expand All @@ -54,7 +57,7 @@ runs:

- name: Upload test summary
if: ${{ github.event_name == 'pull_request' && ((success() || failure()) && steps.add_summary_link.conclusion == 'success') }}
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4.3.1
with:
name: "test-summary-${{ env.SPARK_VERSION }}.md"
path: "test-summary-${{ env.SPARK_VERSION }}.md"
Expand All @@ -63,9 +66,9 @@ runs:

- name: Publish coverage to codecov 📊
if: contains(env.SCOPE, 'uploadReport')
uses: codecov/codecov-action@v3
uses: codecov/codecov-action@v4.1.0
with:
files: ./target/scala-2.12/scoverage-report/scoverage.xml,./target/scala-2.13/scoverage-report/scoverage.xml
file: ./target/scala-*/scoverage-report/scoverage.xml
fail_ci_if_error: true
verbose: false
flags: 'spark-${{ env.SPARK_VERSION }}.x'
Expand Down
2 changes: 0 additions & 2 deletions .github/actions/clean_cache/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@ runs:
- name: Cleaning for cache 🚯
run: |
rm -rf "$HOME/.ivy2/local" || true
find $HOME/Library/Caches/Coursier/v1 -name "ivydata-*.properties" -delete || true
find $HOME/.ivy2/cache -name "ivydata-*.properties" -delete || true
find $HOME/.cache/coursier/v1 -name "ivydata-*.properties" -delete || true
find $HOME/.sbt -name "*.lock" -delete || true
shell: bash
4 changes: 0 additions & 4 deletions .github/labeler.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,6 @@ documentation:
- changed-files:
- any-glob-to-any-file: [ 'docs/**/*', 'notebooks/**/*', '*.md' ]

spark_2.4:
- changed-files:
- any-glob-to-any-file: [ 'core/src/main/spark*2.4*/**/*', 'core/src/main/scala/**/*' ]

spark_3.0:
- changed-files:
- any-glob-to-any-file: [ 'core/src/main/spark*3.0*/**/*', 'core/src/main/scala/**/*' ]
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
name: Check Dependencies
steps:
- name: PR Dependency Check
uses: gregsdennis/[email protected].2
uses: gregsdennis/[email protected].3
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Expand Down Expand Up @@ -51,7 +51,7 @@ jobs:
}" > pr_context.json

- name: Upload PR context
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v4.3.1
with:
name: "pr_context.json"
path: "pr_context.json"
Expand All @@ -69,7 +69,7 @@ jobs:
check_mdoc:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4.1.1

# BUILD MDOC, sbt should take the latest version
- name: Build the microsite 💻
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/dependency-graph.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,6 @@ jobs:
name: Update Dependency Graph
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: scalacenter/sbt-dependency-submission@v2
- uses: actions/[email protected]

- uses: scalacenter/[email protected]
2 changes: 1 addition & 1 deletion .github/workflows/labeler.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ jobs:
triage:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4.1.1

- uses: actions/labeler@v5
with:
Expand Down
17 changes: 2 additions & 15 deletions .github/workflows/pr_comment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
# GITHUB_CONTEXT: ${{ toJson(github) }}
# run: echo "$GITHUB_CONTEXT"
- name: Download PR context
uses: dawidd6/action-download-artifact@v3.0.0 # marcofaggian/[email protected]
uses: dawidd6/action-download-artifact@v3.1.2 # marcofaggian/[email protected]
with:
name: pr_context.json
workflow_conclusion: ""
Expand Down Expand Up @@ -52,19 +52,6 @@ jobs:
issue-number: ${{ fromJson(steps.output_pr_context.outputs.pr_context).pr_number }}
body-includes: "This is an auto-generated comment"

# - uses: actions/checkout@v3
#
# - name: Render template
# id: template
# uses: chuhlomin/[email protected]
# with:
# template: .github/templates/pr_comment.md
# vars: |
# workflow: ${{ github.workflow }}
# run_url: [${{ github.run_id }}](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
# attempt: ${{ github.event.workflow_run.run_attempt }}
# job_name: ${{ github.job }}

- name: Get current date
id: getDate
run: echo "::set-output name=current_date::$(date +'%Y-%m-%d %H:%M:%S %z (%Z)')"
Expand Down Expand Up @@ -133,7 +120,7 @@ jobs:
steps:
- name: Download artifact
id: download_artifacts
uses: dawidd6/action-download-artifact@v3.0.0 # marcofaggian/[email protected]
uses: dawidd6/action-download-artifact@v3.1.2 # marcofaggian/[email protected]
with:
# names: test-summary-*.md
workflow_conclusion: ""
Expand Down
16 changes: 9 additions & 7 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ jobs:
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4.1.1

- id: set-matrix
uses: JoshuaTheMiller/[email protected]
Expand All @@ -45,19 +45,21 @@ jobs:
matrix: ${{fromJson(needs.matrix_prep.outputs.matrix)}}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4.1.1

# SETUP & CACHE
- uses: coursier/[email protected]
id: coursier-cache
with:
ignoreJob: 'true'
ignoreMatrix: 'true'

- uses: coursier/[email protected]
if: steps.coursier-cache.outputs.cache-hit-coursier != 'true'
with:
jvm: adopt:8
apps: sbt scala scalafmt # https://github.com/coursier/apps

- name: Coursier cache
uses: coursier/cache-action@v6
with:
extraKey: "${{ matrix.spark }}"

# MAVEN RELEASE
- name: Maven release 🎁
run: sbt ci-release -DsparkVersion=${{ matrix.spark }}
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/wc_build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4.1.1

- id: set-matrix
uses: JoshuaTheMiller/[email protected]
Expand All @@ -27,7 +27,7 @@ jobs:
matrix: ${{fromJson(needs.matrix_prep.outputs.matrix)}}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4.1.1

- uses: ./.github/actions/build_and_coverage
env:
Expand Down
21 changes: 12 additions & 9 deletions .github/workflows/wc_publishSite.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4.1.1

- id: set-matrix
uses: JoshuaTheMiller/[email protected]
Expand All @@ -29,18 +29,21 @@ jobs:
matrix: ${{fromJson(needs.matrix_prep.outputs.matrix)}}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4.1.1

# SETUP & CACHE
- uses: coursier/[email protected]
id: coursier-cache
with:
ignoreJob: 'true'
ignoreMatrix: 'true'

- uses: coursier/[email protected]
if: steps.coursier-cache.outputs.cache-hit-coursier != 'true'
with:
jvm: adopt:8
apps: sbt scala scalafmt # https://github.com/coursier/apps

- uses: coursier/cache-action@v6
with:
extraKey: "${{ matrix.spark }}"

# BUILD SCALADOCS (API)
- name: Build the scaladocs 💻
run: sbt -DsparkVersion=${{ matrix.spark }} +core/doc
Expand All @@ -61,7 +64,7 @@ jobs:

- name: Upload docs ⬆️
if: steps.prepare_scaladocs.conclusion == 'success'
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v4.3.1
with:
name: "spark-${{ matrix.spark }}"
path: "spark-${{ matrix.spark }}"
Expand All @@ -79,7 +82,7 @@ jobs:
runs-on: ubuntu-latest
steps:

- uses: actions/checkout@v4
- uses: actions/checkout@v4.1.1

# BUILD MDOC, sbt should take the latest version
- name: Build the microsite 💻
Expand All @@ -88,7 +91,7 @@ jobs:

# DOWNLOAD DOCS, this will download all the artifacts because no name provided
- name: Download docs ⬇️
uses: actions/download-artifact@v4
uses: actions/download-artifact@v4.1.4
with:
path: "docs/target/mdoc/docs/api/"

Expand Down
Loading