Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Docker build #8527

Merged
merged 6 commits into from
Apr 16, 2024
Merged

Refactor Docker build #8527

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
f4993dd
chore(docker): Replace `--file` with `--output` for Syft
mnonnenmacher Apr 16, 2024
e680d9b
refactor(docker): Add ORT to the image at the end of the build
mnonnenmacher Apr 16, 2024
8a0e761
ci(github): Add a composite action to free disk space
mnonnenmacher Apr 16, 2024
a570776
ci(github): Rewrite the Docker build job
mnonnenmacher Apr 11, 2024
cd4e8b0
ci(github): Run functional tests in up-to-date Docker container
mnonnenmacher Apr 15, 2024
d8cd014
ci(github): Remove the unused `ortdocker` action
mnonnenmacher Apr 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions .github/actions/free-disk-space/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Copyright (C) 2024 The ORT Project Authors (see <https://github.com/oss-review-toolkit/ort/blob/main/NOTICE>)
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SPDX-License-Identifier: Apache-2.0
# License-Filename: LICENSE

name: "Free Disk Space"
description: "Remove unneeded preinstalled Docker images and software to free disk space."
author: "The ORT Project Authors"

runs:
using: "composite"

steps:
- name: Print Disk Space
shell: bash
run: df -h
- name: List Docker Images
if: ${{ false }} # Can be enabled if the 'Remove Unneeded Docker Images' step below needs to be updated.
shell: bash
run: docker images
- name: Remove Unneeded Docker Images
shell: bash
run: |
docker image rm \
node:16 \
node:16-alpine \
node:18 \
node:18-alpine \
node:20 \
node:20-alpine \
debian:10 \
debian:11 \
ubuntu:20.04 \
ubuntu:22.04
- name: Print Disk Space
shell: bash
run: df -h
- name: Get Size of Installed Tools
if: ${{ false }} # Can be enabled if the 'Remove Unneeded Tools' step below needs to be updated.
shell: bash
run: |
sudo du -hsc /usr/lib/*
sudo du -hsc /usr/local/*
sudo du -hsc /usr/local/lib/*
sudo du -hsc /usr/local/share/*
sudo du -hsc /usr/share/*
- name: Remove Unneeded Tools
shell: bash
run: |
sudo rm -rf /usr/local/lib/android
sudo rm -rf /usr/local/lib/node_modules
sudo rm -rf /usr/local/share/chromium
sudo rm -rf /usr/local/share/powershell
sudo rm -rf /usr/share/az_11.3.1
sudo rm -rf /usr/share/dotnet
sudo rm -rf /usr/share/kotlinc
sudo rm -rf /usr/share/mecab
sudo rm -rf /usr/share/miniconda
sudo rm -rf /usr/share/ri
sudo rm -rf /usr/share/sbt
sudo rm -rf /usr/share/swift
- name: Print Disk Space
shell: bash
run: df -h
123 changes: 0 additions & 123 deletions .github/actions/ortdocker/action.yml
View file
Open in desktop

This file was deleted.

63 changes: 0 additions & 63 deletions .github/actions/ortdocker/check_image.py
View file
Open in desktop

This file was deleted.

38 changes: 33 additions & 5 deletions .github/workflows/build-and-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ on:

env:
GRADLE_OPTS: -Dorg.gradle.daemon=false -Dkotest.assertions.multi-line-diff=unified
REGISTRY: ghcr.io
mnonnenmacher marked this conversation as resolved.
Show resolved Hide resolved
TEST_IMAGE_TAG: ort:test
sschuberth marked this conversation as resolved.
Show resolved Hide resolved

jobs:
build:
Expand Down Expand Up @@ -116,19 +118,45 @@ jobs:
flags: funTest-non-docker
funTest-docker:
runs-on: ubuntu-22.04
container:
image: ghcr.io/oss-review-toolkit/ort:snapshot
options: --user 1001
steps:
- name: Checkout Repository
uses: actions/checkout@v4
with:
submodules: recursive
- name: Run functional tests that do require external tools
- name: Free Disk Space
uses: ./.github/actions/free-disk-space
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build ORT Docker Image
uses: docker/build-push-action@v5
with:
context: .
load: true
tags: ${{ env.TEST_IMAGE_TAG }}
target: all-tools
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ github.repository_owner }}/ort:cache
- name: Setup Gradle
uses: gradle/actions/setup-gradle@v3
with:
gradle-home-cache-cleanup: true
arguments: --scan -Ptests.include=org.ossreviewtoolkit.plugins.packagemanagers.* funTest jacocoFunTestReport
- name: Run functional tests that do require external tools
run: |
# Change the ownership of the Gradle user home and the workspace to the user in the Docker container.
sudo chown -R 1000:1000 /home/runner/.gradle
sudo chown -R 1000:1000 ${{ github.workspace }}
Comment on lines +144 to +146
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of this (and the restoring code below), would it be an alternative to use docker run --user 1001:121 ...?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And probably use $(id -u) / $(id -g) instead of hard-coding the values.

Copy link
Member Author

@mnonnenmacher mnonnenmacher Apr 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote this part of the code several weeks ago, I don't remember exactly which options I tried, but I think I could not get it to work with the --user option (I think one tricky part was also that the Gradle cache works correctly). I would rather do any such changes in a follow-up PR, as currently I'm just happy that it works, and doing the changes in this PR might require a few additional rounds of testing.


# Run the functional tests in the Docker container.
docker run \
-v ${{ github.workspace }}:/workspace \
-v /home/runner/.gradle:/home/ort/.gradle \
-w /workspace \
--entrypoint=/bin/sh \
${{ env.TEST_IMAGE_TAG }} \
-c "GRADLE_USER_HOME=/home/ort/.gradle ./gradlew --scan -Ptests.include=org.ossreviewtoolkit.plugins.packagemanagers.* funTest jacocoFunTestReport"

# Change the ownership of the Gradle user home and the workspace back to the user in the GitHub Actions runner.
sudo chown -R 1001:121 /home/runner/.gradle
sudo chown -R 1001:121 ${{ github.workspace }}
- name: Upload code coverage data
uses: codecov/codecov-action@v4
with:
Expand Down
Loading
Loading