Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI][GHA] Manylinux x86 build #27430

Merged
merged 94 commits into from
Nov 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
fd81090
init manylinux wf
mryzhov Nov 6, 2024
841eeb7
build inside
mryzhov Nov 6, 2024
44f9d49
use ubuntu 22
mryzhov Nov 6, 2024
b3a25e6
smart_ci
mryzhov Nov 6, 2024
24be9ba
command typo
mryzhov Nov 6, 2024
fad94d6
use special images
mryzhov Nov 6, 2024
e116b41
update docker tag
mryzhov Nov 6, 2024
a823f21
build command
mryzhov Nov 6, 2024
cc21c7c
Update .github/dockerfiles/ov_build/manylinux2014_x86_64/Dockerfile
mryzhov Nov 6, 2024
17a2372
revert custom manylinux
mryzhov Nov 6, 2024
43f0d7b
Merge branch 'gha/manylinux' of https://github.com/mryzhov/openvino i…
mryzhov Nov 6, 2024
0d53cc2
Merge branch 'master' into gha/manylinux
mryzhov Nov 6, 2024
6618c00
Update .github/dockerfiles/ov_build/manylinux2014_x86_64/Dockerfile
mryzhov Nov 6, 2024
fc0735d
fixed dockerfile
mryzhov Nov 6, 2024
560c2af
Merge branch 'master' into gha/manylinux
mryzhov Nov 6, 2024
7b830c3
added docker pull
mryzhov Nov 6, 2024
99b4c09
use quay.io registry
mryzhov Nov 7, 2024
2af4896
Merge branch 'gha/manylinux' of https://github.com/mryzhov/openvino i…
mryzhov Nov 7, 2024
70d579a
python 3.11
mryzhov Nov 7, 2024
61cbef4
install python reqs
mryzhov Nov 7, 2024
27d9ad1
fixed src path
mryzhov Nov 7, 2024
0080bf1
use full pathes inside docker
mryzhov Nov 7, 2024
10575e8
debug
mryzhov Nov 7, 2024
b6ec903
changed envs
mryzhov Nov 7, 2024
ba443f2
list host dir
mryzhov Nov 7, 2024
65696e8
changed work dir
mryzhov Nov 7, 2024
cafc6d5
try ${{ github.workspace }}
mryzhov Nov 7, 2024
ddf4abf
removed ninja
mryzhov Nov 7, 2024
31d5a01
set PATH
mryzhov Nov 7, 2024
44f3a6d
removed sccache
mryzhov Nov 7, 2024
69bfe4a
use internal registry
mryzhov Nov 8, 2024
5b6911b
revert repo dir
mryzhov Nov 8, 2024
fba8d3e
fixed src path
mryzhov Nov 8, 2024
7d29877
env.OPENVINO_REPO
mryzhov Nov 8, 2024
685576b
OPENVINO_REPO: ${{ github.workspace }}/openvino
mryzhov Nov 8, 2024
ffd840c
test external image
mryzhov Nov 8, 2024
360a338
revert OPENVINO_REPO
mryzhov Nov 8, 2024
4fbeb8d
debug ls
mryzhov Nov 8, 2024
d073524
debug list dir
mryzhov Nov 8, 2024
6dbb41a
use internal manylinux images
mryzhov Nov 8, 2024
8208690
fixed docker command
mryzhov Nov 8, 2024
2815187
set DOCKER_CONFIG
mryzhov Nov 8, 2024
e29ed62
Revert "set DOCKER_CONFIG"
mryzhov Nov 8, 2024
8ab60eb
set DOCKER_CONFIG
mryzhov Nov 8, 2024
4700067
Merge branch 'master' into gha/manylinux
mryzhov Nov 8, 2024
54161c5
typo
mryzhov Nov 8, 2024
fd03a65
set cc/cxx
mryzhov Nov 8, 2024
f233ba6
exclude ov install scripts
mryzhov Nov 11, 2024
2394a19
revert setting compiler
mryzhov Nov 11, 2024
c7a8b9f
move expoert math to container
mryzhov Nov 11, 2024
5daba28
test install
mryzhov Nov 11, 2024
53691f9
enabled scache
mryzhov Nov 11, 2024
94b800e
cmake install fix
mryzhov Nov 11, 2024
64ee62a
install ov build deps
mryzhov Nov 11, 2024
381585e
install python env
mryzhov Nov 11, 2024
a261c85
use python3.12 by default
mryzhov Nov 11, 2024
843581c
fixed typo in cmake install
mryzhov Nov 11, 2024
65b6791
fixed one more typo
mryzhov Nov 11, 2024
17fc2d8
changed runner pathes
mryzhov Nov 11, 2024
39e19d7
Merge branch 'master' into gha/manylinux
mryzhov Nov 11, 2024
5356cfb
update action path
mryzhov Nov 11, 2024
439a976
fixed INSTALL_DIR env
mryzhov Nov 11, 2024
9392aba
several pythons
mryzhov Nov 11, 2024
f20c683
cleanup
mryzhov Nov 11, 2024
a23a958
fixed python loop
mryzhov Nov 11, 2024
3498e10
pack ov before wheels
mryzhov Nov 11, 2024
9ef1cf0
install python-devel
mryzhov Nov 11, 2024
d7b6461
exclude 3.9
mryzhov Nov 11, 2024
9bfb787
python3-devel
mryzhov Nov 11, 2024
ade7112
DPython3_EXECUTABLE
mryzhov Nov 11, 2024
422250c
removed deps
mryzhov Nov 11, 2024
c4e8b7f
added pip cache
mryzhov Nov 11, 2024
66a7a64
removed space typo
mryzhov Nov 11, 2024
3eb7712
removed ov build deps
mryzhov Nov 12, 2024
e36a9ec
set DPython3_INCLUDE_DIR
mryzhov Nov 12, 2024
b885082
manually set python binaries and libs
mryzhov Nov 12, 2024
ddbccf6
set ie_wheel target only
mryzhov Nov 12, 2024
1f058d9
try path with env vars
mryzhov Nov 12, 2024
53af2c1
install ov deps
mryzhov Nov 12, 2024
24d2ab8
use default image
mryzhov Nov 12, 2024
a387b60
Revert "use default image"
mryzhov Nov 12, 2024
5dbba43
set condition for dev wheel
mryzhov Nov 12, 2024
f5d61eb
Revert "set condition for dev wheel"
mryzhov Nov 12, 2024
c885ca7
set openvino_dev optional
mryzhov Nov 12, 2024
1f999ab
openvino_dev_wheel install optional
mryzhov Nov 12, 2024
5bc5bc2
mount ${{ github.workspace }}
mryzhov Nov 12, 2024
cd93776
revert artifact pathes
mryzhov Nov 12, 2024
61c1e1c
fixed install wheel dir
mryzhov Nov 12, 2024
d14aadf
Merge branch 'master' into gha/manylinux
mryzhov Nov 12, 2024
274d350
Update .github/workflows/manylinux_2014.yml
mryzhov Nov 12, 2024
bf156d7
use OpenVINODeveloperPackage_DIR
mryzhov Nov 13, 2024
26ce5a8
Merge branch 'gha/manylinux' of https://github.com/mryzhov/openvino i…
mryzhov Nov 13, 2024
af0d34b
fixed src path
mryzhov Nov 13, 2024
4eac9df
removed src from mount path
mryzhov Nov 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/dockerfiles/docker_tag
Original file line number Diff line number Diff line change
@@ -1 +1 @@
pr-27384
pr-27430
20 changes: 20 additions & 0 deletions .github/dockerfiles/ov_build/manylinux2014_x86_64/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
ARG REGISTRY="quay.io"
FROM openvinogithubactions.azurecr.io/quayio/pypa/manylinux2014_x86_64

USER root

# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && /install_build_dependencies.sh

# Install sscache
ARG SCCACHE_VERSION="v0.7.5"
ENV SCCACHE_HOME="/opt/sccache" \
SCCACHE_PATH="/opt/sccache/sccache"

RUN mkdir ${SCCACHE_HOME} && cd ${SCCACHE_HOME} && \
SCCACHE_ARCHIVE="sccache-${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" && \
curl -SLO https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/${SCCACHE_ARCHIVE} && \
tar -xzf ${SCCACHE_ARCHIVE} --strip-components=1 && rm ${SCCACHE_ARCHIVE}

ENV PATH="$SCCACHE_HOME:$PATH"
42 changes: 42 additions & 0 deletions .github/dockerfiles/ov_build/ubuntu_22_04_x64_docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
ARG REGISTRY="docker.io"
FROM ${REGISTRY}/library/ubuntu:22.04

USER root

# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf

ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"

RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
gpg-agent \
tzdata \
# parallel gzip
pigz \
python3 \
python3-pip \
&& \
rm -rf /var/lib/apt/lists/*

# Install docker
RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg && \
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] \
https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
tee /etc/apt/sources.list.d/docker.list > /dev/null

RUN apt-get update && \
apt-get install -y docker-ce docker-ce-cli containerd.io

ENV DOCKER_BUILDKIT=1
191 changes: 191 additions & 0 deletions .github/workflows/manylinux_2014.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,191 @@
name: Manylinux 2014
on:
workflow_dispatch:
pull_request:
merge_group:
push:
branches:
- master
- 'releases/**'

concurrency:
# github.ref is not unique in post-commit
group: ${{ github.event_name == 'push' && github.run_id || github.ref }}-manylinux-2014
cancel-in-progress: true

permissions: read-all

env:
PIP_CACHE_PATH: /mount/caches/pip/linux

jobs:
Smart_CI:
runs-on: ubuntu-latest
outputs:
affected_components: "${{ steps.smart_ci.outputs.affected_components }}"
changed_components: "${{ steps.smart_ci.outputs.changed_components }}"
skip_workflow: "${{ steps.smart_ci.outputs.skip_workflow }}"
steps:
- name: checkout action
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
sparse-checkout: .github/actions/smart-ci

- name: Get affected components
id: smart_ci
uses: ./.github/actions/smart-ci
with:
repository: ${{ github.repository }}
pr: ${{ github.event.number }}
commit_sha: ${{ github.sha }}
ref_name: ${{ github.ref_name }}
component_pattern: "category: (.*)"
repo_token: ${{ secrets.GITHUB_TOKEN }}
skip_when_only_listed_labels_set: 'docs'
skip_when_only_listed_files_changed: '*.md,*.rst,*.png,*.jpg,*.svg'

- name: Show affected components
run: |
echo "${{ toJSON(steps.smart_ci.outputs.affected_components) }}"
shell: bash

Docker:
needs: Smart_CI
if: "!needs.smart_ci.outputs.skip_workflow"
runs-on: aks-linux-4-cores-16gb-docker-build
container:
image: openvinogithubactions.azurecr.io/docker_build:0.2
volumes:
- /mount:/mount
outputs:
images: "${{ steps.handle_docker.outputs.images }}"
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

- uses: ./.github/actions/handle_docker
id: handle_docker
with:
images: |
ov_build/ubuntu_22_04_x64_docker
ov_build/manylinux2014_x86_64
registry: 'openvinogithubactions.azurecr.io'
dockerfiles_root_dir: '.github/dockerfiles'
changed_components: ${{ needs.smart_ci.outputs.changed_components }}

Build:
needs: [Docker]
timeout-minutes: 120
defaults:
run:
shell: bash
runs-on: aks-linux-16-cores-32gb-manylinux
if: ${{ github.repository_owner == 'openvinotoolkit' }}
container:
image: ${{ fromJSON(needs.docker.outputs.images).ov_build.ubuntu_22_04_x64_docker }}
volumes:
- /mount:/mount
options: -e SCCACHE_AZURE_BLOB_CONTAINER -e SCCACHE_AZURE_CONNECTION_STRING -e DOCKER_CONFIG -v ${{ github.workspace }}:${{ github.workspace }}
env:
CMAKE_BUILD_TYPE: 'Release'
OPENVINO_REPO: ${{ github.workspace }}/src
INSTALL_DIR: ${{ github.workspace }}/install/openvino
INSTALL_WHEELS_DIR: ${{ github.workspace }}/install/wheels
BUILD_DIR: ${{ github.workspace }}/build
DOCKER_CONFIG: "/mount/.docker"
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
SCCACHE_CACHE_SIZE: 50G
SCCACHE_AZURE_KEY_PREFIX: manylinux_2014

steps:
- name: Clone OpenVINO
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
path: ${{ env.OPENVINO_REPO }}
submodules: 'true'

- name: System info
uses: ./src/.github/actions/system_info

- name: Create docker build cache
run: |
docker volume create ov_build_cache

- name: Build OpenVINO
run: |
docker run --rm \
-v ${{ env.OPENVINO_REPO }}:/work/src \
-v ov_build_cache:/work/build \
-v ${{ env.INSTALL_DIR }}:/work/install \
-e SCCACHE_AZURE_BLOB_CONTAINER \
-e SCCACHE_AZURE_CONNECTION_STRING \
-e SCCACHE_SERVER_PORT \
-e SCCACHE_IGNORE_SERVER_IO_ERROR \
-e SCCACHE_CACHE_SIZE \
-e SCCACHE_AZURE_KEY_PREFIX \
-e CMAKE_CXX_COMPILER_LAUNCHER \
-e CMAKE_C_COMPILER_LAUNCHER \
-w /work/src \
${{ fromJSON(needs.docker.outputs.images).ov_build.manylinux2014_x86_64 }} \
/bin/bash -c "
cmake -DENABLE_CPPLINT=OFF -DENABLE_NCC_STYLE=OFF -DCMAKE_VERBOSE_MAKEFILE=ON -DENABLE_PYTHON=OFF -DENABLE_WHEEL=OFF -S /work/src -B /work/build &&
cmake --build /work/build --parallel $(nproc) --config ${{ env.CMAKE_BUILD_TYPE }} &&
cmake --install /work/build --config ${{ env.CMAKE_BUILD_TYPE }} --prefix /work/install
"

- name: Pack Artifacts
run: mkdir -p ${{ env.BUILD_DIR }} && tar -cvf - * | pigz > ${{ env.BUILD_DIR }}/openvino_package.tar.gz
working-directory: ${{ env.INSTALL_DIR }}

- name: Build Python API(Python 3.9-3.13)
run: |
SUPPORTED_PYTHON_VERSIONS=("39" "310" "311" "312" "313")
for PY_VER in "${SUPPORTED_PYTHON_VERSIONS[@]}"; do
python_path=/opt/python/cp${PY_VER}-cp${PY_VER}/bin
docker run --rm \
-v ${{ env.OPENVINO_REPO }}:/work/src \
-v ${{ env.INSTALL_WHEELS_DIR }}:/work/wheels \
-v ${{ env.PIP_CACHE_PATH }}:/work/pip_cache \
-v ov_build_cache:/work/build \
-e SCCACHE_AZURE_BLOB_CONTAINER \
-e SCCACHE_AZURE_CONNECTION_STRING \
-e SCCACHE_SERVER_PORT \
-e SCCACHE_IGNORE_SERVER_IO_ERROR \
-e SCCACHE_CACHE_SIZE \
-e SCCACHE_AZURE_KEY_PREFIX \
-e CMAKE_CXX_COMPILER_LAUNCHER \
-e CMAKE_C_COMPILER_LAUNCHER \
-w /work/src \
${{ fromJSON(needs.docker.outputs.images).ov_build.manylinux2014_x86_64 }} \
/bin/bash -c "
export PATH=${python_path}:\$PATH
PIP_VER=$(python3 -c "import pip; print(pip.__version__)")
export "PIP_CACHE_DIR=/work/pip_cache/${PIP_VER}"
python3 -m pip install -r /work/src/src/bindings/python/wheel/requirements-dev.txt &&
cmake -DOpenVINODeveloperPackage_DIR=/work/build -DENABLE_PYTHON=ON -DENABLE_WHEEL=ON -S /work/src/src/bindings/python -B /work/build_py${PY_VER} &&
cmake --build /work/build_py${PY_VER} --parallel $(nproc) --target ie_wheel --config ${{ env.CMAKE_BUILD_TYPE }} &&
cmake --install /work/build_py${PY_VER} --config ${{ env.CMAKE_BUILD_TYPE }} --prefix /work/wheels --component python_wheels
"
done

#
# Upload build artifacts
#
- name: Upload openvino package
if: ${{ always() }}
uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
with:
name: openvino_package
path: ${{ env.BUILD_DIR }}/openvino_package.tar.gz
if-no-files-found: 'error'

- name: Upload openvino wheels
if: ${{ always() }}
uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
with:
name: openvino_wheels
path: ${{ env.INSTALL_WHEELS_DIR }}/wheels/*.whl
if-no-files-found: 'error'
Loading