Skip to content

Commit

Permalink
Fix asv problems (#2065)
Browse files Browse the repository at this point in the history 
Fixes problems with publishing of ASV benchmarks

This PR implements the following:
- Upgrade the Python analysis flow and C++ tests to Python 3.11 - this
is needed because some of the benchmarks need at least 3.10
- Fixes the problems with publishing of the benchmarks - because they
can't evaluate np.inf correctly
- Fixes LFS for BI benchmarks
- Incorporates changes from
#2060

See successful run
[here](https://github.com/man-group/ArcticDB/actions/runs/12317667688/job/34380720877)
(had to be started manually due to the changes to the workflow files)

<details>
  <summary>
   Checklist for code changes...
  </summary>

- [ ] Have you updated the relevant docstrings, documentation and
copyright notice?
- [ ] Is this contribution tested against [all ArcticDB's
features](../docs/mkdocs/docs/technical/contributing.md)?
- [ ] Do all exceptions introduced raise appropriate [error
messages](https://docs.arcticdb.io/error_messages/)?
 - [ ] Are API changes highlighted in the PR description?
- [ ] Is the PR labelled as enhancement or bug so it appears in
autogenerated release notes?
</details>

<!--
Thanks for contributing a Pull Request to ArcticDB! Please ensure you
have taken a look at:
- ArcticDB's Code of Conduct:
https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md
- ArcticDB's Contribution Licensing:
https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing
-->

Fix CI issues

Try to override the file during benchmarks

Comment-out for testing

Flaky asv (#2066)

<!--Example: Fixes #1234. See also #3456.-->

<details>
  <summary>
   Checklist for code changes...
  </summary>

- [ ] Have you updated the relevant docstrings, documentation and
copyright notice?
- [ ] Is this contribution tested against [all ArcticDB's
features](../docs/mkdocs/docs/technical/contributing.md)?
- [ ] Do all exceptions introduced raise appropriate [error
messages](https://docs.arcticdb.io/error_messages/)?
 - [ ] Are API changes highlighted in the PR description?
- [ ] Is the PR labelled as enhancement or bug so it appears in
autogenerated release notes?
</details>

<!--
Thanks for contributing a Pull Request to ArcticDB! Please ensure you
have taken a look at:
- ArcticDB's Code of Conduct:
https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md
- ArcticDB's Contribution Licensing:
https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing
-->

---------

Co-authored-by: Georgi Rusev <Georgi Rusev>

Test with LFS checkout

Fix lfs install step

Remove unnecessary check

Fix code coverage flow

try to use ec2 runner for code cov

Test without the port

Fix for 311
  • Loading branch information
@G-D-Petrov
G-D-Petrov committed Dec 16, 2024
1 parent 9b3eed2 commit 3d14897
Show file tree
Hide file tree
Showing 9 changed files with 444 additions and 328 deletions.
2 changes: 1 addition & 1 deletion .github/actions/run_local_pytest/action.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ runs:
npm install -g azurite
cd python
ln -s ../cpp/out/linux-${{ inputs.build_type }}-build/arcticdb/arcticdb_ext.cpython-36m-x86_64-linux-gnu.so
ln -s ../cpp/out/linux-${{ inputs.build_type }}-build/arcticdb/arcticdb_ext.cpython-311m-x86_64-linux-gnu.so
export ARCTICDB_RAND_SEED=$RANDOM
python ${{inputs.other_params}} -m pytest --timeout=3600 -n ${{ inputs.threads }} tests
env:
Expand Down
6 changes: 3 additions & 3 deletions .github/actions/setup_deps/action.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ runs:
dnf update -y
dnf remove -y 'gcc-toolset-13-*'
dnf install -y zip flex bison gcc-toolset-10 gcc-toolset-10-gdb gcc-toolset-10-libatomic-devel krb5-devel cyrus-sasl-devel openssl-devel \
unzip tar epel-release jq wget libcurl-devel python3 \
python3-devel python3-pip perl-IPC-Cmd
unzip tar epel-release jq wget libcurl-devel \
python3.11-devel python3.11-pip perl-IPC-Cmd
dnf groupinstall -y 'Development Tools'
Expand All @@ -19,7 +19,7 @@ runs:
echo "CXX=/opt/rh/gcc-toolset-10/root/bin/g++" | tee -a $GITHUB_ENV
echo "CMAKE_CXX_COMPILER=/opt/rh/gcc-toolset-10/root/bin/g++" | tee -a $GITHUB_ENV
echo "LD_LIBRARY_PATH=/opt/rh/gcc-toolset-10/root/usr/lib64:/opt/rh/gcc-toolset-10/root/usr/lib:/opt/rh/gcc-toolset-10/root/usr/lib64/dyninst" | tee -a $GITHUB_ENV
echo "/opt/rh/devtoolset-10/root/usr/bin" | tee -a $GITHUB_PATH
echo "/opt/rh/devtoolset-10/root/usr/bin:/opt/python/cp311-cp311/bin" | tee -a $GITHUB_PATH
echo $GITHUB_ENV
Expand Down
353 changes: 183 additions & 170 deletions .github/workflows/analysis_workflow.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:
type: boolean
default: false

schedule: # Schdeule the job to run at 12 a.m. daily
schedule: # Schedule the job to run at 12 a.m. daily
- cron: '0 0 * * *'

pull_request_target:
Expand Down Expand Up @@ -99,175 +99,188 @@ jobs:
python -m asv publish -v
python -m asv gh-pages -v --rewrite
# code_coverage:
# runs-on: "ubuntu-22.04"
# container:
# image: quay.io/pypa/manylinux_2_28_x86_64:latest
# services:
# mongodb:
# image: mongo:4.4
# ports:
# - 27017:27017
# env:
# VCPKG_NUGET_USER: ${{secrets.VCPKG_NUGET_USER || github.repository_owner}}
# VCPKG_NUGET_TOKEN: ${{secrets.VCPKG_NUGET_TOKEN || secrets.GITHUB_TOKEN}}
# VCPKG_MAN_NUGET_USER: ${{secrets.VCPKG_MAN_NUGET_USER}} # For forks to download pre-compiled dependencies from the Man repo
# VCPKG_MAN_NUGET_TOKEN: ${{secrets.VCPKG_MAN_NUGET_TOKEN}}
# ARCTIC_CMAKE_PRESET: linux-debug
# ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
# steps:
# - uses: actions/[email protected]
# with:
# submodules: recursive
# ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || '' }} # Note: This is dangerous if we run automatic CI on external PRs

# - name: Get number of CPU cores
# uses: SimenB/[email protected]
# id: cpu-cores

# - name: Install deps
# uses: ./.github/actions/setup_deps

# - name: Extra envs
# shell: bash -l {0}
# run: |
# . build_tooling/vcpkg_caching.sh # Linux follower needs another call in CIBW
# echo -e "VCPKG_BINARY_SOURCES=$VCPKG_BINARY_SOURCES
# VCPKG_ROOT=$PLATFORM_VCPKG_ROOT" | tee -a $GITHUB_ENV
# cmake -P cpp/CMake/CpuCount.cmake | sed 's/^-- //' | tee -a $GITHUB_ENV
# echo "ARCTICDB_CODE_COVERAGE_BUILD=1" | tee -a $GITHUB_ENV
# env:
# CMAKE_BUILD_PARALLEL_LEVEL: ${{vars.CMAKE_BUILD_PARALLEL_LEVEL}}

# - name: Prepare C++ compilation env
# run: . build_tooling/prep_cpp_build.sh
start_ec2_runner_code_coverage:
uses: ./.github/workflows/ec2_runner_jobs.yml
secrets: inherit
with:
job_type: start
instance-type: t2.2xlarge

code_coverage:
needs: [start_ec2_runner_code_coverage]
if: |
always() &&
!cancelled()
runs-on: ${{ needs.start_ec2_runner_code_coverage.outputs.label }}
container:
image: quay.io/pypa/manylinux_2_28_x86_64:latest
services:
mongodb:
image: mongo:4.4
env:
VCPKG_NUGET_USER: ${{secrets.VCPKG_NUGET_USER || github.repository_owner}}
VCPKG_NUGET_TOKEN: ${{secrets.VCPKG_NUGET_TOKEN || secrets.GITHUB_TOKEN}}
VCPKG_MAN_NUGET_USER: ${{secrets.VCPKG_MAN_NUGET_USER}} # For forks to download pre-compiled dependencies from the Man repo
VCPKG_MAN_NUGET_TOKEN: ${{secrets.VCPKG_MAN_NUGET_TOKEN}}
ARCTIC_CMAKE_PRESET: linux-debug
steps:
- uses: actions/[email protected]
with:
submodules: recursive
ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || '' }} # Note: This is dangerous if we run automatic CI on external PRs

- name: Get number of CPU cores
uses: SimenB/[email protected]
id: cpu-cores

- name: Install deps
uses: ./.github/actions/setup_deps

- name: Extra envs
shell: bash -l {0}
run: |
. build_tooling/vcpkg_caching.sh # Linux follower needs another call in CIBW
echo -e "VCPKG_BINARY_SOURCES=$VCPKG_BINARY_SOURCES
VCPKG_ROOT=$PLATFORM_VCPKG_ROOT" | tee -a $GITHUB_ENV
cmake -P cpp/CMake/CpuCount.cmake | sed 's/^-- //' | tee -a $GITHUB_ENV
echo "ARCTICDB_CODE_COVERAGE_BUILD=1" | tee -a $GITHUB_ENV
env:
CMAKE_BUILD_PARALLEL_LEVEL: ${{vars.CMAKE_BUILD_PARALLEL_LEVEL}}

- name: Prepare C++ compilation env
run: . build_tooling/prep_cpp_build.sh

# - name: CMake compile
# # We are pinning the version to 10.6 because >= 10.7, use node20 which is not supported in the container
# uses: lukka/[email protected]
# with:
# cmakeListsTxtPath: ${{github.workspace}}/cpp/CMakeLists.txt
# configurePreset: ${{env.ARCTIC_CMAKE_PRESET}}
# buildPreset: ${{env.ARCTIC_CMAKE_PRESET}}
# env:
# ARCTICDB_DEBUG_FIND_PYTHON: ${{vars.ARCTICDB_DEBUG_FIND_PYTHON}}
# python_impl_name: 'cp311'
- name: CMake compile
# We are pinning the version to 10.6 because >= 10.7, use node20 which is not supported in the container
uses: lukka/[email protected]
with:
cmakeListsTxtPath: ${{github.workspace}}/cpp/CMakeLists.txt
configurePreset: ${{env.ARCTIC_CMAKE_PRESET}}
buildPreset: ${{env.ARCTIC_CMAKE_PRESET}}
env:
ARCTICDB_DEBUG_FIND_PYTHON: ${{vars.ARCTICDB_DEBUG_FIND_PYTHON}}
python_impl_name: 'cp311'

# - name: Run C++ Tests
# shell: bash -l {0}
# run: |
# cd cpp/out/linux-debug-build/
# ls arcticdb
# make -j ${{ steps.cpu-cores.outputs.count }} arcticdb_rapidcheck_tests
# make -j ${{ steps.cpu-cores.outputs.count }} test_unit_arcticdb
# ctest

# # We are chainging the python here because we want to use the default python to build (it is devel version)
# # and this python for the rest of the testing
# - name: Select Python (Linux)
# run: echo /opt/python/cp36-cp36m/bin >> $GITHUB_PATH

# - name: Install local dependencies with pip
# shell: bash
# run: |
# python -m pip install --upgrade pip
# ARCTIC_CMAKE_PRESET=skip pip install -ve .[Testing]

# # - name: Test with pytest
# # uses: ./.github/actions/run_local_pytest
# # with:
# # build_type: debug
# # threads: 1
# # fast_tests_only: 0
# # other_params: '-m coverage run '

# - name: Get python Coverage report
# shell: bash -l {0}
# run: |
# cd python
# python -m coverage report -m | tee output.txt
# python -m coverage html
# zip -r python_cov.zip htmlcov/

# echo "PYTHON_COV_PERCENT=$(cat output.txt | grep 'TOTAL' | awk '{print $NF}' | tr -d '%')" >> $GITHUB_ENV

# - name: Run Gcovr manually post-pytest
# shell: bash -l {0}
# run: |
# cd cpp/out/linux-debug-build/
# python -m pip install gcovr
# mkdir coverage
# python -m gcovr --txt --html-details coverage/index.html -e vcpkg_installed/ -e proto/ -e ../../third_party -e ../../arcticdb/util/test/ -r ../.. --exclude-throw-branches --exclude-unreachable-branches -u --exclude-function-lines | tee output.txt
# zip -r coverage.zip coverage/

# echo "CPP_COV_PERCENT=$(cat output.txt | grep 'TOTAL' | awk '{print $NF}' | tr -d '%')" >> $GITHUB_ENV

# - name: Upload Coverage
# uses: actions/[email protected]
# with:
# name: cpp-coverage-artifact
# path: cpp/out/linux-debug-build/coverage.zip

# - name: Upload Python Coverage
# uses: actions/[email protected]
# with:
# name: python-coverage-artifact
# path: python/python_cov.zip

# - name: Restore cached CPP Coverage Percentage from the previous run
# id: cache-cov-restore
# uses: actions/cache/[email protected]
# with:
# path: prev_coverage.txt
# key: coverage
- name: Run C++ Tests
shell: bash -l {0}
run: |
cd cpp/out/linux-debug-build/
ls arcticdb
make -j ${{ steps.cpu-cores.outputs.count }} arcticdb_rapidcheck_tests
make -j ${{ steps.cpu-cores.outputs.count }} test_unit_arcticdb
ctest
- name: Install local dependencies with pip
shell: bash
run: |
python -m pip install --upgrade pip
ARCTIC_CMAKE_PRESET=skip pip install -ve .[Testing]
- name: Test with pytest
uses: ./.github/actions/run_local_pytest
with:
build_type: debug
fast_tests_only: 0
other_params: '-m coverage run '

- name: Get python Coverage report
shell: bash -l {0}
run: |
cd python
python -m coverage report -m | tee output.txt
python -m coverage html
zip -r python_cov.zip htmlcov/
echo "PYTHON_COV_PERCENT=$(cat output.txt | grep 'TOTAL' | awk '{print $NF}' | tr -d '%')" >> $GITHUB_ENV
- name: Run Gcovr manually post-pytest
shell: bash -l {0}
run: |
cd cpp/out/linux-debug-build/
python -m pip install gcovr
mkdir coverage
python -m gcovr --txt --html-details coverage/index.html -e vcpkg_installed/ -e proto/ -e ../../third_party -e ../../arcticdb/util/test/ -r ../.. --exclude-throw-branches --exclude-unreachable-branches -u --exclude-function-lines | tee output.txt
zip -r coverage.zip coverage/
echo "CPP_COV_PERCENT=$(cat output.txt | grep 'TOTAL' | awk '{print $NF}' | tr -d '%')" >> $GITHUB_ENV
- name: Upload Coverage
uses: actions/[email protected]
with:
name: cpp-coverage-artifact
path: cpp/out/linux-debug-build/coverage.zip

- name: Upload Python Coverage
uses: actions/[email protected]
with:
name: python-coverage-artifact
path: python/python_cov.zip

- name: Restore cached CPP Coverage Percentage from the previous run
id: cache-cov-restore
uses: actions/cache/[email protected]
with:
path: prev_coverage.txt
key: coverage

# - name: Get and compare coverage if cache was restored
# run: |
# # if cache was restored, compare coverage
# if [ -f coverage.txt ]; then
# PREV_COVERAGE=$(cat prev_coverage.txt | cut -d' ' -f2)
# echo "Previous coverage: $PREV_COVERAGE"
# CURR_COVERAGE=${{env.CPP_COV_PERCENT}}
# echo "CPP_COV_PREV_PERCENT=$PREV_COVERAGE" >> $GITHUB_ENV
# echo "Current coverage: $CURR_COVERAGE"
# if [ $CURR_COVERAGE -gt $PREV_COVERAGE ]; then
# echo "Coverage increased"
# elif [ $CURR_COVERAGE -lt $PREV_COVERAGE ]; then
# echo "Coverage decreased"
# else
# echo "Coverage unchanged"
# fi
# fi

# - name: Save CPP Coverage Percentage to file
# run: |
# echo "Coverage: ${{ env.CPP_COV_PERCENT }}" > current_coverage.txt

# - name: Save the current CPP Coverage Percentage to the cache
# id: cache-cov-save
# uses: actions/cache/[email protected]
# with:
# path: current_coverage.txt
# key: coverage

# - name: Check percentage and send Slack notification
# if: ${{ env.CPP_COV_PREV_PERCENT && env.CPP_COV_PERCENT && env.CPP_COV_PERCENT < env.CPP_COV_PREV_PERCENT }}
# uses: slackapi/[email protected]
# with:
# # For posting a rich message using Block Kit
# payload: |
# {
# "text": "The CPP Code Coverage has been reduced",
# "blocks": [
# {
# "type": "section",
# "text": {
# "type": "mrkdwn",
# "text": "The CPP Code Coverage from the current run(${{ env.CPP_COV_PERCENT }}%) is lower the previous one(${{ env.CPP_COV_PREV_PERCENT }}%)."
# }
# }
# ]
# }
# env:
# SLACK_WEBHOOK_URL: ${{ secrets.ARCTICDB_DEV_WEBHOOK_URL }}
# SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK
- name: Get and compare coverage if cache was restored
run: |
# if cache was restored, compare coverage
if [ -f coverage.txt ]; then
PREV_COVERAGE=$(cat prev_coverage.txt | cut -d' ' -f2)
echo "Previous coverage: $PREV_COVERAGE"
CURR_COVERAGE=${{env.CPP_COV_PERCENT}}
echo "CPP_COV_PREV_PERCENT=$PREV_COVERAGE" >> $GITHUB_ENV
echo "Current coverage: $CURR_COVERAGE"
if [ $CURR_COVERAGE -gt $PREV_COVERAGE ]; then
echo "Coverage increased"
elif [ $CURR_COVERAGE -lt $PREV_COVERAGE ]; then
echo "Coverage decreased"
else
echo "Coverage unchanged"
fi
fi
- name: Save CPP Coverage Percentage to file
run: |
echo "Coverage: ${{ env.CPP_COV_PERCENT }}" > current_coverage.txt
- name: Save the current CPP Coverage Percentage to the cache
id: cache-cov-save
uses: actions/cache/[email protected]
with:
path: current_coverage.txt
key: coverage

- name: Check percentage and send Slack notification
if: ${{ env.CPP_COV_PREV_PERCENT && env.CPP_COV_PERCENT && env.CPP_COV_PERCENT < env.CPP_COV_PREV_PERCENT }}
uses: slackapi/[email protected]
with:
# For posting a rich message using Block Kit
payload: |
{
"text": "The CPP Code Coverage has been reduced",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "The CPP Code Coverage from the current run(${{ env.CPP_COV_PERCENT }}%) is lower the previous one(${{ env.CPP_COV_PREV_PERCENT }}%)."
}
}
]
}
env:
SLACK_WEBHOOK_URL: ${{ secrets.ARCTICDB_DEV_WEBHOOK_URL }}
SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK

stop_ec2_runner_code_coverage:
needs: [start_ec2_runner_code_coverage, code_coverage]
if: |
always()
uses: ./.github/workflows/ec2_runner_jobs.yml
secrets: inherit
with:
job_type: stop
label: ${{ needs.start_ec2_runner_code_coverage.outputs.label }}
ec2-instance-id: ${{ needs.start_ec2_runner_code_coverage.outputs.ec2-instance-id }}
Loading

0 comments on commit 3d14897

Please sign in to comment.