Skip to content

Commit

Permalink
Fix asv problems (#2065)
Browse files Browse the repository at this point in the history 
Fixes problems with publishing of ASV benchmarks

This PR implements the following:
- Upgrade the Python analysis flow and C++ tests to Python 3.11 - this
is needed because some of the benchmarks need at least 3.10
- Fixes the problems with publishing of the benchmarks - because they
can't evaluate np.inf correctly
- Fixes LFS for BI benchmarks
- Incorporates changes from
#2060

See successful run
[here](https://github.com/man-group/ArcticDB/actions/runs/12317667688/job/34380720877)
(had to be started manually due to the changes to the workflow files)

<details>
  <summary>
   Checklist for code changes...
  </summary>

- [ ] Have you updated the relevant docstrings, documentation and
copyright notice?
- [ ] Is this contribution tested against [all ArcticDB's
features](../docs/mkdocs/docs/technical/contributing.md)?
- [ ] Do all exceptions introduced raise appropriate [error
messages](https://docs.arcticdb.io/error_messages/)?
 - [ ] Are API changes highlighted in the PR description?
- [ ] Is the PR labelled as enhancement or bug so it appears in
autogenerated release notes?
</details>

<!--
Thanks for contributing a Pull Request to ArcticDB! Please ensure you
have taken a look at:
- ArcticDB's Code of Conduct:
https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md
- ArcticDB's Contribution Licensing:
https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing
-->

Fix CI issues

Try to override the file during benchmarks

Comment-out for testing

Flaky asv (#2066)

<!--Example: Fixes #1234. See also #3456.-->

<details>
  <summary>
   Checklist for code changes...
  </summary>

- [ ] Have you updated the relevant docstrings, documentation and
copyright notice?
- [ ] Is this contribution tested against [all ArcticDB's
features](../docs/mkdocs/docs/technical/contributing.md)?
- [ ] Do all exceptions introduced raise appropriate [error
messages](https://docs.arcticdb.io/error_messages/)?
 - [ ] Are API changes highlighted in the PR description?
- [ ] Is the PR labelled as enhancement or bug so it appears in
autogenerated release notes?
</details>

<!--
Thanks for contributing a Pull Request to ArcticDB! Please ensure you
have taken a look at:
- ArcticDB's Code of Conduct:
https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md
- ArcticDB's Contribution Licensing:
https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing
-->

---------

Co-authored-by: Georgi Rusev <Georgi Rusev>

Test with LFS checkout

Fix lfs install step

Remove unnecessary check

Fix code coverage flow

try to use ec2 runner for code cov

Test without the port

Fix for 311

test different path

Test with fast tests only
  • Loading branch information
@G-D-Petrov
G-D-Petrov committed Dec 17, 2024
1 parent 1a012f2 commit f89a479
Show file tree
Hide file tree
Showing 3 changed files with 219 additions and 179 deletions.
5 changes: 3 additions & 2 deletions .github/actions/run_local_pytest/action.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,13 @@ runs:
- name: Run local pytests
shell: bash
run: |
yum install nodejs npm -y
dnf install -y nodejs npm
npm config set strict-ssl false
npm install -g azurite
cd python
ln -s ../cpp/out/linux-${{ inputs.build_type }}-build/arcticdb/arcticdb_ext.cpython-36m-x86_64-linux-gnu.so
ls ../cpp/out/linux-${{ inputs.build_type }}-build/arcticdb
ln -s ../cpp/out/linux-${{ inputs.build_type }}-build/arcticdb/arcticdb_ext.cpython-311-x86_64-linux-gnu.so
export ARCTICDB_RAND_SEED=$RANDOM
python ${{inputs.other_params}} -m pytest --timeout=3600 -n ${{ inputs.threads }} tests
env:
Expand Down
352 changes: 183 additions & 169 deletions .github/workflows/analysis_workflow.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,175 +99,189 @@ jobs:
python -m asv publish -v
python -m asv gh-pages -v --rewrite
# code_coverage:
# runs-on: "ubuntu-22.04"
# container:
# image: quay.io/pypa/manylinux_2_28_x86_64:latest
# services:
# mongodb:
# image: mongo:4.4
# ports:
# - 27017:27017
# env:
# VCPKG_NUGET_USER: ${{secrets.VCPKG_NUGET_USER || github.repository_owner}}
# VCPKG_NUGET_TOKEN: ${{secrets.VCPKG_NUGET_TOKEN || secrets.GITHUB_TOKEN}}
# VCPKG_MAN_NUGET_USER: ${{secrets.VCPKG_MAN_NUGET_USER}} # For forks to download pre-compiled dependencies from the Man repo
# VCPKG_MAN_NUGET_TOKEN: ${{secrets.VCPKG_MAN_NUGET_TOKEN}}
# ARCTIC_CMAKE_PRESET: linux-debug
# ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
# steps:
# - uses: actions/[email protected]
# with:
# submodules: recursive
# ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || '' }} # Note: This is dangerous if we run automatic CI on external PRs

# - name: Get number of CPU cores
# uses: SimenB/[email protected]
# id: cpu-cores

# - name: Install deps
# uses: ./.github/actions/setup_deps

# - name: Extra envs
# shell: bash -l {0}
# run: |
# . build_tooling/vcpkg_caching.sh # Linux follower needs another call in CIBW
# echo -e "VCPKG_BINARY_SOURCES=$VCPKG_BINARY_SOURCES
# VCPKG_ROOT=$PLATFORM_VCPKG_ROOT" | tee -a $GITHUB_ENV
# cmake -P cpp/CMake/CpuCount.cmake | sed 's/^-- //' | tee -a $GITHUB_ENV
# echo "ARCTICDB_CODE_COVERAGE_BUILD=1" | tee -a $GITHUB_ENV
# env:
# CMAKE_BUILD_PARALLEL_LEVEL: ${{vars.CMAKE_BUILD_PARALLEL_LEVEL}}

# - name: Prepare C++ compilation env
# run: . build_tooling/prep_cpp_build.sh
start_ec2_runner_code_coverage:
uses: ./.github/workflows/ec2_runner_jobs.yml
secrets: inherit
with:
job_type: start
instance-type: t2.2xlarge

code_coverage:
needs: [start_ec2_runner_code_coverage]
if: |
always() &&
!cancelled()
runs-on: ${{ needs.start_ec2_runner_code_coverage.outputs.label }}
container:
image: quay.io/pypa/manylinux_2_28_x86_64:latest
services:
mongodb:
image: mongo:4.4
env:
VCPKG_NUGET_USER: ${{secrets.VCPKG_NUGET_USER || github.repository_owner}}
VCPKG_NUGET_TOKEN: ${{secrets.VCPKG_NUGET_TOKEN || secrets.GITHUB_TOKEN}}
VCPKG_MAN_NUGET_USER: ${{secrets.VCPKG_MAN_NUGET_USER}} # For forks to download pre-compiled dependencies from the Man repo
VCPKG_MAN_NUGET_TOKEN: ${{secrets.VCPKG_MAN_NUGET_TOKEN}}
ARCTIC_CMAKE_PRESET: linux-debug
steps:
- uses: actions/[email protected]
with:
submodules: recursive
ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || '' }} # Note: This is dangerous if we run automatic CI on external PRs

- name: Get number of CPU cores
uses: SimenB/[email protected]
id: cpu-cores

- name: Install deps
uses: ./.github/actions/setup_deps

- name: Extra envs
shell: bash -l {0}
run: |
. build_tooling/vcpkg_caching.sh # Linux follower needs another call in CIBW
echo -e "VCPKG_BINARY_SOURCES=$VCPKG_BINARY_SOURCES
VCPKG_ROOT=$PLATFORM_VCPKG_ROOT" | tee -a $GITHUB_ENV
cmake -P cpp/CMake/CpuCount.cmake | sed 's/^-- //' | tee -a $GITHUB_ENV
echo "ARCTICDB_CODE_COVERAGE_BUILD=1" | tee -a $GITHUB_ENV
env:
CMAKE_BUILD_PARALLEL_LEVEL: ${{vars.CMAKE_BUILD_PARALLEL_LEVEL}}

- name: Prepare C++ compilation env
run: . build_tooling/prep_cpp_build.sh

# - name: CMake compile
# # We are pinning the version to 10.6 because >= 10.7, use node20 which is not supported in the container
# uses: lukka/[email protected]
# with:
# cmakeListsTxtPath: ${{github.workspace}}/cpp/CMakeLists.txt
# configurePreset: ${{env.ARCTIC_CMAKE_PRESET}}
# buildPreset: ${{env.ARCTIC_CMAKE_PRESET}}
# env:
# ARCTICDB_DEBUG_FIND_PYTHON: ${{vars.ARCTICDB_DEBUG_FIND_PYTHON}}
# python_impl_name: 'cp311'
- name: CMake compile
# We are pinning the version to 10.6 because >= 10.7, use node20 which is not supported in the container
uses: lukka/[email protected]
with:
cmakeListsTxtPath: ${{github.workspace}}/cpp/CMakeLists.txt
configurePreset: ${{env.ARCTIC_CMAKE_PRESET}}
buildPreset: ${{env.ARCTIC_CMAKE_PRESET}}
env:
ARCTICDB_DEBUG_FIND_PYTHON: ${{vars.ARCTICDB_DEBUG_FIND_PYTHON}}
python_impl_name: 'cp311'

# - name: Run C++ Tests
# shell: bash -l {0}
# run: |
# cd cpp/out/linux-debug-build/
# ls arcticdb
# make -j ${{ steps.cpu-cores.outputs.count }} arcticdb_rapidcheck_tests
# make -j ${{ steps.cpu-cores.outputs.count }} test_unit_arcticdb
# ctest

# # We are chainging the python here because we want to use the default python to build (it is devel version)
# # and this python for the rest of the testing
# - name: Select Python (Linux)
# run: echo /opt/python/cp36-cp36m/bin >> $GITHUB_PATH

# - name: Install local dependencies with pip
# shell: bash
# run: |
# python -m pip install --upgrade pip
# ARCTIC_CMAKE_PRESET=skip pip install -ve .[Testing]

# # - name: Test with pytest
# # uses: ./.github/actions/run_local_pytest
# # with:
# # build_type: debug
# # threads: 1
# # fast_tests_only: 0
# # other_params: '-m coverage run '

# - name: Get python Coverage report
# shell: bash -l {0}
# run: |
# cd python
# python -m coverage report -m | tee output.txt
# python -m coverage html
# zip -r python_cov.zip htmlcov/

# echo "PYTHON_COV_PERCENT=$(cat output.txt | grep 'TOTAL' | awk '{print $NF}' | tr -d '%')" >> $GITHUB_ENV

# - name: Run Gcovr manually post-pytest
# shell: bash -l {0}
# run: |
# cd cpp/out/linux-debug-build/
# python -m pip install gcovr
# mkdir coverage
# python -m gcovr --txt --html-details coverage/index.html -e vcpkg_installed/ -e proto/ -e ../../third_party -e ../../arcticdb/util/test/ -r ../.. --exclude-throw-branches --exclude-unreachable-branches -u --exclude-function-lines | tee output.txt
# zip -r coverage.zip coverage/

# echo "CPP_COV_PERCENT=$(cat output.txt | grep 'TOTAL' | awk '{print $NF}' | tr -d '%')" >> $GITHUB_ENV

# - name: Upload Coverage
# uses: actions/[email protected]
# with:
# name: cpp-coverage-artifact
# path: cpp/out/linux-debug-build/coverage.zip

# - name: Upload Python Coverage
# uses: actions/[email protected]
# with:
# name: python-coverage-artifact
# path: python/python_cov.zip

# - name: Restore cached CPP Coverage Percentage from the previous run
# id: cache-cov-restore
# uses: actions/cache/[email protected]
# with:
# path: prev_coverage.txt
# key: coverage
- name: Run C++ Tests
shell: bash -l {0}
run: |
cd cpp/out/linux-debug-build/
ls arcticdb
make -j ${{ steps.cpu-cores.outputs.count }} arcticdb_rapidcheck_tests
make -j ${{ steps.cpu-cores.outputs.count }} test_unit_arcticdb
ctest
- name: Install local dependencies with pip
shell: bash
run: |
python -m pip install --upgrade pip
ARCTIC_CMAKE_PRESET=skip pip install -ve .[Testing]
- name: Test with pytest
uses: ./.github/actions/run_local_pytest
with:
build_type: debug
threads: auto
fast_tests_only: 1
other_params: '-m coverage run '

- name: Get python Coverage report
shell: bash -l {0}
run: |
cd python
python -m coverage report -m | tee output.txt
python -m coverage html
zip -r python_cov.zip htmlcov/
echo "PYTHON_COV_PERCENT=$(cat output.txt | grep 'TOTAL' | awk '{print $NF}' | tr -d '%')" >> $GITHUB_ENV
- name: Run Gcovr manually post-pytest
shell: bash -l {0}
run: |
cd cpp/out/linux-debug-build/
python -m pip install gcovr
mkdir coverage
python -m gcovr --txt --html-details coverage/index.html -e vcpkg_installed/ -e proto/ -e ../../third_party -e ../../arcticdb/util/test/ -r ../.. --exclude-throw-branches --exclude-unreachable-branches -u --exclude-function-lines | tee output.txt
zip -r coverage.zip coverage/
echo "CPP_COV_PERCENT=$(cat output.txt | grep 'TOTAL' | awk '{print $NF}' | tr -d '%')" >> $GITHUB_ENV
- name: Upload Coverage
uses: actions/[email protected]
with:
name: cpp-coverage-artifact
path: cpp/out/linux-debug-build/coverage.zip

- name: Upload Python Coverage
uses: actions/[email protected]
with:
name: python-coverage-artifact
path: python/python_cov.zip

- name: Restore cached CPP Coverage Percentage from the previous run
id: cache-cov-restore
uses: actions/cache/[email protected]
with:
path: prev_coverage.txt
key: coverage

# - name: Get and compare coverage if cache was restored
# run: |
# # if cache was restored, compare coverage
# if [ -f coverage.txt ]; then
# PREV_COVERAGE=$(cat prev_coverage.txt | cut -d' ' -f2)
# echo "Previous coverage: $PREV_COVERAGE"
# CURR_COVERAGE=${{env.CPP_COV_PERCENT}}
# echo "CPP_COV_PREV_PERCENT=$PREV_COVERAGE" >> $GITHUB_ENV
# echo "Current coverage: $CURR_COVERAGE"
# if [ $CURR_COVERAGE -gt $PREV_COVERAGE ]; then
# echo "Coverage increased"
# elif [ $CURR_COVERAGE -lt $PREV_COVERAGE ]; then
# echo "Coverage decreased"
# else
# echo "Coverage unchanged"
# fi
# fi

# - name: Save CPP Coverage Percentage to file
# run: |
# echo "Coverage: ${{ env.CPP_COV_PERCENT }}" > current_coverage.txt

# - name: Save the current CPP Coverage Percentage to the cache
# id: cache-cov-save
# uses: actions/cache/[email protected]
# with:
# path: current_coverage.txt
# key: coverage

# - name: Check percentage and send Slack notification
# if: ${{ env.CPP_COV_PREV_PERCENT && env.CPP_COV_PERCENT && env.CPP_COV_PERCENT < env.CPP_COV_PREV_PERCENT }}
# uses: slackapi/[email protected]
# with:
# # For posting a rich message using Block Kit
# payload: |
# {
# "text": "The CPP Code Coverage has been reduced",
# "blocks": [
# {
# "type": "section",
# "text": {
# "type": "mrkdwn",
# "text": "The CPP Code Coverage from the current run(${{ env.CPP_COV_PERCENT }}%) is lower the previous one(${{ env.CPP_COV_PREV_PERCENT }}%)."
# }
# }
# ]
# }
# env:
# SLACK_WEBHOOK_URL: ${{ secrets.ARCTICDB_DEV_WEBHOOK_URL }}
# SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK
- name: Get and compare coverage if cache was restored
run: |
# if cache was restored, compare coverage
if [ -f coverage.txt ]; then
PREV_COVERAGE=$(cat prev_coverage.txt | cut -d' ' -f2)
echo "Previous coverage: $PREV_COVERAGE"
CURR_COVERAGE=${{env.CPP_COV_PERCENT}}
echo "CPP_COV_PREV_PERCENT=$PREV_COVERAGE" >> $GITHUB_ENV
echo "Current coverage: $CURR_COVERAGE"
if [ $CURR_COVERAGE -gt $PREV_COVERAGE ]; then
echo "Coverage increased"
elif [ $CURR_COVERAGE -lt $PREV_COVERAGE ]; then
echo "Coverage decreased"
else
echo "Coverage unchanged"
fi
fi
- name: Save CPP Coverage Percentage to file
run: |
echo "Coverage: ${{ env.CPP_COV_PERCENT }}" > current_coverage.txt
- name: Save the current CPP Coverage Percentage to the cache
id: cache-cov-save
uses: actions/cache/[email protected]
with:
path: current_coverage.txt
key: coverage

- name: Check percentage and send Slack notification
if: ${{ env.CPP_COV_PREV_PERCENT && env.CPP_COV_PERCENT && env.CPP_COV_PERCENT < env.CPP_COV_PREV_PERCENT }}
uses: slackapi/[email protected]
with:
# For posting a rich message using Block Kit
payload: |
{
"text": "The CPP Code Coverage has been reduced",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "The CPP Code Coverage from the current run(${{ env.CPP_COV_PERCENT }}%) is lower the previous one(${{ env.CPP_COV_PREV_PERCENT }}%)."
}
}
]
}
env:
SLACK_WEBHOOK_URL: ${{ secrets.ARCTICDB_DEV_WEBHOOK_URL }}
SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK

stop_ec2_runner_code_coverage:
needs: [start_ec2_runner_code_coverage, code_coverage]
if: |
always()
uses: ./.github/workflows/ec2_runner_jobs.yml
secrets: inherit
with:
job_type: stop
label: ${{ needs.start_ec2_runner_code_coverage.outputs.label }}
ec2-instance-id: ${{ needs.start_ec2_runner_code_coverage.outputs.ec2-instance-id }}
Loading

0 comments on commit f89a479

Please sign in to comment.