Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix readme badges and expected number of tests #1235

Merged
merged 1 commit into from
Aug 29, 2024
Merged

Fix readme badges and expected number of tests #1235

merged 1 commit into from
Aug 29, 2024

Conversation

f0uriest
Copy link
Member

Some stuff that was missing from #1213

  • Updates readme badges to point to the renamed gh actions
  • Tells codecov to expect 14 batches of coverage data, not 10 (we used to have 6 regression + 4 unit, now 6 regression + 8 unit)

@f0uriest f0uriest requested review from a team, rahulgaur104, ddudt, dpanici, kianorr, sinaatalay, unalmis and YigitElma and removed request for a team August 29, 2024 19:30
@YigitElma
Copy link
Collaborator

Oh sorry, I didn't know these exist.
Can we try to resolve #1221 here?

@YigitElma
Copy link
Collaborator

I assume it will not affect a lot of PR's but some small ones (like this one or just docs update) can be faster. The conditions can be

  • if no change in desc or requirements don't run notebook tests
  • if no change in desc, tests (without benchmarks) or requirements don't run unit and regression tests (also disable code-cov)
  • if no change in desc, tests/benchmarks or requirements don't run benchmarks

Also, I asked ChatGPT if there is a way to check changes that are made to comments (not to actual code). Here is a basic example,

name: Check Code Changes

on: [push, pull_request]

jobs:
  check_code_changes:
    runs-on: ubuntu-latest
    steps:
    - name: Check out the repository
      uses: actions/checkout@v2

    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: '3.x'

    - name: Install git diff tool (if needed)
      run: sudo apt-get install diffutils

    - name: Check if there are code changes
      run: |
        git diff HEAD^1 -- '*.py' > changes.diff
        if grep -q -v -e '^\s*#' -e '^\s*$' changes.diff; then
          echo "There are code changes!"
          exit 1
        else
          echo "No code changes detected."
          exit 0
        fi

@f0uriest
Copy link
Member Author

I'd rather fix this for now since it makes the main landing page look bad right now, we can do the testing improvements in another one.

Copy link
Contributor

github-actions bot commented Aug 29, 2024

|             benchmark_name             |         dt(%)          |         dt(s)          |        t_new(s)        |        t_old(s)        | 
| -------------------------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- |
 test_build_transform_fft_lowres         |     +0.13 +/- 3.90     | +6.61e-04 +/- 2.04e-02 |  5.25e-01 +/- 7.4e-03  |  5.24e-01 +/- 1.9e-02  |
 test_equilibrium_init_medres            |     +0.51 +/- 0.72     | +2.10e-02 +/- 2.99e-02 |  4.16e+00 +/- 2.2e-02  |  4.14e+00 +/- 2.0e-02  |
 test_equilibrium_init_highres           |     +0.45 +/- 1.42     | +2.44e-02 +/- 7.80e-02 |  5.50e+00 +/- 7.2e-02  |  5.48e+00 +/- 3.0e-02  |
 test_objective_compile_dshape_current   |     +0.92 +/- 1.25     | +3.51e-02 +/- 4.80e-02 |  3.87e+00 +/- 3.8e-02  |  3.83e+00 +/- 2.9e-02  |
 test_objective_compute_dshape_current   |     -1.24 +/- 1.36     | -4.35e-05 +/- 4.77e-05 |  3.46e-03 +/- 3.5e-05  |  3.51e-03 +/- 3.2e-05  |
 test_objective_jac_dshape_current       |     +4.06 +/- 5.15     | +1.58e-03 +/- 2.00e-03 |  4.04e-02 +/- 1.2e-03  |  3.88e-02 +/- 1.6e-03  |
 test_perturb_2                          |     +0.94 +/- 3.77     | +1.64e-01 +/- 6.56e-01 |  1.75e+01 +/- 6.3e-01  |  1.74e+01 +/- 1.8e-01  |
 test_proximal_freeb_jac                 |     -1.80 +/- 2.56     | -1.37e-01 +/- 1.95e-01 |  7.46e+00 +/- 8.4e-02  |  7.60e+00 +/- 1.8e-01  |
 test_solve_fixed_iter                   |     +0.19 +/- 60.62    | +9.54e-03 +/- 3.06e+00 |  5.06e+00 +/- 2.1e+00  |  5.05e+00 +/- 2.2e+00  |
 test_build_transform_fft_midres         |     +0.80 +/- 7.01     | +5.04e-03 +/- 4.44e-02 |  6.38e-01 +/- 3.8e-02  |  6.33e-01 +/- 2.3e-02  |
 test_build_transform_fft_highres        |     -0.50 +/- 4.81     | -5.11e-03 +/- 4.94e-02 |  1.02e+00 +/- 3.4e-02  |  1.03e+00 +/- 3.6e-02  |
 test_equilibrium_init_lowres            |     -1.26 +/- 4.69     | -5.22e-02 +/- 1.94e-01 |  4.09e+00 +/- 1.7e-01  |  4.14e+00 +/- 9.3e-02  |
 test_objective_compile_atf              |     +1.84 +/- 5.66     | +1.45e-01 +/- 4.48e-01 |  8.06e+00 +/- 3.5e-01  |  7.92e+00 +/- 2.7e-01  |
 test_objective_compute_atf              |    +10.33 +/- 5.65     | +1.06e-03 +/- 5.82e-04 |  1.14e-02 +/- 5.4e-04  |  1.03e-02 +/- 2.2e-04  |
 test_objective_jac_atf                  |     -0.07 +/- 2.13     | -1.30e-03 +/- 4.22e-02 |  1.98e+00 +/- 2.2e-02  |  1.98e+00 +/- 3.6e-02  |
 test_perturb_1                          |     +3.32 +/- 5.73     | +4.25e-01 +/- 7.33e-01 |  1.32e+01 +/- 6.7e-01  |  1.28e+01 +/- 3.0e-01  |
 test_proximal_jac_atf                   |     +0.07 +/- 1.11     | +5.56e-03 +/- 9.21e-02 |  8.29e+00 +/- 7.1e-02  |  8.29e+00 +/- 5.8e-02  |
 test_proximal_freeb_compute             |     -0.70 +/- 1.99     | -1.31e-03 +/- 3.71e-03 |  1.85e-01 +/- 1.5e-03  |  1.86e-01 +/- 3.4e-03  |

Copy link

codecov bot commented Aug 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.34%. Comparing base (416b5fb) to head (788a197).
Report is 1887 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1235   +/-   ##
=======================================
  Coverage   95.33%   95.34%           
=======================================
  Files          90       90           
  Lines       22702    22702           
=======================================
+ Hits        21643    21645    +2     
+ Misses       1059     1057    -2     

see 1 file with indirect coverage changes

@YigitElma
Copy link
Collaborator

Btw, it is not related to #1213 because even before that this was the case, but why do we have extremely high standard deviation in test_fixed_iter?

@f0uriest
Copy link
Member Author

Btw, it is not related to #1213 because even before that this was the case, but why do we have extremely high standard deviation in test_fixed_iter?

Could be due to the first step being compiled? We used to properly account for that but since #1043 it recompiles a lot less so the later ones are faster?

@f0uriest f0uriest merged commit 2ebdf6a into master Aug 29, 2024
24 checks passed
@f0uriest f0uriest deleted the rc/hotfix branch August 29, 2024 21:24
@YigitElma
Copy link
Collaborator

Btw, it is not related to #1213 because even before that this was the case, but why do we have extremely high standard deviation in test_fixed_iter?

Could be due to the first step being compiled? We used to properly account for that but since #1043 it recompiles a lot less so the later ones are faster?

Ah yes, it looks like the std dev increased after that. Nice

@YigitElma YigitElma mentioned this pull request Sep 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants