-
Notifications
You must be signed in to change notification settings - Fork 703
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Speedups: remove dependency on c++ #2796
Speedups: remove dependency on c++ #2796
Conversation
941d835
to
809ad6a
Compare
Here are some numbers for the case where len() is way bigger than 1023 from my machine:
I'm not sure why std::set is that much slower to be honest, but that's just there for reference. So, the 1023 set is "half as fast", but it uses only ~230KB RAM compared to ~800KB for the "full" one. |
I played around with |
I ran ubsan on my local test code and nothing popped up. A thing to note is that the struct with the variable sized array is not valid C++, but it is valid C and supported by extensions to C++ in both GCC and clang (and probably msvc, but I haven't checked). I am still confused about the memset not working for Either way, I'd still like @beauxq to give feedback on the changed realloc before hitting merge. |
yeah, |
original idea was to only use bucket->val if int<pointer, but we always have a union now anyway
This kind of tests that the build succeeds.
…rnings Unnamed unions are not in C99, this got fixed. The overhead of setting count=0 is minimal or optimized-out and silences -fanalizer (see comment).
This won't happen in our cython, but it's still a good addition.
We solve this with env vars in ctest now, and this fails for msvc.
INTxx_MIN % 1 would not produce a negative number, so the test was flawed.
cde9836
to
19109cc
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was hoping to find time to really dig into this one, but I think RC testing and unittests might have to do. Code looks fine at a glance.
* Speedups: remove dependency on c++ * Speedups: intset: handle malloc failing * Speedups: intset: fix corner case for int64 on 32bit systems original idea was to only use bucket->val if int<pointer, but we always have a union now anyway * Speedups: add size comment to player_set bucket configuration * test: more tests for LocationStore.find_item * test: require _speedups in CI This kind of tests that the build succeeds. * test: even more tests for LocationStore.find_item * Speedups: intset uniform comment style * Speedups: intset: avoid memory leak when realloc fails * Speedups: intset: make `gcc -pedantic -std=c99 -fanalyzer` without warnings Unnamed unions are not in C99, this got fixed. The overhead of setting count=0 is minimal or optimized-out and silences -fanalizer (see comment). * Speedups: don't leak memory in case of exception * Speedups: intset: validate alloc and free This won't happen in our cython, but it's still a good addition. * CI: add test framework for C/C++ code * CI: ctest: fix cwd * Speedups: intset: ignore msvc warning * Tests: intset: revert attempt at no-asan We solve this with env vars in ctest now, and this fails for msvc. * Test: cpp: docs: fix typo * Test: cpp: docs: fix another typo * Test: intset: proper bucket count for Negative test INTxx_MIN % 1 would not produce a negative number, so the test was flawed.
* Speedups: remove dependency on c++ * Speedups: intset: handle malloc failing * Speedups: intset: fix corner case for int64 on 32bit systems original idea was to only use bucket->val if int<pointer, but we always have a union now anyway * Speedups: add size comment to player_set bucket configuration * test: more tests for LocationStore.find_item * test: require _speedups in CI This kind of tests that the build succeeds. * test: even more tests for LocationStore.find_item * Speedups: intset uniform comment style * Speedups: intset: avoid memory leak when realloc fails * Speedups: intset: make `gcc -pedantic -std=c99 -fanalyzer` without warnings Unnamed unions are not in C99, this got fixed. The overhead of setting count=0 is minimal or optimized-out and silences -fanalizer (see comment). * Speedups: don't leak memory in case of exception * Speedups: intset: validate alloc and free This won't happen in our cython, but it's still a good addition. * CI: add test framework for C/C++ code * CI: ctest: fix cwd * Speedups: intset: ignore msvc warning * Tests: intset: revert attempt at no-asan We solve this with env vars in ctest now, and this fails for msvc. * Test: cpp: docs: fix typo * Test: cpp: docs: fix another typo * Test: intset: proper bucket count for Negative test INTxx_MIN % 1 would not produce a negative number, so the test was flawed.
* Speedups: remove dependency on c++ * Speedups: intset: handle malloc failing * Speedups: intset: fix corner case for int64 on 32bit systems original idea was to only use bucket->val if int<pointer, but we always have a union now anyway * Speedups: add size comment to player_set bucket configuration * test: more tests for LocationStore.find_item * test: require _speedups in CI This kind of tests that the build succeeds. * test: even more tests for LocationStore.find_item * Speedups: intset uniform comment style * Speedups: intset: avoid memory leak when realloc fails * Speedups: intset: make `gcc -pedantic -std=c99 -fanalyzer` without warnings Unnamed unions are not in C99, this got fixed. The overhead of setting count=0 is minimal or optimized-out and silences -fanalizer (see comment). * Speedups: don't leak memory in case of exception * Speedups: intset: validate alloc and free This won't happen in our cython, but it's still a good addition. * CI: add test framework for C/C++ code * CI: ctest: fix cwd * Speedups: intset: ignore msvc warning * Tests: intset: revert attempt at no-asan We solve this with env vars in ctest now, and this fails for msvc. * Test: cpp: docs: fix typo * Test: cpp: docs: fix another typo * Test: intset: proper bucket count for Negative test INTxx_MIN % 1 would not produce a negative number, so the test was flawed.
* Speedups: remove dependency on c++ * Speedups: intset: handle malloc failing * Speedups: intset: fix corner case for int64 on 32bit systems original idea was to only use bucket->val if int<pointer, but we always have a union now anyway * Speedups: add size comment to player_set bucket configuration * test: more tests for LocationStore.find_item * test: require _speedups in CI This kind of tests that the build succeeds. * test: even more tests for LocationStore.find_item * Speedups: intset uniform comment style * Speedups: intset: avoid memory leak when realloc fails * Speedups: intset: make `gcc -pedantic -std=c99 -fanalyzer` without warnings Unnamed unions are not in C99, this got fixed. The overhead of setting count=0 is minimal or optimized-out and silences -fanalizer (see comment). * Speedups: don't leak memory in case of exception * Speedups: intset: validate alloc and free This won't happen in our cython, but it's still a good addition. * CI: add test framework for C/C++ code * CI: ctest: fix cwd * Speedups: intset: ignore msvc warning * Tests: intset: revert attempt at no-asan We solve this with env vars in ctest now, and this fails for msvc. * Test: cpp: docs: fix typo * Test: cpp: docs: fix another typo * Test: intset: proper bucket count for Negative test INTxx_MIN % 1 would not produce a negative number, so the test was flawed.
What is this fixing or adding?
Removes dependency on C++ for _speedups.
But Why? C++ adds extra dependencies that need to be loaded into the process and afaik only _speedup requires them at the moment. Also currently we also don't handle C++ exceptions that could occur. This change fixes both and saves some memory because it should always have less memory overhead.
How was this tested?
Only pytest and a few benchmarks and tests on C level. I wanted to push this so we can discuss.