Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resampling MVP #1495

Merged
merged 34 commits into from
May 30, 2024
Merged

Resampling MVP #1495

merged 34 commits into from
May 30, 2024

Conversation

alexowens90
Copy link
Collaborator

@alexowens90 alexowens90 commented Apr 12, 2024

Closes #1010

@alexowens90 alexowens90 marked this pull request as draft April 12, 2024 13:17
@alexowens90 alexowens90 force-pushed the enhancement/1010/resample-mvp-rebase branch from 3f06cc7 to d3808d0 Compare May 9, 2024 09:57
@alexowens90 alexowens90 force-pushed the enhancement/1010/resample-mvp-rebase branch from d3808d0 to 2e67393 Compare May 9, 2024 13:01
@alexowens90 alexowens90 changed the title WIP: Resample draft Resampling MVP May 15, 2024
@alexowens90 alexowens90 self-assigned this May 15, 2024
@alexowens90 alexowens90 added the enhancement New feature or request label May 15, 2024
@alexowens90 alexowens90 marked this pull request as ready for review May 15, 2024 10:26
cpp/arcticdb/async/tasks.hpp Show resolved Hide resolved
cpp/arcticdb/pipeline/frame_slice.hpp Outdated Show resolved Hide resolved
cpp/arcticdb/processing/clause.hpp Outdated Show resolved Hide resolved
cpp/arcticdb/processing/clause.hpp Show resolved Hide resolved
cpp/arcticdb/processing/clause.hpp Show resolved Hide resolved
cpp/arcticdb/processing/processing_unit.cpp Outdated Show resolved Hide resolved
@alexowens90 alexowens90 merged commit ac846a8 into master May 30, 2024
113 of 114 checks passed
@alexowens90 alexowens90 deleted the enhancement/1010/resample-mvp-rebase branch May 30, 2024 16:45
alexowens90 added a commit that referenced this pull request May 31, 2024
After establishing that deallocating segments was a bottleneck when
scaling the billion row challenge out to many cores, we've decided to
move to using [mimalloc
everywhere](#1577).
Using `LD_PRELOAD` with mimalloc, these optimisations further speed up
running the billion row challenge (run on a 64 core machine with
hyperthreading):
```
Cores master brc-speedup
1     76.47  61.39
2     40.10  33.79
4     18.70  16.58
8     10.11   8.68
16     6.83   6.44
32     4.78   5.17
64     5.41   5.15
```
This shows that scaling is good out to 8 cores, and drops off after
that. Logging timings shows an obvious bottleneck in `gather_entities`
within `AggregationClause::process`, which will be addressed in a
[future ticket](#1586) to
avoid conflicts with #1495.
grusev pushed a commit that referenced this pull request Nov 25, 2024
grusev pushed a commit that referenced this pull request Nov 25, 2024
After establishing that deallocating segments was a bottleneck when
scaling the billion row challenge out to many cores, we've decided to
move to using [mimalloc
everywhere](#1577).
Using `LD_PRELOAD` with mimalloc, these optimisations further speed up
running the billion row challenge (run on a 64 core machine with
hyperthreading):
```
Cores master brc-speedup
1     76.47  61.39
2     40.10  33.79
4     18.70  16.58
8     10.11   8.68
16     6.83   6.44
32     4.78   5.17
64     5.41   5.15
```
This shows that scaling is good out to 8 cores, and drops off after
that. Logging timings shows an obvious bottleneck in `gather_entities`
within `AggregationClause::process`, which will be addressed in a
[future ticket](#1586) to
avoid conflicts with #1495.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Resampling MVP
2 participants