-
Notifications
You must be signed in to change notification settings - Fork 155
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix garbage collection #1119
Fix garbage collection #1119
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1119 +/- ##
==========================================
- Coverage 85.95% 83.92% -2.04%
==========================================
Files 36 36
Lines 5747 5823 +76
==========================================
- Hits 4940 4887 -53
- Misses 807 936 +129
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Benchmark changes
Comparison: https://github.com/scverse/anndata/compare/b2bdd7f926d54c9ae7a1b56a4e97d37e6e4d1dad..e138fc99fa4d5ab7e9351a45b905e13a4fc2415f More details: https://github.com/scverse/anndata/pull/1119/checks?check_run_id=27745934862 |
Two big outstanding problems:
AnnData.layers
|
Looking at this further: https://github.com/scverse/anndata/pull/1119/checks?check_run_id=25779040354 there is actually no improvement over |
4cf6ea9
to
9e08151
Compare
benchmarks/benchmarks/anndata.py
Outdated
class GarbargeCollectionSuite: | ||
runs = 10 | ||
|
||
# https://github.com/pythonprofilers/memory_profiler/issues/402 and other backend does not pick this up |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok @ivirshup @flying-sheep independent of this issue, were you aware that https://pypi.org/project/memory-profiler/ is a line-by-line profiler? I think this is probably not very good for us, no? When I read get_peakmem
(i.e., the function in utils
we wrote based on memory_profiler
), I would think that it would really track the literal peak memory, and not the peak over individual line operations.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@flying-sheep I would be ready to go on this |
I would still like the things in here addressed: #1119 (review) I fixed everything that I didn’t talk about there, but
|
TODO: try using a |
Hm, maybe this is just a complex problem and can’t be simplified much more without making other parts more complex. The issue here is the complexity of the paths that data takes:
here my argument applies: shouldn’t there be a source of truth where that
¹Classes like |
Co-authored-by: Isaac Virshup <[email protected]>
Still needs some tests.
Demo
Set up
Benchmarking script
Memory usage on main:
Memory usage on this branch: