Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dump memory usage when listing arenas, and add summary option in heap chunks command. #1024

Merged
merged 7 commits into from
Dec 18, 2023
Merged

Dump memory usage when listing arenas, and add summary option in heap chunks command. #1024

merged 7 commits into from
Dec 18, 2023

Conversation

r12f
Copy link
Contributor

@r12f r12f commented Dec 16, 2023 •
edited
Loading

Description

This change adds 2 things:

  1. When listing the arenas, it shows the memory usage and peak memory usage
  2. It adds --summary option to heap chunks command. Instead of dumping all chunks, it dumps the summary of all the chunks, grouped by size and flags.

This helps us to get a high level of the distribution of the objects on the heap, which helps debugging memory issues.

For example, this is one of our recent memory analysis:

  1. With memory usage showing in the arena list, we immediately know the main arena is causing the problem here.

    gef➤  heap arenas
Arena(base=0x7f5b3ebd1b80, top=0x55f4c0a36d60, last_remainder=0x55f47c4147a0, next=0x7f5b08000020, mem=1274572800, mempeak=2359623680)
Arena(base=0x7f5b08000020, top=0x7f5b08051260, last_remainder=0x7f5b08017030, next=0x7f5b10000020, mem=1966080, mempeak=1966080)
...

  2. With --summary options, we immediately know that some huge chunks are the ones causing the problem, and none of them are mmap.

    gef➤  heap chunks --summary
== Chunk distribution by size ==
ChunkSize       Count           TotalSize
273520528       1               273520528
203880320       1               203880320
127695760       1               127695760
85921744        1               85921744
71068752        1               71068752
...

== Chunk distribution by flag ==
Flag            TotalCount      TotalSize
PREV_INUSE      105702          1270695520
IS_MMAPPED      0               0
NON_MAIN_ARENA  0               0

Here are the screenshots as proof.

Arena list:
image

Heap summary:
image

image

  • My code follows the code style of this project.
  • My change includes a change to the documentation, if required.
  • If my change adds new code, adequate tests have been added.
  • I have read and agree to the CONTRIBUTING document.

@github-actions GitHub Actions
Copy link

🤖 Coverage Update

  • Commit: 15b09cf
  • Current Coverage: 71.4300%
  • New Coverage: 71.4300%
  • Diff: 0.0

To this point, this PR:

  • does not include changes to tests
  • does not include changes to documentation
  • does not include forbidden words

4 similar comments
@github-actions GitHub Actions
Copy link

🤖 Coverage Update

  • Commit: 15b09cf
  • Current Coverage: 71.4300%
  • New Coverage: 71.4300%
  • Diff: 0.0

To this point, this PR:

  • does not include changes to tests
  • does not include changes to documentation
  • does not include forbidden words

@github-actions GitHub Actions
Copy link

🤖 Coverage Update

  • Commit: 15b09cf
  • Current Coverage: 71.4300%
  • New Coverage: 71.4300%
  • Diff: 0.0

To this point, this PR:

  • does not include changes to tests
  • does not include changes to documentation
  • does not include forbidden words

@github-actions GitHub Actions
Copy link

🤖 Coverage Update

  • Commit: 15b09cf
  • Current Coverage: 71.4300%
  • New Coverage: 71.4300%
  • Diff: 0.0

To this point, this PR:

  • does not include changes to tests
  • does not include changes to documentation
  • does not include forbidden words

@github-actions GitHub Actions
Copy link

🤖 Coverage Update

  • Commit: 15b09cf
  • Current Coverage: 71.4300%
  • New Coverage: 71.4300%
  • Diff: 0.0

To this point, this PR:

  • does not include changes to tests
  • does not include changes to documentation
  • does not include forbidden words

Grazfather
Grazfather requested changes Dec 16, 2023
View reviewed changes
gef.py Outdated Show resolved Hide resolved
@Grazfather
Copy link
Collaborator

Seems that you wrote this on an older version of Gef. The diff removes a lot of recent changes. Please rebase onto main and fix conflicts before we can review it.

@r12f
Copy link
Contributor Author

r12f commented Dec 16, 2023

This is super bizarre.... yea, will do that and send another one.

@r12f r12f force-pushed the user/r12f/heap-summary branch from e9e2106 to d4e3984 Compare December 16, 2023 18:47
@github-actions GitHub Actions
Copy link

🤖 Coverage Update

  • Commit: f7a2105
  • Current Coverage: 71.4511%
  • New Coverage: 71.4511%
  • Diff: 0.0

To this point, this PR:

  • does not include changes to tests
  • does not include changes to documentation
  • does not include forbidden words

@Grazfather
Copy link
Collaborator

No need for a new PR. This looks better now.

@r12f
Copy link
Contributor Author

r12f commented Dec 16, 2023

tried again after moving to latest main.

heap arenas:
image

heap chunks --summary
image

image

Grazfather
Grazfather reviewed Dec 16, 2023
View reviewed changes
Copy link
Collaborator

@Grazfather Grazfather left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good to me. Have you profiles to see if it runs a lot slower than normally?

gef.py Show resolved Hide resolved
gef.py Show resolved Hide resolved
gef.py Outdated Show resolved Hide resolved
gef.py Outdated Show resolved Hide resolved
gef.py Outdated Show resolved Hide resolved
Grazfather
Grazfather reviewed Dec 16, 2023
View reviewed changes
gef.py Outdated Show resolved Hide resolved
@r12f
Copy link
Contributor Author

r12f commented Dec 16, 2023

Looks pretty good to me. Have you profiles to see if it runs a lot slower than normally?

yea, from my experience, it actually runs faster. likely because we don't need to craft that many strings. additionally, output the chunks take long time, so we don't need to wait anymore.

@github-actions GitHub Actions
Copy link

🤖 Coverage Update

  • Commit: f7a2105
  • Current Coverage: 71.4511%
  • New Coverage: 71.4511%
  • Diff: 0.0

To this point, this PR:

  • does not include changes to tests
  • does not include changes to documentation
  • does not include forbidden words

@github-actions GitHub Actions
Copy link

🤖 Coverage Update

  • Commit: f7a2105
  • Current Coverage: 71.4511%
  • New Coverage: 71.4511%
  • Diff: 0.0

To this point, this PR:

  • does not include changes to tests
  • does not include changes to documentation
  • does not include forbidden words

@r12f
Copy link
Contributor Author

r12f commented Dec 16, 2023 •
edited
Loading

for the performance, I measure it:

  • after the change and using --summary, it takes ~34s to complete enumeration and outputting to screen.
  • before the change, it takes ~72s to complete enumeration and outputting to screen took ~47s.

@Grazfather
Copy link
Collaborator

Looks good, once approved by @hugsy we can merge, but also, since you changed the headings you should also update the screen shots. You can wait until it's approved for that just in case it's changed again.

@Grazfather Grazfather requested a review from hugsy December 16, 2023 19:28
@r12f
Copy link
Contributor Author

r12f commented Dec 16, 2023 •
edited
Loading

Sounds great! Let me update the doc and PR description.

@r12f
Copy link
Contributor Author

r12f commented Dec 16, 2023

both doc and PR descriptions are updated.

@github-actions GitHub Actions
Copy link

🤖 Coverage Update

  • Commit: f7a2105
  • Current Coverage: 71.4511%
  • New Coverage: 71.4511%
  • Diff: 0.0

To this point, this PR:

  • does not include changes to tests
  • does not include changes to documentation
  • does not include forbidden words

@github-actions GitHub Actions
Copy link

🤖 Coverage Update

  • Commit: f7a2105
  • Current Coverage: 71.4511%
  • New Coverage: 71.4511%
  • Diff: 0.0

To this point, this PR:

  • does not include changes to tests
  • does not include changes to documentation
  • does not include forbidden words

@r12f
Copy link
Contributor Author

r12f commented Dec 17, 2023

Thanks a lot, @hugsy ! Does this mean this PR can be merged? @Grazfather 🥳

@Grazfather Grazfather merged commit 17c496c into hugsy:main Dec 18, 2023
5 of 6 checks passed
@hugsy hugsy added this to the 2024.01 milestone Dec 22, 2023
@r12f r12f deleted the user/r12f/heap-summary branch January 3, 2024 01:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hugsy hugsy hugsy approved these changes

@Grazfather Grazfather Grazfather approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
2024.01
Development

Successfully merging this pull request may close these issues.
3 participants
@r12f @Grazfather @hugsy