Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use mmap for reading the input file #762

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Conversation

utkinn
Copy link

@utkinn utkinn commented Nov 24, 2024

This pull request changes how Binwalk reads the input files. Instead of reading the whole file into memory, files are mapped into memory, using the memmap2 crate. This approach improves the experience of using Binwalk with large files, especially if the file size is greater that the amount of memory on your system.

@devttys0 devttys0 self-assigned this Nov 24, 2024
@devttys0
Copy link
Collaborator

I initially wrote binwalkv3 to use memmap2, the problem I encountered was that when extracting large files the binwalk process would die and take the terminal window it was running in with it. This would not always happen, sometimes everything would work fine, but about 25% of the time the process and associated terminal would just disappear. I suspected it was something to do with the Out Of Memory killer on Linux, though fiddling with OOM settings didn't seem to make a difference.

Ultimately I decided to just read-everything-into-memory, as it was simple and I'd rather throw an "out of memory" error than have the entire process mysteriously die mid-analysis.

For reference, when I say "large files", the files in question were several gigs (specifically an Ubuntu installer ISO, which is just under 6GB), but still had plenty of room to fit into memory (24GB of RAM). After switching over to reading everything into memory these issues went away, as there was indeed enough free memory to fit the file in question. This also brought into question whether the issue with mmaping had anything to do with the OOM killer, or if it was some issue in memmap2, or perhaps user-error of some kind on my part.

I tested your PR against the same Ubuntu ISO and got the same result (process killed mid-extraction, and the terminal window it was running in closed as well).

Unfortunately this is probably hard to test/replicate, since it is going to depend a lot on the system that binwalk is being run on. Any suggestions or fixes would be welcome, as I would love for this to work. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants