Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize downloadMedia #490

Closed

Conversation

brunoferreiras
Copy link
Contributor

I had the same problem as described on #384

This pull request introduces a significant optimization in a specific function of the code. I replaced the usage of io.ReadAll with io.Copy to improve memory efficiency.

In my tests, I used a document of 10Mb using the go tool pprof to get this.

Before this change:
image

After this change:
image

This change aims to optimize memory consumption when processing large amounts of data. Additionally, I included examples of using go tool pprof to demonstrate performance improvements.

@tulir
Copy link
Owner

tulir commented Nov 16, 2023

That doesn't make any sense. You need 10mb of memory to download a 10mb file into memory, so my first guess would be that the second pprof is hiding the actual memory usage somewhere else. bytes.Buffer also shouldn't be any different from io.ReadAll, both of them have a byte slice as a buffer and expand it dynamically as they read data.

A potential small optimization would be pre-allocating the correct size of byte array and using io.ReadFull, rather than having bytes.Buffer or io.ReadAll dynamically grow a byte slice.

@tulir tulir closed this Nov 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants