Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Core, WebHost: lazy-load worlds in unpickler, WebHost and WebHostLib #2156

Merged
merged 7 commits into from
Sep 20, 2023
Merged

Core, WebHost: lazy-load worlds in unpickler, WebHost and WebHostLib #2156

merged 7 commits into from
Sep 20, 2023

Conversation

black-sliver
Copy link
Member

What is this fixing or adding?

this should hopefully fix customserver's memory consumption

How was this tested?

haven't tested it yet

@black-sliver
Core: lazy-load worlds in unpickler
001b29c 
this should hopefully fix customserver's memory consumption
Berserker66
Berserker66 approved these changes Sep 9, 2023
View reviewed changes
Copy link
Member

@Berserker66 Berserker66 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it should work as advertised.

@black-sliver
Copy link
Member Author

customserver still uses 128MB on my machine, so the fix is incomplete :(

@black-sliver black-sliver changed the title Core: lazy-load worlds in unpickler Core, WebHost: lazy-load worlds in unpickler, WebHost and WebHostLib Sep 9, 2023
@black-sliver
Copy link
Member Author

Current version saves ~6MB per idle process (generation queue, hosting queue)
And ~55MB per customserver instance.

The cython stuff eats ~20MB per MP. and the rest is also ~20MB more than expected (psutil reports 51MB before importing _speedups).

After investigating the cython memory overhead, I think splitting Utils and lazy-loading settings would be the next steps.

@black-sliver
MultiServer: prefer loading _speedups without pyximport
b8d64a1 
This saves ~15MB per MP and speeds up module import if it was built in-place.
@black-sliver
Tests: fix tests for changed WebHost imports
c24bbf6
@black-sliver
Copy link
Member Author

The majority of the _speedups memory allocation was due to pyximport (or its dependencies). After merging this, you can run cythonize -b -i _speedups.pyx on the server to pre-build the module and skip the pyximport. CustomServer is now down to 57MB for me (vs 73MB with pyximport vs 127MB on main).

Biggest fish is now probably Utils.py with ~10MB total and WebHostLib/__init__.py with ~16MB.

@black-sliver black-sliver merged commit d471dcc into ArchipelagoMW:main Sep 20, 2023
12 checks passed
@ThePhar ThePhar added is: refactor/cleanup Improvements to code/output readability or organizization. affects: core Issues/PRs that touch core and may need additional validation. affects: webhost Issues/PRs that touch webhost and may need additional validation. labels Oct 16, 2023
FlySniper pushed a commit to FlySniper/Archipelago that referenced this pull request Nov 14, 2023
@black-sliver @FlySniper
Core, WebHost: lazy-load worlds in unpickler, WebHost and WebHostLib (A…
5fde565 
…rchipelagoMW#2156)

* Core: lazy-load worlds in unpickler

this should hopefully fix customserver's memory consumption

* WebHost: move imports around to save memory in MP

* MultiServer: prefer loading _speedups without pyximport

This saves ~15MB per MP and speeds up module import if it was built in-place.

* Tests: fix tests for changed WebHost imports

* CustomServer: run GC after setup

* CustomServer: cleanup exception handling
@black-sliver black-sliver deleted the customserver-memory-fix branch February 23, 2024 07:41
Jouramie pushed a commit to Jouramie/Archipelago that referenced this pull request Feb 28, 2024
@black-sliver @Jouramie
Core, WebHost: lazy-load worlds in unpickler, WebHost and WebHostLib (A…
0caec84 
…rchipelagoMW#2156)

* Core: lazy-load worlds in unpickler

this should hopefully fix customserver's memory consumption

* WebHost: move imports around to save memory in MP

* MultiServer: prefer loading _speedups without pyximport

This saves ~15MB per MP and speeds up module import if it was built in-place.

* Tests: fix tests for changed WebHost imports

* CustomServer: run GC after setup

* CustomServer: cleanup exception handling
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Berserker66 Berserker66 Berserker66 approved these changes

Assignees
No one assigned
Labels
affects: core Issues/PRs that touch core and may need additional validation. affects: webhost Issues/PRs that touch webhost and may need additional validation. is: refactor/cleanup Improvements to code/output readability or organizization.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@black-sliver @Berserker66 @ThePhar