Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use initializer to ensure that all worker run worker_setup() #1919

Merged
merged 1 commit into from
Sep 1, 2023

Conversation

samtygier-stfc
Copy link
Collaborator

@samtygier-stfc samtygier-stfc commented Sep 1, 2023

Issue

Closes #1918

Description

When creating the multiprocessing pool we want to have each work run the worker_setup() function which ensure that it is started, and that it has the operation modules loaded.

For low core counts calling pool.map() usually did the job. But it does not guarantee that each task goes to a different worker. So on machines with large numbers of CPU its possible for some workers to run the task multiple times, and others to not run it at all.

Instead use the initializer which runs a task on each worker.

"If initializer is not None then each worker process will call initializer(*initargs) when it starts." --
https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool

We no longer need to pass a dummy argument to setup_worker().

Testing & Acceptance Criteria

I can't reliably trigger this issue, even on the large 60-core IDAaaS workspaces.

Best is to check that it does not make things worse. I.e. check that MI starts up and an operations such as Rotate Stack work.

Documentation

Release notes

@samtygier-stfc samtygier-stfc marked this pull request as ready for review September 1, 2023 15:33
Copy link
Collaborator

@JackEAllen JackEAllen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change seems like a reasonable solution without testing so happy to approve. Due the difficulty of trying to replicate the issue this PR is trying to fix, we should probably check this from time to time post merge to ensure we have definitely resolved the issue. We should add this as a manual test to our internet manual testing spreadsheet to check whenever we do a release for the next couple of releases..

@JackEAllen JackEAllen added this pull request to the merge queue Sep 1, 2023
Merged via the queue into main with commit 7011025 Sep 1, 2023
8 checks passed
@JackEAllen JackEAllen deleted the 1918-pool-start branch September 1, 2023 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ModuleNotFoundError: No module named 'rotate_stack'
2 participants