executor: fail early on reattach if listener is not executor #24538
+16
−2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mismithhisler
added
backport/ent/1.7.x+ent
tgross
approved these changes
Nov 22, 2024
drivers/shared/executor/utils.go
Outdated
Comment on lines
78
to
81
| // ReattachToExecutor launches a plugin with a given plugin config and validates it can call the executor. | ||
| // On Windows, go-plugin listens on a localhost port. It is possible on a reboot that another process is | ||
| // listening on that port, and a process is running with the previous executors PID, leading Nomad to kill | ||
| // "random" processes. Fail early via the Version RPC if we detect the listener isn't actually an Executor. |
There was a problem hiding this comment.
Great addition to the comment. Were you able to to track down where the kill was being emitted downstream? It might be nice to mention that here, as I know both of us spent a good bit of time trying to find it.
There was a problem hiding this comment.
In my reproduction I noticed the downstream call happening here in the taskRunner after a failed RPC happened. I will update the comment to add this context.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes a bug reported by Windows users where Nomad was killing random processes on reboot. This was due to the reattach config containing a PID and RPC address that was now occupied by other processes/listeners. When Nomad would attempt to make an RPC, it would fail, and kill the process. With this change, we fail early on reattach, and do not kill the process.
Testing & Reproduction steps
Reproduction involves simulating this scenario via overriding the reattach PID when starting a task to a known PID (another bash session), rebooting Nomad, killing the executor process, and listening on the reattach address.
Links
Fixes GH #23969
Contributor Checklist
changelog entry using the
make clcommand.ensure regressions will be caught.
and job configuration, please update the Nomad website documentation to reflect this. Refer to
the website README for docs guidelines. Please also consider whether the
change requires notes within the upgrade guide.
Reviewer Checklist
backporting document.
in the majority of situations. The main exceptions are long-lived feature branches or merges where
history should be preserved.
within the public repository.