Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure psutil for the process is accurate #937

Merged
merged 11 commits into from
Jun 9, 2022
Merged

Ensure psutil for the process is accurate #937

Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
473eee9
ensure pstutil for the process is accurate
echarles May 23, 2022
3812b54
maintain the process map in kernelbase
echarles May 24, 2022
add40d2
remove unneeded imports
echarles May 24, 2022
676b1b1
remove unneeded imports
echarles May 24, 2022
d08c94a
remove unneeded imports
echarles May 24, 2022
d82ad72
prune the unexisting processes
echarles Jun 6, 2022
a7b2b27
do not creazte intermediate structure to prune the proceses
echarles Jun 8, 2022
2b28ce8
correctly maange the process map
echarles Jun 8, 2022
3a9d9de
lint
echarles Jun 9, 2022
3ceabee
var annotation for the processes class field
echarles Jun 9, 2022
361bd90
use typing package for python 3.8
echarles Jun 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions ipykernel/kernelbase.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,8 @@ class Kernel(SingletonConfigurable):
# attribute to override with a GUI
eventloop = Any(None)

processes: t.Dict[str, psutil.Process] = {}

@observe("eventloop")
def _update_eventloop(self, change):
"""schedule call to eventloop from IOLoop"""
Expand Down Expand Up @@ -953,10 +955,8 @@ async def debug_request(self, stream, ident, parent):
reply_msg = self.session.send(stream, "debug_reply", reply_content, parent, ident)
self.log.debug("%s", reply_msg)

# Taken from https://github.com/jupyter-server/jupyter-resource-usage/blob/e6ec53fa69fdb6de8e878974bcff006310658408/jupyter_resource_usage/metrics.py#L16
def get_process_metric_value(self, process, name, attribute=None):
try:
# psutil.Process methods will either return...
metric_value = getattr(process, name)()
if attribute is not None: # ... a named tuple
return getattr(metric_value, attribute)
Expand All @@ -971,12 +971,22 @@ async def usage_request(self, stream, ident, parent):
reply_content = {"hostname": socket.gethostname(), "pid": os.getpid()}
current_process = psutil.Process()
all_processes = [current_process] + current_process.children(recursive=True)
process_metric_value = self.get_process_metric_value
# Ensure 1) self.processes is updated to only current subprocesses
# and 2) we reuse processes when possible (needed for accurate CPU)
self.processes = {
process.pid: self.processes.get(process.pid, process) for process in all_processes
}
reply_content["kernel_cpu"] = sum(
[process_metric_value(process, "cpu_percent", None) for process in all_processes]
[
self.get_process_metric_value(process, "cpu_percent", None)
for process in self.processes.values()
]
)
reply_content["kernel_memory"] = sum(
[process_metric_value(process, "memory_info", "rss") for process in all_processes]
[
self.get_process_metric_value(process, "memory_info", "rss")
for process in self.processes.values()
]
)
cpu_percent = psutil.cpu_percent()
# https://psutil.readthedocs.io/en/latest/index.html?highlight=cpu#psutil.cpu_percent
Expand Down