Ensure psutil for the process is accurate

[echarles]

Fixes #912

This add to the control_thread a map of processes to allow the requesting the cpu_usage in a way psutil excepts (the cpu is only correct if is requested on the same process object).

[echarles]

For example, using https://github.com/Quansight/jupyterlab-kernel-usage returns the correct value. (see issue Quansight/jupyterlab-kernel-usage#11), e.g. CPU 99.9

[echarles]

self.processes = {process.pid: self.processes.get(process.pid, process) for process in all_processes}

I have used that block code and the CPU does not return the correct value.

...or should I also remove https://github.com/ipython/ipykernel/pull/937/files#diff-874f94d86eaa767fb34865311dd8fd8c0d9a2fa6c49bc21f28690058f3dfa83cR961 as you said ? (I am not sure to get the correct link, it points to just one line).

[mlucool]

This works for me:

import subprocess
from time import sleep

import psutil


class Foo:
    processes = {}

    def update(self):
        current_process = psutil.Process()
        all_processes = [current_process] + current_process.children(recursive=True)
        self.processes = {
            process.pid: self.processes.get(process.pid, process)
            for process in all_processes
        }

    def print_cpu(self):
        print("-----")
        self.update()
        for p, v in self.processes.items():
            print(f"{p}: {v.cpu_percent()}")


def work():
    y = 0
    for x in range(10):
        y += x

    return y


f = Foo()
f.print_cpu()
work()
f.print_cpu()
# Let's use CPU
p = subprocess.Popen(
    ["dd", "if=/dev/zero", "of=/dev/null"],
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
)
f.print_cpu()
sleep(0.1)
work()
f.print_cpu()
p.kill()
p.wait()
work()
sleep(2)
f.print_cpu()  # Less work

Output:

-----
24567: 0.0
-----
24567: 87.9
-----
24567: 65.0
127672: 0.0
-----
24567: 23.8
127672: 95.1
-----
24567: 1.5

Basically, if we are going to prune on every run, I think we should prune + keep the list up to date to limit the number of times we need to iterate these datastructures.

[echarles]

@mlucool I have retested and now it work as expected on my env. Thx for the suggestion and better implementation. I have committed those changes.

Anything else before asking a review?

[blink1073]

Docs failure is unrelated, but the pre-commit ones are.

[echarles]

I have fixed 2 format issues and added a type to the new processes class field. pre-commit is now happy, but the typed class field generates other errors, eg on https://github.com/ipython/ipykernel/runs/6805383473?check_suite_focus=true

    processes: dict[str, psutil.Process] = {}
E   TypeError: 'type' object is not subscriptable

[blink1073]

Yeah you need to use typings.Dict before Python 3.8.

[echarles]

@blink1073 CI is now green.

[blink1073]

Thanks!

[blink1073]

I'll cut a new release early next week.

[echarles]

@blink1073 👍

I'll cut a new release early next week.

Super!