jupyter: the cpu indicator doesn't work and the cpu time usage is wrong for the default Python3 kernel (Sage kernel works fine)

[williamstein]

Look at this screenshot from a clean new session using the Python 3 kernel:

There are two major bugs.

1. The time in the upper right of the cells should be around 13 seconds, not a fraction of a second. The walltime calculation for time.sleep is fine, but the cells that involve 13s of cpu intensive computation have the time completely wrong.
2. The CPU indicator should show 100% cpu usage, but it never changes at all.

NOTE: I have never noticed these bugs before because I always use Sage kernels for nontrivial computations. Both of the above are NOT issues with the Sage kernels; there the cpu indicator works fine, and the total time is also correct.

@haraldschilly I think you wrote the backend code that these rely on. Did you use a heuristic to try to find the process whose cpu/memory usage is watched, and somehow the current python3 kernel is not detected properly? Note that this is likely the most popular kernel people use, so this is a high priority bug. This is also easy to reproduce in a dev project.

For copy paste:

%%time
sum(range(10**9))

[williamstein]

I just tried using a dev project on the left and a normal production project on the right, both with the Sage kernel. The production project works, but the dev project doesn't show any cpu usage, and the timing in the upper right is completely wrong, just like with the Python3 kernel:

Anyway, I think this is really, really bad, and we either need to fix this ASAP, or hide all the functionality related to timing, so as not to provide incorrect and misleading info.

[williamstein]

Here's more motivation for making sure these functions work robustly: https://trac.sagemath.org/ticket/33637

(They are evidently pretty hard to properly implement in Sage...)

[haraldschilly]

This is strange. I don't think this is primarily caused by how the process information is collected, it's something with the frontend or some back and forth state communication. I.e. why is it saying "kernel is idle" if it is clearly not? I bet if it is in the busy state, it would show the cpu progress.

Interestingly, a slight variation does show cpu usage:

for i in range(100):
    sum(range(10**7))

Did you use a heuristic to try to find the process whose cpu/memory usage is watched.

No, there is no heuristic.

[williamstein]

why is it saying "kernel is idle" if it is clearly not?

The kernel state information is sent by the kernel. The kernel is running in the same process as the computation, so maybe in some cases CPU bound computations totally block sending update messages about state? In any case, your observation about the relation with kernel state is useful hint as to what is going on.

[haraldschilly]

well, "worst case", this is actually an upstream issue? 🤷🏻

[williamstein]

well, "worst case", this is actually an upstream issue?

I don't see it that way. Where does the cpu and memory information come from? I think you implemented this and it has nothing to do with the Jupyter kernel spec, right?

If the problem is that the kernel state says "idle" when doing cpu heavy computation for some kernels, and your code just ignores cpu usage when the state is idle, then the solution is to change your code to not ignore cpu usage when the state is idle.