Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

per-block cpu and db usage metrics are no longer reported accurately #7370

Closed
richvdh opened this issue Apr 29, 2020 · 4 comments
Closed

per-block cpu and db usage metrics are no longer reported accurately #7370

richvdh opened this issue Apr 29, 2020 · 4 comments
Assignees
Labels
z-bug (Deprecated Label)

Comments

@richvdh
Copy link
Member

richvdh commented Apr 29, 2020

this is a regression introduced by #7120.

@richvdh
Copy link
Member Author

richvdh commented May 4, 2020

The symptoms of this bug are basically this:

image

(no, we didn't magically make those processes 3 times as efficient. But we did deploy develop on 1st May.)

Those graphs chart the CPU usage used by with Measure() blocks, which are implemented via LoggingContexts. I think the basic problem is that, as of this change, the CPU usage stats are moved to the parent LoggingContext each time we switch away from the LoggingContext for the Measure block (ie, when it starts a blocking operation such as a database transaction); previously this happened only when the Measure block was all done and dusted here - ie, after Measure had added the CPU usage to its prometheus stats.

The problem is that I made that change for quite a good reason - which was that I was annoyed that LoggingContexts which did most of their work inside a child LoggingContext didn't report any CPU usage until the child finished (when it would suddenly report 5000% CPU usage).

@richvdh
Copy link
Member Author

richvdh commented May 4, 2020

Basically, I think we need to get rid of that self._resource_usage.reset(), and instead just add the delta to the parent context. But if we do that, we need to make sure that add_database_transaction and co are propagated correctly.

@clokep
Copy link
Member

clokep commented May 5, 2020

I'm going to see if I can figure this out.

@clokep
Copy link
Member

clokep commented May 6, 2020

Done in #7426

@clokep clokep closed this as completed May 6, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
z-bug (Deprecated Label)
Projects
None yet
Development

No branches or pull requests

2 participants