syslog.py: report when ceph works are stuck #1945
Conversation
teuthology/task/internal/syslog.py
Outdated
| @@ -107,7 +107,7 @@ def syslog(ctx, config): | |||
| stdout = rem.sh( | |||
| [ | |||
| 'egrep', '--binary-files=text', | |||
| '\\bBUG\\b|\\bINFO\\b|\\bDEADLOCK\\b', | |||
| '\\bBUG\\b|\\bINFO\\b|\\bDEADLOCK\\b|\\bceph\\b.*\\bWQ_UNBOUND\\b', | |||
There was a problem hiding this comment.
[Sun Sep 10 07:19:00 2023] workqueue: delayed_work [ceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
[Sun Sep 10 08:41:24 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
This regex would match only the first message (ceph), but not the second (libceph).
There was a problem hiding this comment.
Also, I wouldn't cling to WQ_UNBOUND -- it's an implementation detail that could change. If the concern is work items taking longer than expected to run, it would probably be better to match on "workqueue: ... hogged CPU for ...".
There was a problem hiding this comment.
[Sun Sep 10 07:19:00 2023] workqueue: delayed_work [ceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
[Sun Sep 10 08:41:24 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUNDThis regex would match only the first message (ceph), but not the second (libceph).
Okay, let me run the test again.
Also, I wouldn't cling to
WQ_UNBOUND-- it's an implementation detail that could change. If the concern is work items taking longer than expected to run, it would probably be better to match on "workqueue: ... hogged CPU for ...".
I just want to match the ceph ones, this will match all the possible warnings, is that okay ?
There was a problem hiding this comment.
I just want to match the ceph ones
That is totally fine -- I just wanted to say that we shouldn't be matching on name of the internal flag.
This doesn't mean that the work item is stuck -- just that it's taking longer than a configurable threshold. BTW https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c25da5b7baf1d243e6612ba2b97e2a2c4a1376f6 |
Yeah, you are right. Just this warning could give us a hint that this code could be improved. |
Usually there will be performance bug when hitting the works are stuck: workqueue: delayed_work [ceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND URL: https://www.spinics.net/lists/ceph-users/msg78775.html Signed-off-by: Xiubo Li <xiubli@redhat.com>
Usually there will be performance bug when hitting the works are stuck:
workqueue: delayed_work [ceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
URL: https://www.spinics.net/lists/ceph-users/msg78775.html