contrib: add script to dump irq stats. #336

imran-kn · 2023-07-07T01:09:15Z

script to dump irq stats.

A sample output:

python3 -m drgn -s ~/Local-vmcores/sched-study/v4.14/vmlinux -c ~/Local-vmcores/sched-study/v4.14/vmcore contrib/irq.py
warning: could not get debugging information for:
kernel modules (could not read depmod: open: /lib/modules/4.14.307/modules.dep.bin: No such file or directory)
###################################################
List of IRQs
###################################################
irq: 0 name: timer (struct irq_desc *)0xffff9b42fd00d400 (struct irqaction *)0xffffffff88425240
irq: 1 name: i8042 (struct irq_desc *)0xffff9b42fd00d600 (struct irqaction *)0xffff9b42fcb13c00
irq: 4 name: ttyS0 (struct irq_desc *)0xffff9b42fd00dc00 (struct irqaction *)0xffff9b42fcb2bc00
irq: 8 name: rtc0 (struct irq_desc *)0xffff9b42fd00e400 (struct irqaction *)0xffff9b42fcb59380
irq: 9 name: acpi (struct irq_desc *)0xffff9b42fd00e600 (struct irqaction *)0xffff9b42fcafca80
irq: 11 name: eth0 (struct irq_desc *)0xffff9b42fd00ea00 (struct irqaction *)0xffff9b42fc878d80
irq: 12 name: i8042 (struct irq_desc *)0xffff9b42fd00ec00 (struct irqaction *)0xffff9b42fcb13900
irq: 14 name: ata_piix (struct irq_desc *)0xffff9b42fd00f000 (struct irqaction *)0xffff9b42fc690800
irq: 15 name: ata_piix (struct irq_desc *)0xffff9b42fd00f200 (struct irqaction *)0xffff9b42fc690780

###################################################
IRQ affinities
###################################################
irq: 0 name: timer affinity: 0-7
irq: 1 name: i8042 affinity: 0-7
irq: 4 name: ttyS0 affinity: 0-7
irq: 8 name: rtc0 affinity: 0-7
irq: 9 name: acpi affinity: 0-7
irq: 11 name: eth0 affinity: 0-7
irq: 12 name: i8042 affinity: 0-7
irq: 14 name: ata_piix affinity: 0-7
irq: 15 name: ata_piix affinity: 0-7

###################################################
IRQ stats
###################################################
irq: 0 name: timer (struct irq_desc *)0xffff9b42fd00d400
CPU: 0 count: 162
Total: 162
irq: 1 name: i8042 (struct irq_desc *)0xffff9b42fd00d600
CPU: 0 count: 10
Total: 10
irq: 4 name: ttyS0 (struct irq_desc *)0xffff9b42fd00dc00
CPU: 0 count: 220
Total: 220
irq: 8 name: rtc0 (struct irq_desc *)0xffff9b42fd00e400
CPU: 0 count: 1
Total: 1
Total: 0
irq: 11 name: eth0 (struct irq_desc *)0xffff9b42fd00ea00
CPU: 0 count: 85
Total: 85
irq: 12 name: i8042 (struct irq_desc *)0xffff9b42fd00ec00
CPU: 0 count: 125
Total: 125
irq: 14 name: ata_piix (struct irq_desc *)0xffff9b42fd00f000
CPU: 0 count: 496
Total: 496
irq: 15 name: ata_piix (struct irq_desc *)0xffff9b42fd00f200
CPU: 0 count: 11
Total: 11

###################################################
cpuwise IRQ stats
###################################################
IRQ stats for cpu: 0
irq: 0 name: timer (struct irq_desc *)0xffff9b42fd00d400 count: 162
irq: 1 name: i8042 (struct irq_desc *)0xffff9b42fd00d600 count: 10
irq: 4 name: ttyS0 (struct irq_desc *)0xffff9b42fd00dc00 count: 220
irq: 8 name: rtc0 (struct irq_desc *)0xffff9b42fd00e400 count: 1
irq: 11 name: eth0 (struct irq_desc *)0xffff9b42fd00ea00 count: 85
irq: 12 name: i8042 (struct irq_desc *)0xffff9b42fd00ec00 count: 125
irq: 14 name: ata_piix (struct irq_desc *)0xffff9b42fd00f000 count: 496
irq: 15 name: ata_piix (struct irq_desc *)0xffff9b42fd00f200 count: 11
Total: 1110

IRQ stats for cpu: 1
Total: 0

IRQ stats for cpu: 2
Total: 0

IRQ stats for cpu: 3
Total: 0

IRQ stats for cpu: 4
Total: 0

IRQ stats for cpu: 5
Total: 0

IRQ stats for cpu: 6
Total: 0

IRQ stats for cpu: 7
Total:0

maheshsal · 2023-09-11T16:28:19Z

Hi @imran-kn, Nice work. However it looks like this script supports pretty old kernel level 4.14.307 and fails with upstream kernel level v6.4+.

Is there any plan to get this script working for upstream kernel as well ?

$ sudo python3 -m drgn contrib/irq.py
warning: could not get debugging information for:
kernel (could not find vmlinux for 6.4.12-200.fc38.x86_64)
kernel modules (could not find loaded kernel modules: could not find 'struct module')
###################################################
List of IRQs
###################################################
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/salgaonkarm/Mahesh/drgn/drgn/drgn/main.py", line 16, in
_main()
File "/home/salgaonkarm/Mahesh/drgn/drgn/drgn/cli.py", line 299, in _main
runpy.run_path(script, init_globals={"prog": prog}, run_name="main")
File "", line 291, in run_path
File "", line 98, in _run_module_code
File "", line 88, in _run_code
File "contrib/irq.py", line 456, in
print_all_irqs(prog)
File "contrib/irq.py", line 442, in print_all_irqs
for irq in for_each_irq(prog):
File "contrib/irq.py", line 90, in for_each_irq
count = len(prog["irq_desc"])
~~~~^^^^^^^^^^^^
KeyError: 'irq_desc'

imran-kn · 2023-09-11T16:47:53Z

Hi @imran-kn, Nice work. However it looks like this script supports pretty old kernel level 4.14.307 and fails with upstream kernel level v6.4+.

Is there any plan to get this script working for upstream kernel as well ?

$ sudo python3 -m drgn contrib/irq.py warning: could not get debugging information for: kernel (could not find vmlinux for 6.4.12-200.fc38.x86_64) kernel modules (could not find loaded kernel modules: could not find 'struct module') ################################################### List of IRQs ################################################### Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/home/salgaonkarm/Mahesh/drgn/drgn/drgn/main.py", line 16, in _main() File "/home/salgaonkarm/Mahesh/drgn/drgn/drgn/cli.py", line 299, in _main runpy.run_path(script, init_globals={"prog": prog}, run_name="main") File "", line 291, in run_path File "", line 98, in _run_module_code File "", line 88, in _run_code File "contrib/irq.py", line 456, in print_all_irqs(prog) File "contrib/irq.py", line 442, in print_all_irqs for irq in for_each_irq(prog): File "contrib/irq.py", line 90, in for_each_irq count = len(prog["irq_desc"]) ~~~~^^^^^^^^^^^^ KeyError: 'irq_desc'

Hello Mahesh,
Thanks for pointing this out. Let me fix this for newer upstream kernels as well. I remember testing it till v6.1. I will update the PR soon.

maheshsal · 2023-09-11T16:51:51Z

ouch my bad.. After pointing it to debuginfo vmlinux using '-s' option it started working..

$ sudo python3 -m drgn -s /usr/lib/debug/lib/modules/6.4.15-200.fc38.x86_64/vmlinux contrib/irq.py
warning: could not get debugging information for:
/usr/lib/modules/6.4.12-200.fc38.x86_64/kernel/drivers/input/misc/uinput.ko.xz (libdwfl error: No DWARF information found)
/usr/lib/modules/6.4.12-200.fc38.x86_64/kernel/fs/nfs/nfsv3.ko.xz (libdwfl error: No DWARF information found)
/usr/lib/modules/6.4.12-200.fc38.x86_64/kernel/fs/nfs_common/nfs_acl.ko.xz (libdwfl error: No DWARF information found)
/usr/lib/modules/6.4.12-200.fc38.x86_64/kernel/drivers/net/usb/r8153_ecm.ko.xz (libdwfl error: No DWARF information found)
/usr/lib/modules/6.4.12-200.fc38.x86_64/kernel/fs/lockd/lockd.ko.xz (libdwfl error: No DWARF information found)
... 189 more
###################################################
List of IRQs
###################################################
irq: 1 name: i8042 (struct irq_desc *)0xffff9b9f4015fc00 (struct irqaction *)0xffff9b9f49d1b780
irq: 8 name: rtc0 (struct irq_desc *)0xffff9b9f4015c600 (struct irqaction *)0xffff9b9f49d1bb80
irq: 9 name: acpi (struct irq_desc *)0xffff9b9f4015ca00 (struct irqaction *)0xffff9b9f413e0e80
irq: 12 name: i8042 (struct irq_desc *)0xffff9b9f4015c400 (struct irqaction *)0xffff9b9f49d1be00
irq: 14 name: INT34C5:00 (struct irq_desc *)0xffff9b9f4015dc00 (struct irqaction *)0xffff9b9f425ac500
irq: 16 name: i801_smbus (struct irq_desc *)0xffff9b9f41cafc00 (struct irqaction *)0xffff9b9f4b8fc680
irq: 27 name: i2c_designware.0 (struct irq_desc *)0xffff9b9f418cd000 (struct irqaction *)0xffff9b9f49705080
irq: 40 name: i2c_designware.1 (struct irq_desc *)0xffff9b9f43e63200 (struct irqaction *)0xffff9b9f417b7480
irq: 61 name: GTCH7503:00 (struct irq_desc *)0xffff9b9f4a9dfc00 (struct irqaction *)0xffff9b9f4b85a380
irq: 120 name: dmar2 (struct irq_desc *)0xffff9b9f40290200 (struct irqaction *)0xffff9b9f40162700
irq: 121 name: dmar1 (struct irq_desc *)0xffff9b9f40293200 (struct irqaction *)0xffff9b9f40162480
irq: 122 name: dmar0 (struct irq_desc *)0xffff9b9f40292600 (struct irqaction *)0xffff9b9f40162580
irq: 123 name: dmar3 (struct irq_desc *)0xffff9b9f40293c00 (struct irqaction *)0xffff9b9f40162c00
irq: 124 name: PCIe PME (struct irq_desc *)0xffff9b9f418c7600 (struct irqaction *)0xffff9b9f4266de80
irq: 125 name: PCIe PME (struct irq_desc *)0xffff9b9f418c7c00 (struct irqaction *)0xffff9b9f49705700
irq: 126 name: PCIe PME (struct irq_desc *)0xffff9b9f41cafa00 (struct irqaction *)0xffff9b9f41fa3580
[...]

imran-kn · 2023-09-11T16:55:03Z

ouch my bad.. After pointing it to debuginfo vmlinux using '-s' option it started working..

$ sudo python3 -m drgn -s /usr/lib/debug/lib/modules/6.4.15-200.fc38.x86_64/vmlinux contrib/irq.py warning: could not get debugging information for: /usr/lib/modules/6.4.12-200.fc38.x86_64/kernel/drivers/input/misc/uinput.ko.xz (libdwfl error: No DWARF information found) /usr/lib/modules/6.4.12-200.fc38.x86_64/kernel/fs/nfs/nfsv3.ko.xz (libdwfl error: No DWARF information found) /usr/lib/modules/6.4.12-200.fc38.x86_64/kernel/fs/nfs_common/nfs_acl.ko.xz (libdwfl error: No DWARF information found) /usr/lib/modules/6.4.12-200.fc38.x86_64/kernel/drivers/net/usb/r8153_ecm.ko.xz (libdwfl error: No DWARF information found) /usr/lib/modules/6.4.12-200.fc38.x86_64/kernel/fs/lockd/lockd.ko.xz (libdwfl error: No DWARF information found) ... 189 more ################################################### List of IRQs ################################################### irq: 1 name: i8042 (struct irq_desc *)0xffff9b9f4015fc00 (struct irqaction *)0xffff9b9f49d1b780 irq: 8 name: rtc0 (struct irq_desc *)0xffff9b9f4015c600 (struct irqaction *)0xffff9b9f49d1bb80 irq: 9 name: acpi (struct irq_desc *)0xffff9b9f4015ca00 (struct irqaction *)0xffff9b9f413e0e80 irq: 12 name: i8042 (struct irq_desc *)0xffff9b9f4015c400 (struct irqaction *)0xffff9b9f49d1be00 irq: 14 name: INT34C5:00 (struct irq_desc *)0xffff9b9f4015dc00 (struct irqaction *)0xffff9b9f425ac500 irq: 16 name: i801_smbus (struct irq_desc *)0xffff9b9f41cafc00 (struct irqaction *)0xffff9b9f4b8fc680 irq: 27 name: i2c_designware.0 (struct irq_desc *)0xffff9b9f418cd000 (struct irqaction *)0xffff9b9f49705080 irq: 40 name: i2c_designware.1 (struct irq_desc *)0xffff9b9f43e63200 (struct irqaction *)0xffff9b9f417b7480 irq: 61 name: GTCH7503:00 (struct irq_desc *)0xffff9b9f4a9dfc00 (struct irqaction *)0xffff9b9f4b85a380 irq: 120 name: dmar2 (struct irq_desc *)0xffff9b9f40290200 (struct irqaction *)0xffff9b9f40162700 irq: 121 name: dmar1 (struct irq_desc *)0xffff9b9f40293200 (struct irqaction *)0xffff9b9f40162480 irq: 122 name: dmar0 (struct irq_desc *)0xffff9b9f40292600 (struct irqaction *)0xffff9b9f40162580 irq: 123 name: dmar3 (struct irq_desc *)0xffff9b9f40293c00 (struct irqaction *)0xffff9b9f40162c00 irq: 124 name: PCIe PME (struct irq_desc *)0xffff9b9f418c7600 (struct irqaction *)0xffff9b9f4266de80 irq: 125 name: PCIe PME (struct irq_desc *)0xffff9b9f418c7c00 (struct irqaction *)0xffff9b9f49705700 irq: 126 name: PCIe PME (struct irq_desc *)0xffff9b9f41cafa00 (struct irqaction *)0xffff9b9f41fa3580 [...]

No problem. Thanks for the timely update :)

osandov

It's really cool to visualize this on my workstation :) a few comments on the new helper.

drgn/helpers/linux/cpumask.py

…nges. In some cases (for example getting affinity of an irq), it is better to have an easily understandable list of cpus corresonding to a given cpumask. This helper converts a given cpumask to string, such that the string represents the range of CPUs that are present in the given mask. Signed-off-by: Imran Khan <imran.f.khan@oracle.com>

Sample output: python3 -m drgn -s vmlinux -c vmcore contrib/irq.py List of IRQs irq: 0 name: timer (struct irq_desc *)0xffff9b42fd00d400 (struct irqaction *)0xffffffff88425240 irq: 1 name: i8042 (struct irq_desc *)0xffff9b42fd00d600 (struct irqaction *)0xffff9b42fcb13c00 irq: 4 name: ttyS0 (struct irq_desc *)0xffff9b42fd00dc00 (struct irqaction *)0xffff9b42fcb2bc00 irq: 8 name: rtc0 (struct irq_desc *)0xffff9b42fd00e400 (struct irqaction *)0xffff9b42fcb59380 irq: 9 name: acpi (struct irq_desc *)0xffff9b42fd00e600 (struct irqaction *)0xffff9b42fcafca80 irq: 11 name: eth0 (struct irq_desc *)0xffff9b42fd00ea00 (struct irqaction *)0xffff9b42fc878d80 irq: 12 name: i8042 (struct irq_desc *)0xffff9b42fd00ec00 (struct irqaction *)0xffff9b42fcb13900 irq: 14 name: ata_piix (struct irq_desc *)0xffff9b42fd00f000 (struct irqaction *)0xffff9b42fc690800 irq: 15 name: ata_piix (struct irq_desc *)0xffff9b42fd00f200 (struct irqaction *)0xffff9b42fc690780 IRQ affinities irq: 0 name: timer affinity: 0-7 irq: 1 name: i8042 affinity: 0-7 irq: 4 name: ttyS0 affinity: 0-7 irq: 8 name: rtc0 affinity: 0-7 irq: 9 name: acpi affinity: 0-7 irq: 11 name: eth0 affinity: 0-7 irq: 12 name: i8042 affinity: 0-7 irq: 14 name: ata_piix affinity: 0-7 irq: 15 name: ata_piix affinity: 0-7 IRQ stats irq: 0 name: timer (struct irq_desc *)0xffff9b42fd00d400 CPU: 0 count: 162 Total: 162 irq: 1 name: i8042 (struct irq_desc *)0xffff9b42fd00d600 CPU: 0 count: 10 Total: 10 irq: 4 name: ttyS0 (struct irq_desc *)0xffff9b42fd00dc00 CPU: 0 count: 220 Total: 220 irq: 8 name: rtc0 (struct irq_desc *)0xffff9b42fd00e400 CPU: 0 count: 1 Total: 1 Total: 0 irq: 11 name: eth0 (struct irq_desc *)0xffff9b42fd00ea00 CPU: 0 count: 85 Total: 85 irq: 12 name: i8042 (struct irq_desc *)0xffff9b42fd00ec00 CPU: 0 count: 125 Total: 125 irq: 14 name: ata_piix (struct irq_desc *)0xffff9b42fd00f000 CPU: 0 count: 496 Total: 496 irq: 15 name: ata_piix (struct irq_desc *)0xffff9b42fd00f200 CPU: 0 count: 11 Total: 11 cpuwise IRQ stats IRQ stats for cpu: 0 irq: 0 name: timer (struct irq_desc *)0xffff9b42fd00d400 count: 162 irq: 1 name: i8042 (struct irq_desc *)0xffff9b42fd00d600 count: 10 irq: 4 name: ttyS0 (struct irq_desc *)0xffff9b42fd00dc00 count: 220 irq: 8 name: rtc0 (struct irq_desc *)0xffff9b42fd00e400 count: 1 irq: 11 name: eth0 (struct irq_desc *)0xffff9b42fd00ea00 count: 85 irq: 12 name: i8042 (struct irq_desc *)0xffff9b42fd00ec00 count: 125 irq: 14 name: ata_piix (struct irq_desc *)0xffff9b42fd00f000 count: 496 irq: 15 name: ata_piix (struct irq_desc *)0xffff9b42fd00f200 count: 11 Total: 1110 IRQ stats for cpu: 1 Total: 0 IRQ stats for cpu: 2 Total: 0 IRQ stats for cpu: 3 Total: 0 IRQ stats for cpu: 4 Total: 0 IRQ stats for cpu: 5 Total: 0 IRQ stats for cpu: 6 Total: 0 IRQ stats for cpu: 7 Total: 0 Signed-off-by: Imran Khan <imran.f.khan@oracle.com>

drgn/helpers/linux/cpumask.py

imran-kn force-pushed the contrib-irq-work branch from be5dd93 to 9c899f0 Compare October 15, 2023 16:25

imran-kn changed the title ~~Contrib irq work~~ contrib: add script to dump irq stats. Oct 15, 2023

osandov requested changes Oct 16, 2023

View reviewed changes

drgn/helpers/linux/cpumask.py Outdated Show resolved Hide resolved

drgn/helpers/linux/cpumask.py Outdated Show resolved Hide resolved

drgn/helpers/linux/cpumask.py Outdated Show resolved Hide resolved

drgn/helpers/linux/cpumask.py Outdated Show resolved Hide resolved

imran-kn force-pushed the contrib-irq-work branch from 983da10 to 9c899f0 Compare October 22, 2023 21:14

imran-kn force-pushed the contrib-irq-work branch from 9c899f0 to 2baf40b Compare October 23, 2023 02:25

imran-kn requested a review from osandov October 23, 2023 03:11

osandov reviewed Oct 25, 2023

View reviewed changes

drgn/helpers/linux/cpumask.py Outdated Show resolved Hide resolved

osandov force-pushed the contrib-irq-work branch from 2baf40b to 46d9446 Compare October 25, 2023 16:27

osandov approved these changes Oct 25, 2023

View reviewed changes

osandov merged commit 754b7be into osandov:main Oct 25, 2023
6 of 34 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

contrib: add script to dump irq stats. #336

contrib: add script to dump irq stats. #336

imran-kn commented Jul 7, 2023

maheshsal commented Sep 11, 2023

imran-kn commented Sep 11, 2023

maheshsal commented Sep 11, 2023

imran-kn commented Sep 11, 2023

osandov left a comment

contrib: add script to dump irq stats. #336

contrib: add script to dump irq stats. #336

Conversation

imran-kn commented Jul 7, 2023

maheshsal commented Sep 11, 2023

imran-kn commented Sep 11, 2023

maheshsal commented Sep 11, 2023

imran-kn commented Sep 11, 2023

osandov left a comment

Choose a reason for hiding this comment