entity-manager take up too many resources

[jack206]

CPU: 63% usr 35% sys 0% nic 0% idle 0% io 0% irq 0% sirq
Load average: 5.90 5.09 4.25 2/102 2848
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
1177 1 root R 10416 2% 55% /usr/bin/entity-manager
1182 1 root D 7808 2% 10% /usr/bin/fansensor
1180 1 root D 7432 2% 7% /usr/bin/adcsensor
1185 1 root D 7804 2% 6% /usr/bin/psusensor
1184 1 root S 6804 2% 3% /usr/bin/hwmontempsensor
2836 31946 root R 3116 1% 2% top
2210 1661 root S 3116 1% 1% top
343 340 messageb S 3696 1% 1% dbus-broker --log 4 --controller 9 --machine-id 734d44fa48f34a3a9a089754db3d802e --max-bytes 536870912 --max-fds 4096 --max-matches 16384 --audit
31929 1 root S 3752 1% 1% /usr/sbin/dropbear -i -r /etc/dropbear/dropbear_rsa_host_key -G priv-admin -B
973 1 root S 45060 11% 1% /usr/sbin/rsyslogd -n
171 1 root S 34072 8% 1% /lib/systemd/systemd-journald
426 1 root S 18040 4% 1% ipmid

Sep 27 08:37:09 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W_______ xyz.openbmc_project.FruDevice
Sep 27 08:37:09 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________0 xyz.openbmc_project.FruDevice
Sep 27 08:37:09 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________1 xyz.openbmc_project.FruDevice
Sep 27 08:37:09 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________2 xyz.openbmc_project.FruDevice
Sep 27 08:37:11 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/BMC xyz.openbmc_project.FruDevice
Sep 27 08:37:11 BMC entity-manager[1177]: probeDBus: Found probe match on /xyz/openbmc_project/FruDevice/BMC xyz.openbmc_project.FruDevice
Sep 27 08:37:11 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W_______ xyz.openbmc_project.FruDevice
Sep 27 08:37:11 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________0 xyz.openbmc_project.FruDevice
Sep 27 08:37:11 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________1 xyz.openbmc_project.FruDevice
Sep 27 08:37:11 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________2 xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/BMC xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W_______ xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus: Found probe match on /xyz/openbmc_project/FruDevice/CRPS1600W_______ xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________0 xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus: Found probe match on /xyz/openbmc_project/FruDevice/CRPS1600W________0 xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________1 xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus: Found probe match on /xyz/openbmc_project/FruDevice/CRPS1600W________1 xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________2 xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus: Found probe match on /xyz/openbmc_project/FruDevice/CRPS1600W________2 xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/BMC xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W_______ xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________0 xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________1 xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________2 xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: run 604
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/BMC xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W_______ xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________0 xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________1 xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________2 xyz.openbmc_project.FruDevice
Sep 27 08:37:15 BMC entity-manager[1177]: ~PerformScan 621
Sep 27 08:37:16 BMC entity-manager[1177]: ~PerformScan 630
Sep 27 08:37:16 BMC entity-manager[1177]: findDbusObjects 205
Sep 27 08:37:16 BMC entity-manager[1177]: run 604
Sep 27 08:37:17 BMC entity-manager[1177]: getInterfaces 86
Sep 27 08:37:17 BMC entity-manager[1177]: getInterfaces 86
Sep 27 08:37:17 BMC entity-manager[1177]: getInterfaces 86
Sep 27 08:37:17 BMC entity-manager[1177]: getInterfaces 86
Sep 27 08:37:17 BMC entity-manager[1177]: getInterfaces 86
Sep 27 08:37:17 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/BMC xyz.openbmc_project.FruDevice
Sep 27 08:37:17 BMC entity-manager[1177]: probeDBus: Found probe match on /xyz/openbmc_project/FruDevice/BMC xyz.openbmc_project.FruDevice
Sep 27 08:37:17 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W_______ xyz.openbmc_project.FruDevice
Sep 27 08:37:17 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________0 xyz.openbmc_project.FruDevice
Sep 27 08:37:17 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________1 xyz.openbmc_project.FruDevice
Sep 27 08:37:17 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________2 xyz.openbmc_project.FruDevice
Sep 27 08:37:18 BMC ipmid[426]: thresholdChanged: deassert
Sep 27 08:37:20 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/BMC xyz.openbmc_project.FruDevice
Sep 27 08:37:21 BMC entity-manager[1177]: probeDBus: Found probe match on /xyz/openbmc_project/FruDevice/BMC xyz.openbmc_project.FruDevice
Sep 27 08:37:21 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W_______ xyz.openbmc_project.FruDevice
Sep 27 08:37:21 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________0 xyz.openbmc_project.FruDevice
Sep 27 08:37:21 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________1 xyz.openbmc_project.FruDevice
Sep 27 08:37:21 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________2 xyz.openbmc_project.FruDevice
Sep 27 08:37:24 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/BMC xyz.openbmc_project.FruDevice
Sep 27 08:37:24 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W_______ xyz.openbmc_project.FruDevice
Sep 27 08:37:24 BMC entity-manager[1177]: probeDBus: Found probe match on /xyz/openbmc_project/FruDevice/CRPS1600W_______ xyz.openbmc_project.FruDevice
Sep 27 08:37:24 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________0 xyz.openbmc_project.FruDevice
Sep 27 08:37:24 BMC entity-manager[1177]: probeDBus: Found probe match on /xyz/openbmc_project/FruDevice/CRPS1600W________0 xyz.openbmc_project.FruDevice
Sep 27 08:37:24 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________1 xyz.openbmc_project.FruDevice
Sep 27 08:37:24 BMC entity-manager[1177]: probeDBus: Found probe match on /xyz/openbmc_project/FruDevice/CRPS1600W________1 xyz.openbmc_project.FruDevice
Sep 27 08:37:24 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________2 xyz.openbmc_project.FruDevice
Sep 27 08:37:24 BMC entity-manager[1177]: probeDBus: Found probe match on /xyz/openbmc_project/FruDevice/CRPS1600W________2 xyz.openbmc_project.FruDevice
Sep 27 08:37:24 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/BMC xyz.openbmc_project.FruDevice
Sep 27 08:37:24 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W_______ xyz.openbmc_project.FruDevice
Sep 27 08:37:24 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________0 xyz.openbmc_project.FruDevice
Sep 27 08:37:24 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________1 xyz.openbmc_project.FruDevice
Sep 27 08:37:24 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________2 xyz.openbmc_project.FruDevice
Sep 27 08:37:25 BMC entity-manager[1177]: run 604
Sep 27 08:37:25 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/BMC xyz.openbmc_project.FruDevice
Sep 27 08:37:25 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W_______ xyz.openbmc_project.FruDevice
Sep 27 08:37:25 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________0 xyz.openbmc_project.FruDevice
Sep 27 08:37:25 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________1 xyz.openbmc_project.FruDevice
Sep 27 08:37:25 BMC entity-manager[1177]: probeDBus /xyz/openbmc_project/FruDevice/CRPS1600W________2 xyz.openbmc_project.FruDevice
Sep 27 08:37:25 BMC entity-manager[1177]: ~PerformScan 621

[jack206]

The background log keeps scrolling

[jack206]

root@BMC:~# time busctl tree xyz.openbmc_project.EntityManager | cat
└─/xyz
└─/xyz/openbmc_project
├─/xyz/openbmc_project/EntityManager
└─/xyz/openbmc_project/inventory
└─/xyz/openbmc_project/inventory/system
├─/xyz/openbmc_project/inventory/system/chassis
│ └─/xyz/openbmc_project/inventory/system/chassis/BMC
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/BOARD_CURRENT2
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_P0V6_VTT
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_P1V05
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_P1V2_VDDQ
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_P1V2_VDDQ_V
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_P1V8
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_P1V8_AUX
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_P2V5_VPP
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_P3V3_AUX
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_P3V3_STBY
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_P5V_AUX
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_TEMP
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_V1P8_EMMC_O
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_V3P3_EMMC_O
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_VCCIN
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_VCC_ANA
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_VNN_NAC
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_VNN_PCH
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_XDPE_P1V05_V
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_XDPE_P1V8_V
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_XDPE_PCH_V
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_XDPE_VCCIN_V
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_XDPE_VCC_V
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/CPU_XDPE_VNN_V
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/FAN_OUTLET_TEMP0
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/FAN_OUTLET_TEMP1
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_TEMP
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_VDD12V
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_VDD12V_1
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_VDD1V0_FPGA
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_VDD1V2
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_VDD1V2_FPGA
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_VDD1V8_CLK
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_VDD1V8_FPGA
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_VDD3V3
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_VDD3V3_CLK
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_VDD3V3_STBY
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_VDD5V_VR
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_VDDO1V2
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_VDDO1V8
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MAC_VDD_CORE
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MGMT_INLET_TEMP0
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/MGMT_INLET_TEMP1
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN1_1
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN1_2
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN2_1
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN2_2
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN3_1
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN3_2
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN4_1
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN4_2
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN5_1
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN5_2
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN6_1
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN6_2
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN7_1
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN7_2
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN8_1
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/ODM_FAN8_2
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/PSU1
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/PSU2
│ ├─/xyz/openbmc_project/inventory/system/chassis/BMC/PSU3
│ └─/xyz/openbmc_project/inventory/system/chassis/BMC/PSU4
└─/xyz/openbmc_project/inventory/system/powersupply
├─/xyz/openbmc_project/inventory/system/powersupply/CRPS1600W_1
├─/xyz/openbmc_project/inventory/system/powersupply/CRPS1600W_2
├─/xyz/openbmc_project/inventory/system/powersupply/CRPS1600W_3
└─/xyz/openbmc_project/inventory/system/powersupply/CRPS1600W_4

real 3m9.069s
user 0m0.490s
sys 0m0.123s

[williamspatrick]

What hardware are you running this on? Is the EM JSON all upstreamed or do you have your own JSON you are loading? Are you able to share it if it isn’t upstreamed?

[jack206]

AST2500,the EM file is:
bmc_chassis.json
crps1600w_psu.json
fan12k8080_fan.json

[jack206]

oem_xxx type You can interpret it as xxx type

[edtanous]

I see a number of logs, but given you're using a non upstreamed system on a fork, it's nearly impossible for anyone to debug this for you. One possibility is that something on your system is repeatedly calling the DBus ReScan() method, which is causing EM to rescan over and over. It's difficult to make any more guesses without being able to see the complete code, and being based on a fork.

[williamspatrick]

AST2500,the EM file is:
bmc_chassis.json
crps1600w_psu.json
fan12k8080_fan.json

I don’t see anything obviously wrong here that would have itself cause performance issues as you’ve described. I will say that the probe statement in the chassis object is far too unspecified. Just looking for “BMC” is a lot less than desirable; certainly this never could be upstreamed in its current state. I would suggest you get your backplane eeprom to have more specific details in it.

[jack206]

I am based on that commit：
commit b0c7bd1 (tag: devtool-base, master)
Author: Zev Weiss zev@bewilderbeest.net
Date: Tue Apr 19 17:58:01 2022 -0700

romed8hm3: add VDDCR_CPU warning & critical thresholds

These had been omitted initially while we were sorting out some
confusion with the VRM that provides this supply rail, but I forgot to
go back and add them after that was resolved.

Tested: warning & critical thresholds appear as expected on dbus and in
webui-vue on romed8hm3.

Signed-off-by: Zev Weiss <zev@bewilderbeest.net>
Change-Id: I605c7813f70b5004f50be831eea4746e48a4a5bd