physical entity data missing from SNMP

[daall]

Description
We noticed a lot of failures in a new test added by sonic-net/sonic-mgmt#2379. This test verifies that the info stored in the physical entity MIB is correct. What we find is that there is physical entity data present in state DB that is not present in SNMP.

Steps to reproduce the issue:

1. Check State DB:

redis-cli -n 6 keys PHYSICAL_ENTITY_INFO*
1) "PHYSICAL_ENTITY_INFO|PSU 2"
2) "PHYSICAL_ENTITY_INFO|Front-panel temp sensor 2"
3) "PHYSICAL_ENTITY_INFO|PSU 1"
4) "PHYSICAL_ENTITY_INFO|ASIC temp sensor"
5) "PHYSICAL_ENTITY_INFO|Front-panel temp sensor 1"
6) "PHYSICAL_ENTITY_INFO|Rear-panel temp sensor 1"
7) "PHYSICAL_ENTITY_INFO|PSU-2 FAN-1"
8) "PHYSICAL_ENTITY_INFO|PSU-1 FAN-1"
9) "PHYSICAL_ENTITY_INFO|Rear-panel temp sensor 2"

2. Check SNMP for physical object types:

docker exec -it snmp snmpwalk -v2c -c obfuscated_community_string 127.0.0.1 1.3.6.1.2.1.47.1.1.1.1.5
iso.3.6.1.2.1.47.1.1.1.1.5.1 = INTEGER: 3
iso.3.6.1.2.1.47.1.1.1.1.5.1000 = INTEGER: 10
iso.3.6.1.2.1.47.1.1.1.1.5.1001 = INTEGER: 8
iso.3.6.1.2.1.47.1.1.1.1.5.1002 = INTEGER: 8
iso.3.6.1.2.1.47.1.1.1.1.5.1011 = INTEGER: 8
iso.3.6.1.2.1.47.1.1.1.1.5.1012 = INTEGER: 8
iso.3.6.1.2.1.47.1.1.1.1.5.1013 = INTEGER: 8
iso.3.6.1.2.1.47.1.1.1.1.5.1021 = INTEGER: 8
iso.3.6.1.2.1.47.1.1.1.1.5.1022 = INTEGER: 8
iso.3.6.1.2.1.47.1.1.1.1.5.1023 = INTEGER: 8
iso.3.6.1.2.1.47.1.1.1.1.5.1031 = INTEGER: 8
iso.3.6.1.2.1.47.1.1.1.1.5.1032 = INTEGER: 8
iso.3.6.1.2.1.47.1.1.1.1.5.1033 = INTEGER: 8
iso.3.6.1.2.1.47.1.1.1.1.5.1041 = INTEGER: 8
iso.3.6.1.2.1.47.1.1.1.1.5.1042 = INTEGER: 8
iso.3.6.1.2.1.47.1.1.1.1.5.1043 = INTEGER: 8
<8s and 10s repeat many more times corresponding to ports and port sensors in the system>

3. Check SNMP for physical object descriptions:

iso.3.6.1.2.1.47.1.1.1.1.2.1 = ""
iso.3.6.1.2.1.47.1.1.1.1.2.1000 = STRING: "QSFP28 or later for etp1"
iso.3.6.1.2.1.47.1.1.1.1.2.1001 = STRING: "DOM Temperature Sensor for etp1"
iso.3.6.1.2.1.47.1.1.1.1.2.1002 = STRING: "DOM Voltage Sensor for etp1"
iso.3.6.1.2.1.47.1.1.1.1.2.1011 = STRING: "DOM RX Power Sensor for etp1/1"
iso.3.6.1.2.1.47.1.1.1.1.2.1012 = STRING: "DOM TX Bias Sensor for etp1/1"
iso.3.6.1.2.1.47.1.1.1.1.2.1013 = STRING: "DOM TX Power Sensor for etp1/1"
iso.3.6.1.2.1.47.1.1.1.1.2.1021 = STRING: "DOM RX Power Sensor for etp1/2"
iso.3.6.1.2.1.47.1.1.1.1.2.1022 = STRING: "DOM TX Bias Sensor for etp1/2"
iso.3.6.1.2.1.47.1.1.1.1.2.1023 = STRING: "DOM TX Power Sensor for etp1/2"
iso.3.6.1.2.1.47.1.1.1.1.2.1031 = STRING: "DOM RX Power Sensor for etp1/3"
iso.3.6.1.2.1.47.1.1.1.1.2.1032 = STRING: "DOM TX Bias Sensor for etp1/3"
iso.3.6.1.2.1.47.1.1.1.1.2.1033 = STRING: "DOM TX Power Sensor for etp1/3"
iso.3.6.1.2.1.47.1.1.1.1.2.1041 = STRING: "DOM RX Power Sensor for etp1/4"
iso.3.6.1.2.1.47.1.1.1.1.2.1042 = STRING: "DOM TX Bias Sensor for etp1/4"
iso.3.6.1.2.1.47.1.1.1.1.2.1043 = STRING: "DOM TX Power Sensor for etp1/4"
<descriptions repeat many more times for all the ports and port sensors in the system>

Describe the results you received:
We only see ports, port sensors, and the overall device chassis in the physical entity MIB.

Describe the results you expected:
We should see objects corresponding to each physical entity present in state DB.

Additional information you deem important (e.g. issue happens only occasionally):

Output of show version:

show ver

SONiC Software Version: SONiC.HEAD.169-c146eeaa
Distribution: Debian 10.7
Kernel: 4.19.0-9-2-amd64
Build commit: c146eeaa
Build date: Tue Dec 22 13:40:46 UTC 2020
Built by: johnar@jenkins-worker-22

Platform: x86_64-cel_seastone-r0
HwSKU: Celestica-DX010-C32
ASIC: broadcom
ASIC Count: 1
Serial Number: DX010F2B118711MS100005
Uptime: 19:00:26 up  5:05,  1 user,  load average: 2.39, 3.19, 3.31

Docker images:
REPOSITORY                    TAG                 IMAGE ID            SIZE
docker-sonic-telemetry        HEAD.169-c146eeaa   d6f316245613        555MB
docker-sonic-telemetry        latest              d6f316245613        555MB
docker-syncd-brcm             HEAD.169-c146eeaa   6cf635032824        726MB
docker-syncd-brcm             latest              6cf635032824        726MB
docker-snmp                   HEAD.169-c146eeaa   e76565656569        525MB
docker-snmp                   latest              e76565656569        525MB
docker-teamd                  HEAD.169-c146eeaa   da0d0d5b3a44        524MB
docker-teamd                  latest              da0d0d5b3a44        524MB
docker-sonic-mgmt-framework   HEAD.169-c146eeaa   55718ad9e66b        641MB
docker-sonic-mgmt-framework   latest              55718ad9e66b        641MB
docker-router-advertiser      HEAD.169-c146eeaa   2558e1230a8d        480MB
docker-router-advertiser      latest              2558e1230a8d        480MB
docker-platform-monitor       HEAD.169-c146eeaa   ddd90e632b2f        605MB
docker-platform-monitor       latest              ddd90e632b2f        605MB
docker-lldp                   HEAD.169-c146eeaa   9d9b758fff1f        520MB
docker-lldp                   latest              9d9b758fff1f        520MB
docker-dhcp-relay             HEAD.169-c146eeaa   ee7e2f914bfa        487MB
docker-dhcp-relay             latest              ee7e2f914bfa        487MB
docker-database               HEAD.169-c146eeaa   cf96413b03e7        480MB
docker-database               latest              cf96413b03e7        480MB
docker-orchagent              HEAD.169-c146eeaa   425d0ad7b0da        554MB
docker-orchagent              latest              425d0ad7b0da        554MB
docker-nat                    HEAD.169-c146eeaa   1ef25954b9f9        526MB
docker-nat                    latest              1ef25954b9f9        526MB
docker-fpm-frr                HEAD.169-c146eeaa   2f134a3a81ce        539MB
docker-fpm-frr                latest              2f134a3a81ce        539MB
docker-sflow                  HEAD.169-c146eeaa   cc3ad45aa34c        523MB
docker-sflow                  latest              cc3ad45aa34c        523MB

(this is also seen on arista7050cx3-32s devices running the same version)