[Bug]: kea-dhcp4 process killed and not spawned after a config reload

[harish-kalyanaraman]

Issue Description

Running MX Topology running 202311 based image, During nightly sonic-mgmt tests run , some testcases have a teardown error due to loganalyzer. This happens intermittently.
The error is:
ERR dhcp_server#supervisor-proc-exit-listener: Process 'kea-dhcp4' is not running in namespace 'host' (8.0 minutes).

This error happens because the dhcp-server-ipv4:kea-dhcp4 process inside the dhcp_server docker gets terminated by SIGHUP and doesnt recover

To Reproduce this issue:

1. Run config reload -y
2. In syslog, monitor kea-dhcp4 process

Results you see

In syslog:

Sep 22 09:45:01.061068 ixs-7215-pizza-GA-4 INFO python[12475]: ansible-ansible.legacy.command Invoked with executable=/bin/bash _raw_params=config reload -h _uses_shell=True warn=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None creates=None removes=None stdin=None
Sep 22 09:45:34.100994 ixs-7215-pizza-GA-4 INFO python[12835]: ansible-ansible.legacy.command Invoked with executable=/bin/bash _raw_params=config reload -y -f &>/dev/null _uses_shell=True warn=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None creates=None removes=None stdin=None
Sep 22 09:45:36.094052 ixs-7215-pizza-GA-4 NOTICE switch_hash: 'reload' executing with command: config reload -y -f
Sep 22 09:45:36.096413 ixs-7215-pizza-GA-4 NOTICE switch_hash: 'reload' stopping services...
Sep 22 09:45:36.290620 ixs-7215-pizza-GA-4 INFO monit[21093]: 'container_checker' unmonitor on user request
Sep 22 09:45:36.291202 ixs-7215-pizza-GA-4 INFO monit[21093]: Monit daemon with PID 21093 awakened
Sep 22 09:45:36.291521 ixs-7215-pizza-GA-4 INFO monit[21093]: Awakened by User defined signal 1
Sep 22 09:45:36.389142 ixs-7215-pizza-GA-4 INFO monit[21093]: 'container_checker' unmonitor action done

Sep 22 09:48:09.066488 ixs-7215-pizza-GA-4 DEBUG dhcp_server#container_startup.py: container_up: BEGIN
Sep 22 09:48:09.080425 ixs-7215-pizza-GA-4 DEBUG dhcp_server#container_startup.py: container_up: args: feature=dhcp_server, owner=local, version=20231110.22 DB: set_owner=local state_data={#12 "current_owner": "none",#12 "update_time": "",#12 "container_id": "",#12 "remote_state": "none",#12 "container_version": "0.0.0",#12 "system_state": "",#12 "state": "enabled"#12}
Sep 22 09:48:09.411262 ixs-7215-pizza-GA-4 INFO dhcp_server#supervisord 2024-09-22 09:48:09,392 INFO exited: start (exit status 0; expected)
Sep 22 09:48:10.640635 ixs-7215-pizza-GA-4 INFO dhcp_server#supervisord 2024-09-22 09:48:10,597 INFO spawned: 'dhcpservd' with pid 26
Sep 22 09:48:11.965180 ixs-7215-pizza-GA-4 INFO dhcp_server#supervisord 2024-09-22 09:48:11,954 INFO success: dhcpservd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Sep 22 09:48:13.154386 ixs-7215-pizza-GA-4 INFO dhcp_server#supervisord 2024-09-22 09:48:13,137 INFO spawned: 'kea-dhcp4' with pid 27
Sep 22 09:48:13.702863 ixs-7215-pizza-GA-4 INFO dhcp_server#supervisord: kea-dhcp4 2024-09-22 09:48:13.701 INFO [kea-dhcp4.dhcp4/27.3069419536] DHCP4_STARTING Kea DHCPv4 server version 2.2.0 (stable) starting
Sep 22 09:48:14.760639 ixs-7215-pizza-GA-4 INFO dhcp_server#supervisord 2024-09-22 09:48:14,753 INFO success: kea-dhcp4 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Sep 22 09:48:14.761124 ixs-7215-pizza-GA-4 INFO dhcp_server#supervisord 2024-09-22 09:48:14,754 INFO exited: kea-dhcp4 (terminated by SIGHUP; not expected)
Sep 22 09:48:17.188822 ixs-7215-pizza-GA-4 INFO dhcp_server#supervisord 2024-09-22 09:48:17,181 INFO exited: dependent-startup (exit status 0; expected)
Sep 22 09:48:48.177810 ixs-7215-pizza-GA-4 DEBUG bgp#bgpcfgd: Received message : '('localhost', 'SET', (('bgp_asn', '64001'), ('buffer_model', 'traditional'), ('cloudtype', 'Public'), ('default_bgp_status', 'down'), ('default_pfcwd_status', 'enable'), ('deployment_id', '1'), ('dhcp_server', 'enabled'), ('docker_routing_config_mode', 'separated'), ('hostname', 'ixs-7215-pizza-GA-4'), ('hwsku', 'Nokia-7215'), ('mac', '8C:7A:00:96:BC:76'), ('platform', 'armhf-nokia_ixs7215_52x-r0'), ('region', 'None'), ('synchronous_mode', 'enable'), ('timezone', 'UTC'), ('type', 'BmcMgmtToRRouter'), ('yang_config_validation', 'disable')))'
Sep 22 09:49:16.337879 ixs-7215-pizza-GA-4 ERR dhcp_server#supervisor-proc-exit-listener: Process 'kea-dhcp4' is not running in namespace 'host' (1.0 minutes).
Sep 22 09:49:54.501081 ixs-7215-pizza-GA-4 INFO lldp#lldpmgrd[54]: Device Config Opcode: SET Dict {'bgp_asn': '64001', 'buffer_model': 'traditional', 'cloudtype': 'Public', 'default_bgp_status': 'down', 'default_pfcwd_status': 'enable', 'deployment_id': '1', 'dhcp_server': 'enabled', 'docker_routing_config_mode': 'separated', 'hostname': 'ixs-7215-pizza-GA-4', 'hwsku': 'Nokia-7215', 'mac': '8C:7A:00:96:BC:76', 'platform': 'armhf-nokia_ixs7215_52x-r0', 'region': 'None', 'synchronous_mode': 'enable', 'timezone': 'UTC', 'type': 'BmcMgmtToRRouter', 'yang_config_validation': 'disable'} Key localhost
Sep 22 09:50:16.469622 ixs-7215-pizza-GA-4 ERR dhcp_server#supervisor-proc-exit-listener: Process 'kea-dhcp4' is not running in namespace 'host' (2.0 minutes).

Results you expected to see

kea-dhcp4 process should not crash

Is it platform specific

generic

Relevant log output

No response

Output of show version

No response

Attach files (if any)

No response

[yaqiangz]

Hi @harish-kalyanaraman could you please help to provide details of fail case? Which case hit this issue?
I tried config reload for 300+ times, didn't repro this issue