Log messages missed at boot for containers not depending on interfaces-config.service

[stepanblyschak]

Description

The logging infrastructure works buy running one host rsyslogd and rsyslogd running per each container.
rsyslogd inside containers have configuration to forward messages via UDP 127.0.0.1 to host rsyslogd that will write them.

This causes that at boot, when services start and a container service does not define a dependency on interfaces-config.service - some logging messages will be lost due to interfaces-config service reconfigures lo interface.

Steps to reproduce the issue:

1. Create an app ext that logs messages every second
2. Don't define "after" interfaces-config
3. Install on the switch
4. Do reboot/fast-reboot/config reload

Describe the results you received:

Mar 28 16:36:48.573640 r-tigon-04 NOTICE test#root: Tue Mar 28 13:36:48 UTC 2023 msg from test
Mar 28 16:36:49.577431 r-tigon-04 NOTICE test#root: Tue Mar 28 13:36:49 UTC 2023 msg from test
Mar 28 16:36:50.581154 r-tigon-04 NOTICE test#root: Tue Mar 28 13:36:50 UTC 2023 msg from test
Mar 28 16:36:50.805143 r-tigon-04 INFO systemd[1]: interfaces-config.service: Succeeded.

Mar 28 16:36:53.000521 r-tigon-04 INFO networking[778152]: warning: lo: post-up cmd 'ip addr del 127.0.0.1/8 dev lo' failed: returned 2 (RTNETLINK answers: Cannot assign requested address

Mar 28 16:36:53.485899 r-tigon-04 INFO systemd[1]: Finished Network initialization.
Mar 28 16:36:53.489068 r-tigon-04 INFO systemd[1]: Finished Update interfaces configuration.

Mar 28 16:36:53.591488 r-tigon-04 NOTICE test#root: Tue Mar 28 13:36:53 UTC 2023 msg from test
Mar 28 16:36:54.594922 r-tigon-04 NOTICE test#root: Tue Mar 28 13:36:54 UTC 2023 msg from test
Mar 28 16:36:55.598508 r-tigon-04 NOTICE test#root: Tue Mar 28 13:36:55 UTC 2023 msg from test

3 seconds with no logs

Describe the results you expected:

No log messages lost.

Setting dependency on interfaces-config is unwanted and considered as a W/A due to reasons:

• Unwanted delay at boot
• Container might run without network attached or not in the same host network and can't forward logs to 127.0.0.1

The logging infrastructure shall work regardless of interfaces-config.service

Output of show version:

SONiC Software Version: SONiC.202211_RC8.3-696a6a43b_Internal
Distribution: Debian 11.6
Kernel: 5.10.0-18-2-amd64
Build commit: 696a6a43b
Build date: Mon Mar 27 10:50:51 UTC 2023
Built by: sw-r2d2-bot@r-build-sonic-ci03-244

Platform: x86_64-mlnx_msn2700_simx-r0
HwSKU: ACS-MSN2700
ASIC: mellanox
ASIC Count: 1
Serial Number: MT1623X09522
Model Number: MSN2700-CS2FO
Hardware Revision: N/A
Uptime: 16:46:00 up 14:39,  1 user,  load average: 1.08, 0.81, 0.79
Date: Tue 28 Mar 2023 16:46:00

Docker images:
REPOSITORY                                         TAG                               IMAGE ID       SIZE
docker-orchagent                                   202211_RC8.3-696a6a43b_Internal   84055ebb4c77   539MB
docker-orchagent                                   latest                            84055ebb4c77   539MB
docker-fpm-frr                                     202211_RC8.3-696a6a43b_Internal   83d5f57abddb   550MB
docker-fpm-frr                                     latest                            83d5f57abddb   550MB
docker-teamd                                       202211_RC8.3-696a6a43b_Internal   e6b58409a5c6   520MB
docker-teamd                                       latest                            e6b58409a5c6   520MB
docker-macsec                                      latest                            d76f633dda63   522MB
docker-syncd-mlnx                                  202211_RC8.3-696a6a43b_Internal   2bd090589266   939MB
docker-syncd-mlnx                                  latest                            2bd090589266   939MB
docker-platform-monitor                            202211_RC8.3-696a6a43b_Internal   4fcfc27b9bbe   942MB
docker-platform-monitor                            latest                            4fcfc27b9bbe   942MB
docker-sonic-telemetry                             202211_RC8.3-696a6a43b_Internal   be9f9e750657   802MB
docker-sonic-telemetry                             latest                            be9f9e750657   802MB
docker-snmp                                        202211_RC8.3-696a6a43b_Internal   62432cf85f40   549MB
docker-snmp                                        latest                            62432cf85f40   549MB
docker-eventd                                      202211_RC8.3-696a6a43b_Internal   855b7c3d7218   503MB
docker-eventd                                      latest                            855b7c3d7218   503MB
docker-dhcp-relay                                  latest                            c90d1c5967eb   513MB
docker-lldp                                        202211_RC8.3-696a6a43b_Internal   e33be345d3f2   546MB
docker-lldp                                        latest                            e33be345d3f2   546MB
docker-mux                                         202211_RC8.3-696a6a43b_Internal   e9f6fa7ef277   552MB
docker-mux                                         latest                            e9f6fa7ef277   552MB
docker-database                                    202211_RC8.3-696a6a43b_Internal   9cae58961a25   503MB
docker-database                                    latest                            9cae58961a25   503MB
docker-sonic-p4rt                                  202211_RC8.3-696a6a43b_Internal   77b20e5b758d   586MB
docker-sonic-p4rt                                  latest                            77b20e5b758d   586MB
docker-router-advertiser                           202211_RC8.3-696a6a43b_Internal   f3a99688e9df   503MB
docker-router-advertiser                           latest                            f3a99688e9df   503MB
docker-sflow                                       202211_RC8.3-696a6a43b_Internal   780973343beb   479MB
docker-sflow                                       latest                            780973343beb   479MB
docker-nat                                         202211_RC8.3-696a6a43b_Internal   83454bb6925c   481MB
docker-nat                                         latest                            83454bb6925c   481MB
docker-sonic-mgmt-framework                        202211_RC8.3-696a6a43b_Internal   20cfc12ab3c5   612MB
docker-sonic-mgmt-framework                        latest                            20cfc12ab3c5   612MB
urm.nvidia.com/sw-nbu-sws-sonic-docker/doroce      1.1.0-202211-internal-5           078ba366e374   200MB
urm.nvidia.com/sw-nbu-sws-sonic-docker/sonic-wjh   1.4.0-202211-internal-5           b4f92852d9ca   323MB

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):