Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Time sync issue between SONiC and FRR stack #13046

Closed
nmoray-ebay opened this issue Dec 14, 2022 · 10 comments · Fixed by #14000
Closed

Time sync issue between SONiC and FRR stack #13046

nmoray-ebay opened this issue Dec 14, 2022 · 10 comments · Fixed by #14000
Labels
P1 Priority of the issue, lower than P0 Triaged this issue has been triaged

Comments

@nmoray-ebay
Copy link

nmoray-ebay commented Dec 14, 2022 •
edited
Loading

Description

There is a time sync issue between SONIC and the FRR stack. For instance, Inside vtysh, "Show interface status" CLI output and FRR log always shows the UTC timestamp which does not match with the SONIC in case a specific time zone is configured on the SONIC side.

Steps to reproduce the issue:

  1. Configure specific timezone except UTC in SONIC via "timedatectl set-timezone <>" CLI
  2. Flap the UP interface
  3. Go inside vtysh and check the output of "show interface <>" for the flap timestamp
  4. Alternatively check the zebra.log file

Describe the results you received:

LOGS:
root@SN2100-Leaf1:# show clock
Wed 14 Dec 2022 09:47:17 AM UTC
root@SN2100-Leaf1:# timedatectl set-timezone America/Los_Angeles ========> configured as PST ( using set timezone)
root@SN2100-Leaf1:#
root@SN2100-Leaf1:#
root@SN2100-Leaf1:#
root@SN2100-Leaf1:# show clock
Wed 14 Dec 2022 01:48:12 AM PST
root@SN2100-Leaf1:# config interface shutdown Ethernet0
root@SN2100-Leaf1:# config interface startup Ethernet0
root@SN2100-Leaf1:~# vtysh

Hello, this is FRRouting (version 7.5.1-sonic).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

SN2100-Leaf1# date
% Unknown command: date
SN2100-Leaf1# show int Ethernet0
Interface Ethernet0 is up, line protocol is down
Link ups: 1 last: 2022/12/14 06:35:42.89. ====> shows UTC timestamp
Link downs: 6 last: 2022/12/14 09:48:48.29
vrf: default
index 748 metric 0 mtu 9100 speed 0
flags: <UP,BROADCAST,MULTICAST>
Type: Unknown
HWaddr: 1c:34:da:36:f8:00
Interface Type Other
Interface Slave Type None

root@SN2100-Leaf1:/var/log/frr# tail -5f zebra.log
Dec 14 09:48:37.953850 SN2100-Leaf1 ERR bgp#zebra[35]: [EC 4043309093] netlink-dp (NS 0) error: Network is down, type=RTM_NEWNEXTHOP(104), seq=165, pid=2957759799
Dec 14 09:48:37.954253 SN2100-Leaf1 ERR bgp#zebra[35]: [EC 4043309074] Failed to install Nexthop ID (6) into the kernel
Dec 14 09:48:48.306387 SN2100-Leaf1 ERR bgp#zebra[35]: Extended Error: Carrier for nexthop device is down
Dec 14 09:48:48.306387 SN2100-Leaf1 ERR bgp#zebra[35]: [EC 4043309093] netlink-dp (NS 0) error: Network is down, type=RTM_NEWNEXTHOP(104), seq=169, pid=2957759799
Dec 14 09:48:48.306529 SN2100-Leaf1 ERR bgp#zebra[35]: [EC 4043309074] Failed to install Nexthop ID (6) into the kernel

Describe the results you expected:

Expecting a time sync between SONIC timezone and the FRR stack so that there will not be any confusion while interpreting the logs.

Output of show version:

SONiC Software Version: SONiC.202111.0-dirty-20221010.231127
Distribution: Debian 11.5
Kernel: 5.10.0-8-2-amd64
Build commit: ec691aa9f
Build date: Tue Oct 11 07:54:06 UTC 2022
Built by: kuldip.patel@aviz01

Platform: x86_64-mlnx_msn2100-r0
HwSKU: ACS-MSN2100
ASIC: mellanox
ASIC Count: 1
Serial Number: MT1950X05002
Model Number: MSN2100-CB2FO
Hardware Revision: A2
Uptime: 01:51:04 up 1 day,  3:20,  2 users,  load average: 1.87, 1.16, 1.00

Docker images:
REPOSITORY                    TAG                              IMAGE ID       SIZE
avizdock/ones-agent           v1.0.0                           5a2603b6a936   645MB
docker-dhcp-relay             latest                           890165d1d890   442MB
docker-teamd                  202111.0-dirty-20221010.231127   9e11eeacb228   443MB
docker-teamd                  latest                           9e11eeacb228   443MB
docker-syncd-mlnx             202111.0-dirty-20221010.231127   27f47464cdb1   928MB
docker-syncd-mlnx             latest                           27f47464cdb1   928MB
docker-sonic-telemetry        202111.0-dirty-20221010.231127   25aadf90f50c   517MB
docker-sonic-telemetry        latest                           25aadf90f50c   517MB
docker-sonic-mgmt-framework   202111.0-dirty-20221010.231127   843df6f803e6   585MB
docker-sonic-mgmt-framework   latest                           843df6f803e6   585MB
docker-snmp                   202111.0-dirty-20221010.231127   46007feaed7f   472MB
docker-snmp                   latest                           46007feaed7f   472MB
docker-sflow                  202111.0-dirty-20221010.231127   e89436c8880a   443MB
docker-sflow                  latest                           e89436c8880a   443MB
docker-router-advertiser      202111.0-dirty-20221010.231127   5df23e70624a   428MB
docker-router-advertiser      latest                           5df23e70624a   428MB
docker-platform-monitor       202111.0-dirty-20221010.231127   fe3e4ad48b2f   820MB
docker-platform-monitor       latest                           fe3e4ad48b2f   820MB
docker-orchagent              202111.0-dirty-20221010.231127   bf418d947316   460MB
docker-orchagent              latest                           bf418d947316   460MB
docker-nat                    202111.0-dirty-20221010.231127   291ea6094720   445MB
docker-nat                    latest                           291ea6094720   445MB
docker-mux                    202111.0-dirty-20221010.231127   b597c3040a27   481MB
docker-mux                    latest                           b597c3040a27   481MB
docker-macsec                 202111.0-dirty-20221010.231127   646a92998af7   446MB
docker-macsec                 latest                           646a92998af7   446MB
docker-lldp                   202111.0-dirty-20221010.231127   d1cf3fae008a   468MB
docker-lldp                   latest                           d1cf3fae008a   468MB
docker-fpm-frr                202111.0-dirty-20221010.231127   74a326b8bfff   461MB
docker-fpm-frr                latest                           74a326b8bfff   461MB
docker-database               202111.0-dirty-20221010.231127   19a368f709dc   428MB
docker-database               latest                           19a368f709dc   428MB
@madhupalu
Copy link

madhupalu commented Dec 14, 2022 •
edited
Loading

@nmoray-ebay is the issue in seen master branch?

@nmoray
Copy link
Contributor

nmoray commented Dec 14, 2022

@nmoray-ebay is the issue in seen master branch?

Yeah @madhupalu

@gechiang
Copy link
Collaborator

gechiang commented Jan 4, 2023

@nmoray-ebay can you confirm if this issue can be resolved if you restart BGP?

@gechiang gechiang added P1 Priority of the issue, lower than P0 Triaged this issue has been triaged labels Jan 4, 2023
@nmoray-ebay
Copy link
Author

@nmoray-ebay can you confirm if this issue can be resolved if you restart BGP?

No @gechiang as there is no time sync between the bgp container and host.

@nmoray-ebay
Copy link
Author

@madhupalu @gechiang ...I already have the fix for this issue. We can resolve this issue by mounting host's /etc/localtime and /etc/timezone on the bgp container.

@madhupalu
Copy link

@gechiang Thanks for triaging the issue.

@nmoray-ebay please raise a PR and add myself and @gechiang for code review.
@gechiang also let us know what releases need to back port the fix.

@nmoray-ebay
Copy link
Author

@gechiang Thanks for triaging the issue.

@nmoray-ebay please raise a PR and add myself and @gechiang for code review. @gechiang also let us know what releases need to back port the fix.

Sure @madhupalu. Thanks!

@nmoray
Copy link
Contributor

nmoray commented Jan 5, 2023

timedatectl set-timezone America/Los_Angeles ========> configured as PST

@nmoray-ebay can you confirm if this issue can be resolved if you restart BGP?

No @gechiang as there is no time sync between the bgp container and host.

@gechiang :
Please refer to the following logs.
admin@SN2100-Leaf1:$ date
Thu 05 Jan 2023 08:39:42 AM UTC
root@SN2100-Leaf1:# timedatectl set-timezone America/Los_Angeles
root@SN2100-Leaf1:#
root@SN2100-Leaf1:# date
Thu 05 Jan 2023 12:40:20 AM PST
root@SN2100-Leaf1:# docker exec -it bgp bash
root@SN2100-Leaf1:/# date
Thu Jan 5 08:40:30 UTC 2023
root@SN2100-Leaf1:/# exit
exit
root@SN2100-Leaf1:# docker stop bgp
bgp
root@SN2100-Leaf1:# docker start bgp
bgp
root@SN2100-Leaf1:# docker exec -it bgp bash
root@SN2100-Leaf1:/# date
Thu Jan 5 08:40:58 UTC 2023

@madhupalu
Copy link

@gechiang @zhangyanzhao
We are hitting authentication errors while raising PRs for sonic-buildimage repo, is there any process changed?
Note: we were able to contribute earlier and using github personal token to push changes to sonic branch.
Thanks
nikhil.moray@aviz01:~/worksapces/community/sonic-buildimage$ git push --set-upstream origin issue-13046
Username for ‘https://github.com/’: nmoray-ebay
Password for ‘https://nmoray-ebay/@github.com’:
remote: Permission to sonic-net/sonic-buildimage.git denied to nmoray-ebay.
fatal: unable to access ‘https://github.com/sonic-net/sonic-buildimage.git/’: The requested URL returned error: 403

@nmoray
Copy link
Contributor

nmoray commented Feb 17, 2023

@madhupalu @gechiang..PR has been raised. [https://github.com//pull/13852]

nmoray-ebay added a commit to nmoray-ebay/sonic-buildimage that referenced this issue Feb 27, 2023
@nmoray-ebay
Fixes sonic-net#13046: Frr - SONiC time sync issue

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode
f46acf8
@nmoray nmoray mentioned this issue Jun 5, 2023
Merged
8 tasks
qiluo-msft pushed a commit that referenced this issue Jun 25, 2023
@nmoray
Timezone sync issue between the host and containers (#14000)
f978b2b 
#### Why I did it
To fix the timezone sync issue between the containers and the host. If a certain timezone has been configured on the host (SONIC) then the expectation is to reflect the same across all the containers.

This will fix [Issue:13046](#13046).

For instance, a PST timezone has been set on the host and if the user checks the link flap logs (inside the FRR), it shows the UTC timestamp. Ideally, it should be PST.
sonic-otn pushed a commit to sonic-otn/sonic-buildimage that referenced this issue Sep 20, 2023
@nmoray @sonic-otn
Timezone sync issue between the host and containers (sonic-net#14000)
5e37660 
#### Why I did it
To fix the timezone sync issue between the containers and the host. If a certain timezone has been configured on the host (SONIC) then the expectation is to reflect the same across all the containers.

This will fix [Issue:13046](sonic-net#13046).

For instance, a PST timezone has been set on the host and if the user checks the link flap logs (inside the FRR), it shows the UTC timestamp. Ideally, it should be PST.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P1 Priority of the issue, lower than P0 Triaged this issue has been triaged
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Timezone sync issue between the host and containers nmoray/sonic-buildimage
4 participants
@madhupalu @gechiang @nmoray @nmoray-ebay