[Bookworm] Timezone setting is not propagated to containers

[vivekrnv]

Description

Timezone setting is expected to propagate to the containers. But after bookworm upgrade in SONiC, it is broken

202311

1. Set a new timezone: config clock timezone Asia/Pyongyang
2. Updated on host immediately:

timedatectl show
Timezone=Asia/Pyongyang
TimeUSec=Wed 2024-11-27 06:17:03 KST

3. config save & config reload for the change to be updated in the container.

docker exec eventd zdump /etc/localtime
/etc/localtime  Wed Nov 27 06:20:09 2024 KST

docker exec eventd cat /etc/timezone
Asia/Pyongyang

docker exec eventd date
Wed Nov 27 06:20:39 KST 2024

202405 and above:

config clock timezone Asia/Pyongyang

timedatectl show
Timezone=Asia/Pyongyang
TimeUSec=Wed 2024-11-27 06:37:01 KST

zdump /etc/localtime
/etc/localtime  Wed Nov 27 06:37:15 2024 KST

After config reload, containers are expected to have timezone change but it is not seen

docker exec eventd zdump /etc/localtime
/etc/localtime  Tue Nov 26 21:39:00 2024 UTC

docker exec eventd date
Tue Nov 26 21:39:09 UTC 2024

docker exec eventd cat /etc/timezone
Etc/UTC

cat /etc/timezone
Etc/UTC (Even on host, this file is not updated)

Triage

Issue lies in timedatectl command not updating /etc/timezone in systemd version used on bookworm
Ref: https://unix.stackexchange.com/questions/451709/timedatectl-set-timezone-doesnt-update-etc-timezone

/etc/timezone appears to be mostly Debian-specific (including derivatives). On Debian systems, timedatectl set-timezone also updates /etc/timezone, in version of systemd older than 252.6-1 (so until Debian 11 included).

This PR #14000 (Which fixes the timesync issue between host and container) uses /etc/timezone as reference to update the timezone in container.

[vivekrnv]

@nmoray can the logic be updated instead to use /etc/localtime directly from host instead of using /etc/timezone

[nmoray]

st instead of using /etc/timezone

@vivekrnv It is not recommended to mount the symlink. That is why, I mounted only the timezone and is creating the symlink at docker runtime.

[vivekrnv]

Hmm, so one way i could think is this. @nmoray Can you try if this solves the problem?

-TZ=$(cat /etc/timezone)
+TZ=$(timedatectl show -p Timezone --value)

[prabhataravind]

@nmoray, could you please take a look at Vivek's comments and rework your patch?

[nmoray]

@prabhataravind Okay, let me look into it.
@vivekrnv BTW did you try using "dpkg-reconfigure tzdata" command? It does update the /etc/timezone file

[vivekrnv]

@prabhataravind Okay, let me look into it. @vivekrnv BTW did you try using "dpkg-reconfigure tzdata" command? It does update the /etc/timezone file

I haven't but looks like that command is used in older systems or systems without systemd. Besides "config clock" command, in our deployments we use DHCP for mgmt ip. When eth0 recieved mgmt-ip, it also invokes the timedatectl command and changes the timezone.

Thus, we should consider timedatectl as the command that is used to change timezone and work upon it to solve this issue

[nmoray]

@vivekrnv Okay. IMO, following change will not work out. When the container is rebooted, "timedatectl show -p Timezone --value" will always return the timezone set inside the container only. So it will eventually create the symlink based on the default timezone only.

-TZ=$(cat /etc/timezone)
+TZ=$(timedatectl show -p Timezone --value)

Presently, I can think of three options to solve this issue.

1. Use "dpkg-reconfigure tzdata" in place of timedatectl
2. Update /etc/timezone file on the host at the time of updating timezone
3. Directly mount /etc/localtime on to the container (It is not a recommended option :-) )

[nmoray]

@vivekrnv There is one better way. The method timezone_update() from hostcfgd is responsible for updating the timezone. We can update the /etc/timezone file inside this method.

    def timezone_update(self, data):
        """
        Apply timezone handler.
        Run the following command in Linux: timedatectl set-timezone <timezone>
        Args:
            data: Read table's key's data.
        """
        new_timezone = data.get('timezone')
        syslog.syslog(syslog.LOG_DEBUG,
                      f'DeviceMetaCfg: timezone update to {new_timezone}')

        if new_timezone is None:
            syslog.syslog(syslog.LOG_DEBUG,
                          f'DeviceMetaCfg: Recieved empty timezone')
            return

        if new_timezone == self.timezone:
            syslog.syslog(syslog.LOG_DEBUG,
                          f'DeviceMetaCfg: No change in timezone')
            return

        # run command will print out log error in case of error
        run_cmd(['timedatectl', 'set-timezone', new_timezone])
        self.timezone = new_timezone

        run_cmd(['systemctl', 'restart', 'rsyslog'], True, False)
        syslog.syslog(syslog.LOG_INFO, 'DeviceMetaCfg: Restart rsyslog after '
                      'changing timezone')

[nmoray]

@vivekrnv Please try out this fix and let me know if it works.

[vivekrnv]

@nmoray, I'm in favour of option 3, did some preliminary testing and the change is working. I know its not ideal but this should work as long as the container has tzdata package installed.

Problem with option 1 & 2, they can solve the "config cli" flow but not the timezone setting through DHCP flow.

Eg for the dhcp flow:

2024 Nov 20 11:44:05.949094 sonic INFO dhclient[13076]: XMT: Solicit on eth0, interval 18420ms.
2024 Nov 20 11:44:24.387003 sonic INFO dhclient[13076]: XMT: Solicit on eth0, interval 36390ms.
2024 Nov 20 11:44:33.473125 sonic INFO dbus-daemon[1089]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.4' (uid=0 pid=14055 comm="timedatectl set-timezone Israel")
2024 Nov 20 11:44:33.509363 sonic INFO systemd[1]: Starting systemd-timedated.service - Time & Date Service...
2024 Nov 20 11:44:33.620416 sonic INFO dbus-daemon[1089]: [system] Successfully activated service 'org.freedesktop.timedate1'
2024 Nov 20 11:44:33.620665 sonic INFO systemd[1]: Started systemd-timedated.service - Time & Date Service.
2024 Nov 20 11:44:33.623377 sonic INFO systemd-timedated[14056]: Changed time zone to 'Israel' (IDT).

Once there is a config reload, the rsyslog also adjusts. 

2024 Nov 20 11:45:10.009245 sonic INFO systemd[1]: Starting backend-acl.service - Enable backend acl on storage backend ToRs...
2024 Nov 20 13:45:10.009965 sonic INFO systemd[1]: Stopping rsyslog.service - System Logging Service...
2024 Nov 20 13:45:10.010806 sonic INFO rsyslogd: [origin software="rsyslogd" swVersion="8.2302.0" x-pid="7380" x-info="https://www.rsyslog.com"] exiting on signal 15.

Containers were adjusting till 2311 but broken from 202405.

For both flows to work, i could only think of option 3. I will raise a fix for this

[nmoray]

@vivekrnv one way to fix your DHCP flow is by overriding systemd-timedated.service. We can write the new timezone in /etc/timezone file as a part of ExecStartPost. This way, we can achieve our goal with minimal change.

[nmoray]

@vivekrnv for your reference,

root@CLS-DS2000-Leaf-1:~# cat /etc/nik_timezone 
America/Adak
root@CLS-DS2000-Leaf-1:~# timedatectl set-timezone America/Los_Angeles 
root@CLS-DS2000-Leaf-1:~# cat /etc/nik_timezone 
America/Los_Angeles
root@CLS-DS2000-Leaf-1:~# 
root@CLS-DS2000-Leaf-1:~# systemctl cat systemd-timedated.service 
# /lib/systemd/system/systemd-timedated.service
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Time & Date Service
Documentation=man:systemd-timedated.service(8)
Documentation=man:localtime(5)
Documentation=man:org.freedesktop.timedate1(5)

[Service]
BusName=org.freedesktop.timedate1
CapabilityBoundingSet=CAP_SYS_TIME
DeviceAllow=char-rtc r
ExecStart=/lib/systemd/systemd-timedated
IPAddressDeny=any
LockPersonality=yes
MemoryDenyWriteExecute=yes
NoNewPrivileges=yes
PrivateTmp=yes
ProtectProc=invisible
ProtectControlGroups=yes
ProtectHome=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectSystem=strict
ReadWritePaths=/etc
RestrictAddressFamilies=AF_UNIX
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
SystemCallArchitectures=native
SystemCallErrorNumber=EPERM
SystemCallFilter=@system-service @clock
WatchdogSec=3min
ExecStartPost=/bin/bash -c 'echo "$(timedatectl show -p Timezone --value)" > /etc/nik_timezone'

Note: I tested it on the earlier debian version so I used nik_timezone as a temporary file in place of /etc/timezone.

[vivekrnv]

/etc/timezone is deprecated in Debian. In SONiC, we should try to stay close to Debian for generic config and thus I’m not in favor in patching systemd to update /etc/timezone.