systemd timers treat all timezones as UTC

[ncfavier]

Describe the bug

The OnCalendar field of systemd timer units is treated as UTC, even if a different timezone is specified.

To Reproduce

1. Declare a trivial service and make it start at midnight on some non-UTC timezone:

{                                                                                            
  systemd.services.foo = {                                                                   
    script = "true";                                                                         
    startAt = "00:00:00 Pacific/Auckland";                                                      
  };                                                                                         
}

2. Run systemctl status foo.timer and observe that the Trigger field says midnight UTC (in my case, Fri 2020-11-27 01:00:00 CET).

Expected behavior

In this case, the Trigger field should say Fri 2020-11-27 12:00:00 CET (or Fri 2020-11-27 11:00:00 UTC).

Additional context

• I have tested this with two different non-UTC timezones.
• systemd-analyze calendar "00:00:00 Pacific/Auckland" on the same machine gives the correct result, so this is linked to timer units.
• I am using the systemd package from the NixOS 20.09 release, version 246.
• I could not reproduce the issue on an Arch Linux server (systemd 246.4-1-arch), so it's (probably) a NixOS problem.

Notify maintainers

@edolstra @andir @flokli @kloenk

Metadata

 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.64, NixOS, 20.09.20201125.7a7bae5 (Nightingale)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.4pre20201118_79aa7d9`
 - channels(root): `"nixos-20.09.1721.896270d629e"`
 - channels(n): `"home-manager"`
 - nixpkgs: `/nix/store/c0ri3lxx3ymf3spqlhz7dkycmf8fwd3w-source`

[flokli]

Maybe our patch in 0009-Change-usr-share-zoneinfo-to-etc-zoneinfo.patch (from #80038 (comment)) isn't sufficient and this needs to be patched in another location, too?

[ncfavier]

Note that if I specify a timezone that doesn't exist, I get an error on activation. So presumably it can find the timezone info, it just doesn't respect it.

[flokli]

Can you reproduce on non-NixOS?

[ncfavier]

Nope, see the fourth bullet point under Additional context.

[flokli]

I was able to reproduce this on NixOS, and verified it works correctly on Archlinux.

I still don't understand why it behaves the way it does (it seems we did properly patch all occurences this might touch).

This might require some deeper digging into the codebase, and adding some logs along the way to see what's going on.

[stale]

I marked this as stale due to inactivity. → More info

[flokli]

Not stale, but help welcome ;-)

[teh]

I haven't had time to test this but a cursory reading of the systemd code shows that it uses tzset():

https://github.com/systemd/systemd/blob/main/src/shared/calendarspec.c#L1411

tzset uses TZDIR. On a normal nixos system I get this:

$ export | rg TZ
declare -x TZDIR="/etc/zoneinfo

but pid 1 doesn't set TZDIR:

$ cat /proc/1/environ | tr '\00' '\n'
LOCALE_ARCHIVE=/run/current-system/sw/lib/locale/locale-archive
PATH=/run/current-system/systemd/lib/systemd:/nix/store/5kp6328y6wgws17anpl2wvc8w6yny0wc-nfs-utils-2.5.1/bin:/nix/store/nva6mlbsb1gv6q60xblk4dbspyg2l51j-e2fsprogs-1.45.6-bin/bin:/nix/store/gf088si5zlfryb0pmw5mx8x6ymf16lq2-btrfs-progs-5.9/bin:/nix/store/9kfd3p0s56bzqwzcapzm5wqdyqy17v72-dosfstools-4.1/bin
PWD=/
HOME=/root
SHLVL=0

[teh]

I confirmed TZDIR is the issue by applying the following patch. I think my fix is wrong but I couldn't figure out where in systemd pid 1 I should set env vars so I did id in a function that I know will be called.

diff --git a/src/shared/calendarspec.c b/src/shared/calendarspec.c
index bf24d8d5bb..2ae788f21e 100644
--- a/src/shared/calendarspec.c
+++ b/src/shared/calendarspec.c
@@ -901,6 +901,8 @@ int calendar_spec_from_string(const char *p, CalendarSpec **spec) {
                 const char *e = NULL;
                 int j;

+		setenv("TZDIR", "/etc/zoneinfo", 1);
+
                 tzset();

                 /* Check if the local timezone was specified? */

without patch:

NEXT                        LEFT       LAST PASSED UNIT                         ACTIVATES
Sun 2021-10-03 09:18:23 UTC 14min left n/a  n/a    systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Mon 2021-10-04 09:00:00 UTC 23h left   n/a  n/a    oncalendar-test.timer        oncalendar-test.service

with patch:

NEXT                        LEFT          LAST PASSED UNIT                         ACTIVATES
Sun 2021-10-03 09:18:48 UTC 14min left    n/a  n/a    systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Sun 2021-10-03 13:00:00 UTC 3h 56min left n/a  n/a    oncalendar-test.timer        oncalendar-test.service

unit used:

    systemd.services.oncalendar-test = {
      description = "xx";
      serviceConfig.Type = "oneshot";
      wantedBy = [ "default.target" ];
      startAt = "09:00 America/New_York";
      script = "touch $HOME/xx";
    };```

[ncfavier]

I wonder if this is related to #65415 and #89487

[flokli]

Yeah, this looks about right. Maybe we should see if we can steer glibc to look for zoneinfo in these two paths by default, instead of in the glibc output directly.