errno seemingly lost during fstatat when -target with -gnu is specified

[squeek502]

Zig Version

0.12.0-dev.244+f4c9e19bc

Steps to Reproduce and Observed Behavior

When linking glibc, the fstatat error seems to get lost, and some other error is returned from getErrno/c._errno().* instead (seen ESUCCESS and EEXIST, but expected ENOENT).

Test code:

const std = @import("std");

pub fn main() !void {
    {
        // this gives a bogus errno
        var stat = std.mem.zeroes(std.c.Stat);
        const rc = std.c.fstatat64(std.os.AT.FDCWD, "test_file_that_doesnt_exist", &stat, 0);
        const err = std.c.getErrno(rc);
        std.debug.print("glibc: {} {}\n", .{ rc, err });
    }

    {
        // this gives the correct errno
        var stat = std.mem.zeroes(std.os.linux.Stat);
        const rc = std.os.linux.fstatat(std.os.AT.FDCWD, "test_file_that_doesnt_exist", &stat, 0);
        const err = std.os.linux.getErrno(rc);
        std.debug.print("syscall: {} {}\n", .{ rc, err });
    }
}

built with:

zig build-exe fstatat.zig -lc -target x86_64-linux-gnu

output:

glibc: -1 os.linux.errno.generic.E.SUCCESS
syscall: 18446744073709551614 os.linux.errno.generic.E.NOENT

commented strace with only the relevant parts:

// std.c.fstatat64
newfstatat(AT_FDCWD, "test_file_that_doesnt_exist", 0x7ffc3675cfd8, 0) = -1 ENOENT (No such file or directory)
gettid()                                = 117441
write(2, "SUCCESS", 7SUCCESS)                  = 7

// std.os.linux.fstatat
newfstatat(AT_FDCWD, "test_file_that_doesnt_exist", 0x7ffc3675d0f8, 0) = -1 ENOENT (No such file or directory)
write(2, "NOENT", 5NOENT)                    = 5

(so both syscalls are the same, but somehow the glibc version doesn't

The same problem exists in the equivalent C code compiled by Zig:

#include <errno.h>
#include <stdio.h>
#include <sys/stat.h>
#include <fcntl.h>

int main() {
    struct stat statbuf;

    int rc = fstatat(AT_FDCWD, "test_file_that_doesnt_exist", &statbuf, 0);

    printf("%d %d\n", rc, errno);
}

$ zig build-exe fstatat.c -lc -target x86_64-linux-gnu -femit-bin=fstatat-c

$ ./fstatat-c
-1 0

$ ldd ./fstatat-c
    linux-vdso.so.1 (0x00007ffed69c3000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f77c7504000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f77c7312000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f77c754e000)

but the same C code works as expected when compiled with clang directly:

$ clang fstatat.c -o fstatat-clang

$ ./fstatat-clang
-1 2

$ ldd ./fstatat-clang
    linux-vdso.so.1 (0x00007ffd5dbc1000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc74afa3000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fc74b1c1000)

(2 is ENOENT)

The strace between the Zig-compiled C version and the Clang-compiled C version are the same:

// zig compiled
newfstatat(AT_FDCWD, "test_file_that_doesnt_exist", 0x7ffd8b185998, 0) = -1 ENOENT (No such file or directory)
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x1), ...}) = 0
brk(NULL)                               = 0x855000
brk(0x876000)                           = 0x876000
write(1, "-1 0\n", 5-1 0
)                   = 5

// clang compiled
newfstatat(AT_FDCWD, "test_file_that_doesnt_exist", 0x7fff3985d0b0, 0) = -1 ENOENT (No such file or directory)
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x1), ...}) = 0
brk(NULL)                               = 0x558433e61000
brk(0x558433e82000)                     = 0x558433e82000
write(1, "-1 2\n", 5-1 2
)                   = 5

Expected Behavior

fstatat errors to be accurately returned when linking glibc

[squeek502]

This actually only reproduces when the gnu abi is given explicitly via -target:

$ zig build-exe fstatat.c -lc -target native-native-gnu -femit-bin=fstatat-c-target
$ ./fstatat-c-target 
-1 0
$ ldd ./fstatat-c-target 
	linux-vdso.so.1 (0x00007ffcc63ae000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdae3991000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdae379f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fdae39db000)

Works fine without -target even though the same libc is linked AFAICT:

$ zig build-exe fstatat.c -lc -femit-bin=fstatat-c-notarget
$ ./fstatat-c-notarget 
-1 2
$ ldd fstatat-c-notarget 
	linux-vdso.so.1 (0x00007fff5ff81000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f181fd86000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f181fb94000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f181fdd0000)

EDIT: --show-builtin gives identical outputs for the -target and -target-less commands

EDIT#2: Here's a better/more understandable test case:

const std = @import("std");

test {
    try std.testing.expectError(error.FileNotFound, std.fs.cwd().statFile("test_file_that_doesnt_exist"));
}

Works with -lc without explicit -target (even though in theory this should be identical to -target native-native-gnu), works with musl abi, but doesn't work when -target is specified with the gnu abi:

$ zig test fstatat-os.zig -lc
All 1 tests passed.

$ zig test fstatat-os.zig -lc -target native-native-musl
All 1 tests passed.

$ zig test fstatat-os.zig -lc -target native-native-gnu
Test [1/1] test_0... expected error.FileNotFound, found fs.file.File.Stat{ .inode = 0, .size = 0, .mode = 0, .kind = fs.file.File.Kind.unknown, .atime = 0, .mtime = 0, .ctime = 0 }
Test [1/1] test_0... FAIL (TestUnexpectedError)
/home/ryan/Programming/zig/zig/lib/std/testing.zig:36:9: 0x2266b6 in expectError__anon_3058 (test)
        return error.TestUnexpectedError;
        ^
/home/ryan/Programming/zig/tmp/fstatat-os.zig:4:5: 0x226927 in test_0 (test)
    try std.testing.expectError(error.FileNotFound, std.fs.cwd().statFile("test_file_that_doesnt_exist"));
    ^
0 passed; 0 skipped; 1 failed.

[Jarred-Sumner]

we experienced a strange potentially related bug here: oven-sh/bun#4440

[rootbeer]

Stepping through the generated code in a debugger, the fstatat syscall executes, fails, returns the correct status code, then that code is converted into an errno and stashed in a thread-local variable. The call returns and Zig code fetches errno, but seems to fetch it from a different place than where errno was stored.

In the fstatat code the negative syscall return code is converted to an errno and saved with:

   0x00000000002b49f7 <+23>:    neg    %eax
   0x00000000002b49f9 <+25>:    mov    $0xfffffffffffffff8,%rcx
   0x00000000002b4a00 <+32>:    mov    %eax,%fs:(%rcx)

From what I understand the %fs register is the thread-local-storage segment, and this writes the errno value at -8 from the TLS pointer. I believe it is "normal" for a TLS pointer to point to the middle of the logical structure, so I think this is fine.

Invoking "write" in a similar manner to fstatat (e.g., var writeResult = std.c.write(99, "bad fd, no?", 12);) does not have a problem storing errno.

The write code for saving errno looks quite different:

   0x00007ffff7eb61a0 <+112>:   mov    0xd9c39(%rip),%rdx        # 0x7ffff7f8fde0
   0x00007ffff7eb61a7 <+119>:   neg    %eax
   0x00007ffff7eb61a9 <+121>:   mov    %eax,%fs:(%rdx)
   0x00007ffff7eb61ac <+124>:   mov    $0xffffffffffffffff,%rax

This looks up a constant at a RIP-relative address. GDB helpfully decodes the address in its comment there:

(gdb) p* 0x7ffff7f8fde0
$1 = -152

So I think this code stores the errno at -152 from the TLS pointer.

The errno-lookup code is:

(gdb) disassemble __errno_location
Dump of assembler code for function __GI___errno_location:
   0x00007ffff7de55e0 <+0>:     mov    0x1aa7f9(%rip),%rax        # 0x7ffff7f8fde0
   0x00007ffff7de55e7 <+7>:     add    %fs:0x0,%rax
   0x00007ffff7de55f0 <+16>:    ret

This is trying to return the address of errno, not the value, so it just adds the offset to the %fs base. This results in the same
-152 offset.

So the fstatat code is using the wrong errno …

Oh, the fstatat code isn't from the external C library. Its Zig-compiled code. (You can see the difference in the disassembly addresses above.) GDB says its compiled from zig/lib/libc/glibc/sysdeps/unix/sysv/linux/fstatat64.c.

I suspect how this can happen will be obvious to one or two people on the planet, but in case they're not paying attention I'll keep digging and see if I can figure out why the wrong fstatat is being used.

My test case is (compiled with -lc -target native-native-gnu on my x86_64 machine):

const std = @import("std");
const builtin = @import("builtin");

pub fn main() !void {
    var cwdFd = std.fs.cwd().fd;
    const stdout = std.io.getStdOut().writer();

    var addrErrno = std.c._errno();

    var stat = std.mem.zeroes(std.c.Stat);
    var statResult = std.c.fstatat(cwdFd, "test_file_that_doesnt_exist", &stat, 0);
    var sysErrno = std.c.getErrno(statResult);
    var cErrno = std.c._errno().*;

    try stdout.print("fstatat: Result={} sysErrno={} addrErrno={} cErrno={}\n", .{
        statResult, sysErrno, addrErrno, cErrno,
    });

    var writeResult = std.c.write(99, "bad fd, no?", 12);
    sysErrno = std.c.getErrno(writeResult);
    cErrno = std.c._errno().*;

    try stdout.print("write: Result={} sysErrno={} addrErrno={} cErrno={}\n", .{
        writeResult, sysErrno, addrErrno, cErrno,
    });
}

[rootbeer]

The --verbose-link option shows the object files used to build the executable:

$ ../zig/build/stage4/bin/zig build-exe confused-tls.zig -lc  -target native-native-gnu --verbose-link
LLD Link... ld.lld --error-limit=0 -O0 -z stack-size=16777216 --gc-sections --eh-frame-hdr -znow -m elf_x86_64 -o confused-tls /home/pat/.cache/zig/o/a45b68fa2cae11b917a784682ea7f49c/Scrt1.o /home/pat/.cache/zig/o/cdc9269eec429a886e126024dfd16237/crti.o -dynamic-linker /lib64/ld-linux-x86-64.so.2 confused-tls.o --as-needed /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/libm.so.6 /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/libpthread.so.0 /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/libc.so.6 /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/libdl.so.2 /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/librt.so.1 /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/libld.so.2 /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/libutil.so.1 /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/libresolv.so.2 /home/pat/.cache/zig/o/399bb63a0a17949c21c2be889f11532d/libc_nonshared.a /home/pat/.cache/zig/o/88a877fb4a624a9c9b292a039c202d2b/libcompiler_rt.a /home/pat/.cache/zig/o/0d21451ae78f87aae2c7e6e9697b31df/crtn.o

I wrote a short script to show the "fstatat" symbols ("nm | grep fstatat") in each of the object files:

From /home/pat/.cache/zig/o/a45b68fa2cae11b917a784682ea7f49c/Scrt1.o:
From /home/pat/.cache/zig/o/cdc9269eec429a886e126024dfd16237/crti.o:
From confused-tls.o:
                     U fstatat64
From /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/libm.so.6:
From /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/libpthread.so.0:
From /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/libc.so.6:
From /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/libdl.so.2:
From /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/librt.so.1:
From /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/libld.so.2:
From /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/libutil.so.1:
From /home/pat/.cache/zig/o/4239799cbcba63f88983e8241d2c8d3e/libresolv.so.2:
From /home/pat/.cache/zig/o/399bb63a0a17949c21c2be889f11532d/libc_nonshared.a:
                     U __fstatat64
                     U __fstatat64
                     U __fstatat64
    /home/pat/.cache/zig/o/411786b7364ebc09bb9202293c7a2dc6/fstatat.o:
    /home/pat/.cache/zig/o/b59d3ef63bc52d2b9f232e4d2442c16f/fstatat64.o:
    0000000000000000 T __fstatat
    0000000000000000 W fstatat
    0000000000000000 T __fstatat64
    0000000000000000 W fstatat64
    0000000000000000 T __GI___fstatat
From /home/pat/.cache/zig/o/88a877fb4a624a9c9b292a039c202d2b/libcompiler_rt.a:
From /home/pat/.cache/zig/o/0d21451ae78f87aae2c7e6e9697b31df/crtn.o:

So "fstatat64" is the undefined symbol in the main object file (confused-tls.o). Oddly the "libc.so.6" binary doesn't have any fstatat symbols. Then libc_nonshared.a has undefined references to "__fstatat64", and also includes "__fstatat64" and a weak "fstatat64". I believe this weak symbol is resolving the undefined symbol in the main binary.

I'll look into why there is no fstatat* in libc.so.6. It also seems like the libc_nonshared should also be using the real C library's errno? Or maybe the libc_nonshared should not be linked in if the system libc is being used?

[jacobly0]

I came to the same conclusion as above, and I found that this patch happens to prevent the wrong errno from being used:

diff --git a/src/glibc.zig b/src/glibc.zig
index cf12e8ea4..7179cfde4 100644
--- a/src/glibc.zig
+++ b/src/glibc.zig
@@ -353,7 +353,6 @@ pub fn buildCRTFile(comp: *Compilation, crt_file: CRTFile, prog_node: *std.Progr
                     "-D_LIBC_REENTRANT",
                     "-include",
                     try lib_path(comp, arena, lib_libc_glibc ++ "include" ++ path.sep_str ++ "libc-modules.h"),
-                    "-DMODULE_NAME=libc",
                     "-Wno-nonportable-include-path",
                     "-include",
                     try lib_path(comp, arena, lib_libc_glibc ++ "include" ++ path.sep_str ++ "libc-symbols.h"),

However, I have no idea if this change is correct or even makes sense.

[rootbeer]

It seems libc_nonshared.a is the correct, canonical location for the stat-related syscalls (see https://stackoverflow.com/questions/66701091/what-is-the-purpose-of-libc-nonshared-a), I thought this was a Zig-specific library.

So, removing the "-DMODULE_NAME=libc" from the libc_nonshared compile might be the right fix. Presumably that tells the nonshared library to use an external errno (instead of its own, local, static one). I'll see what I can dig up to verify this.

I'm very curious how you came to this diff. :)

[jacobly0]

I'm very curious how you came to this diff. :)

I just looked at the code and changed defines until it worked. :)

[rootbeer]

The errno.c that is linked into libc_nonshared (.{ .path = lib_libc_glibc ++ "csu" ++ s ++ "errno.c" }) looks a bit suspicious to me too, since it seems like libc_nonshared should not have its own errno (it should reference the dynamically linked one from the real libc).

The other thing that doesn't make sense is that the libc_nonshared.c fstat implementation was a raw syscall. The libc_noshared implementation is meant to forward to the "real" libc, via a non-standard "xstat()" implementation. (The whole point of the nonshared.a library is to link in stubs that jump into the dynamically linked library using a version-aware API....)

Hrm, it looks like the libc_nonshared (also called "static-only-routines" in the Makefiles) fstat* wrappers were excised from libc_nonshared in Oct 2020: https://sourceware.org/git/?p=glibc.git;a=commit;h=4d97cc8cf3da925fd06fc37d4daebafce3247719

Yeah, my local (Debian) libc_nonshared.a doesn't have the "stat" family of functions in it:

$ nm /usr/lib/x86_64-linux-gnu/libc_nonshared.a

at_quick_exit.oS:
0000000000000000 T at_quick_exit
                 U __cxa_at_quick_exit
                 U __dso_handle

atexit.oS:
0000000000000000 T atexit
                 U __cxa_atexit
                 U __dso_handle

pthread_atfork.oS:
                 U __dso_handle
0000000000000000 T __pthread_atfork
0000000000000000 W pthread_atfork
                 U __register_atfork

stack_chk_fail_local.oS:
                 U __stack_chk_fail
0000000000000000 T __stack_chk_fail_local

They're in libc.so.6 now:

$ readelf -a /usr/lib/x86_64-linux-gnu/libc.so.6 | grep fstatat
   591: 00000000000f7650    42 FUNC    WEAK   DEFAULT   16 fstatat@@GLIBC_2.33
   705: 00000000000f7650    42 FUNC    WEAK   DEFAULT   16 fstatat64@@GLIBC_2.33

Here's some historical Zig context: #11137 I found too.

Having Zig clone the /build system/ for glibc and keep in sync with internal changes like this seems like a losing game. Problems in this area are usually more subtle than losing errno because they don't break anything in the short-term. They just create future backwards compatibility problems.

[rootbeer]

Here's a half a draft change to update the libc_nonshared stub as far as I understand how it works. I'm not sure how to add the symbols removed from libc_nonshared (fstat*) to the libc.so though: #17521