Compilation outputs contain absolute paths that can't be stripped

[fmeum]

Steps to reproduce:

$ cd examples/rules_cc
$ bazel build //:hello_world
$ $(bazel info output_base)/external/zig_sdk/tools/x86_64-linux-gnu.2.34/c++ -o out.o -c hello_world.cc -ffile-prefix-map=$(bazel info output_base)=redacted && strings out.o | grep zig_sdk | grep -v redacted
/home/fhenneke/.cache/bazel/_bazel_fhenneke/5ac561ed54d4f15173255e62293f6537/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/ios
/home/fhenneke/.cache/bazel/_bazel_fhenneke/5ac561ed54d4f15173255e62293f6537/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/string
/home/fhenneke/.cache/bazel/_bazel_fhenneke/5ac561ed54d4f15173255e62293f6537/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/ostream
/home/fhenneke/.cache/bazel/_bazel_fhenneke/5ac561ed54d4f15173255e62293f6537/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/__iterator/ostreambuf_iterator.h
/home/fhenneke/.cache/bazel/_bazel_fhenneke/5ac561ed54d4f15173255e62293f6537/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/__memory/allocator.h
/home/fhenneke/.cache/bazel/_bazel_fhenneke/5ac561ed54d4f15173255e62293f6537/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/streambuf
/home/fhenneke/.cache/bazel/_bazel_fhenneke/5ac561ed54d4f15173255e62293f6537/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/__memory/compressed_pair.h
/home/fhenneke/.cache/bazel/_bazel_fhenneke/5ac561ed54d4f15173255e62293f6537/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/locale
/home/fhenneke/.cache/bazel/_bazel_fhenneke/5ac561ed54d4f15173255e62293f6537/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/__locale
'const union (anonymous union at /home/fhenneke/.cache/bazel/_bazel_fhenneke/5ac561ed54d4f15173255e62293f6537/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/string:784:9)'
'const struct (anonymous struct at /home/fhenneke/.cache/bazel/_bazel_fhenneke/5ac561ed54d4f15173255e62293f6537/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/string:761:9)'
'union (anonymous union at /home/fhenneke/.cache/bazel/_bazel_fhenneke/5ac561ed54d4f15173255e62293f6537/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/string:784:9)'

As you can see in the logs above, some references to headers shipped with the toolchain are not affected by -ffile-prefix-map. I haven't found a way to work around this issue yet.

This directly affects usage with rules_go as a build of the standard library with CGo enabled changes the working directory from the Bazel execroot to various temporary directories, which results in absolute paths to header files being included in the stdlib archives.

Full list

    hermeticity_test.go:41: /home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/56/execroot/io_bazel_rules_go/bazel-out/k8-fastbuild/bin/tests/core/stdlib/hermeticity_test_/hermeticity_test.runfiles/io_bazel_rules_go/stdlib_/pkg/linux_amd64_race/os/user.a leaks an absolute path:
        "g\x0f\xb9@\x16H\x8bE\xc0H\x83\xc0\bH\x89E\xb0H\x83\xf8\x00\x0f\x85\x05\x00\x00\x00g\x0f\xb9@\x16H\x8bE\xb0H\x8bM\xb8H\x89\bH\x83\xc4P]\xc3/home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/46/execroot/io_bazel_rules_go/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/pwd.h\x00io_bazel_rules_go/bazel-out/k8-fas"
    hermeticity_test.go:41: /home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/56/execroot/io_bazel_rules_go/bazel-out/k8-fastbuild/bin/tests/core/stdlib/hermeticity_test_/hermeticity_test.runfiles/io_bazel_rules_go/stdlib_/pkg/linux_amd64_race/os/user.a leaks an absolute path:
        "075c99/bin/stdlib_/src/os/user/cgo_lookup_cgo.go\x00/home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/46/execroot/io_bazel_rules_go/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/string.h\x00cgo-gcc-prolog\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\x00\x00'st"
    hermeticity_test.go:41: /home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/56/execroot/io_bazel_rules_go/bazel-out/k8-fastbuild/bin/tests/core/stdlib/hermeticity_test_/hermeticity_test.runfiles/io_bazel_rules_go/stdlib_/pkg/linux_amd64_race/plugin.a leaks an absolute path:
        "dlib_/src/plugin/plugin_dlopen.go\x00cgo-gcc-prolog\x00/home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/46/execroot/io_bazel_rules_go/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/dlfcn.h\x00\x00\x00\x00\xff\xff\x00\x00'struct (unnamed struct a"
    hermeticity_test.go:41: /home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/56/execroot/io_bazel_rules_go/bazel-out/k8-fastbuild/bin/tests/core/stdlib/hermeticity_test_/hermeticity_test.runfiles/io_bazel_rules_go/stdlib_/pkg/linux_amd64_race/runtime/cgo.a leaks an absolute path:
        "\xa8\x01\x0f\x85\x05\x00\x00\x00g\x0f\xb9@\x10H\x8d=\x00\x00\x00\x00\xe8\x00\x00\x00\x00H\x8bE\xf8H\x83\xc4\x10]\xc3gcc_libinit.c\x00/home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/46/execroot/io_bazel_rules_go/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/pthread.h\x00pthread_create failed: %s\x00\x00\x00\x00\x00"
    hermeticity_test.go:41: /home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/56/execroot/io_bazel_rules_go/bazel-out/k8-fastbuild/bin/tests/core/stdlib/hermeticity_test_/hermeticity_test.runfiles/io_bazel_rules_go/stdlib_/pkg/linux_amd64_race/runtime/cgo.a leaks an absolute path:
        "H\x8bE\xd8H\x8b8H\x8b5\x00\x00\x00\x00H\x8bU\xe0\xe8\x00\x00\x00\x001\xc0H\x83\xc40]\xc3gcc_linux_amd64.c\x00/home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/46/execroot/io_bazel_rules_go/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/signal.h\x00malloc failed: %s\x00/home/fhennek"
    hermeticity_test.go:41: /home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/56/execroot/io_bazel_rules_go/bazel-out/k8-fastbuild/bin/tests/core/stdlib/hermeticity_test_/hermeticity_test.runfiles/io_bazel_rules_go/stdlib_/pkg/linux_amd64_race/runtime/cgo.a leaks an absolute path:
        "include/generic-glibc/signal.h\x00malloc failed: %s\x00/home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/46/execroot/io_bazel_rules_go/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/pthread.h\x00bad stack bounds: lo=%p hi=%p\n"
    hermeticity_test.go:41: /home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/56/execroot/io_bazel_rules_go/bazel-out/k8-fastbuild/bin/tests/core/stdlib/hermeticity_test_/hermeticity_test.runfiles/io_bazel_rules_go/stdlib_/pkg/linux_amd64_race/runtime/cgo.a leaks an absolute path:
        "\x00\x00\x00g\x0f\xb9@\x16H\x8bE\xf0H\x8b\x00H\x89E\xe8H\x83\xf8\x00\x0f\x85\x05\x00\x00\x00g\x0f\xb9@\x10H\x8b}\xe8\xe8\x00\x00\x00\x00H\x83\xc4 ]\xc3/home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/46/execroot/io_bazel_rules_go/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/stdlib.h\x00gcc_setenv.c\x00\x00\x00\x00\xff\xff\x00\x00'char *'\x00\x00\x00"
    hermeticity_test.go:41: /home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/56/execroot/io_bazel_rules_go/bazel-out/k8-fastbuild/bin/tests/core/stdlib/hermeticity_test_/hermeticity_test.runfiles/io_bazel_rules_go/stdlib_/pkg/linux_amd64_race/runtime/cgo.a leaks an absolute path:
        "Ȩ\x01\x0f\x85\x05\x00\x00\x00g\x0f\xb9@\x16H\x8b\x85\x90\xfd\xff\xffH\x8b\x8d\x98\xfd\xff\xffH\x89\b\x8bE܉E\xfc\x8bE\xfcH\x81\xc4p\x02\x00\x00]\xc3/home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/46/execroot/io_bazel_rules_go/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/signal.h\x00gcc_sigaction.c\x00/home/fhenneke/"
    hermeticity_test.go:41: /home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/56/execroot/io_bazel_rules_go/bazel-out/k8-fastbuild/bin/tests/core/stdlib/hermeticity_test_/hermeticity_test.runfiles/io_bazel_rules_go/stdlib_/pkg/linux_amd64_race/runtime/cgo.a leaks an absolute path:
        "c/include/generic-glibc/signal.h\x00gcc_sigaction.c\x00/home/fhenneke/.cache/bazel/_bazel_fhenneke/da2ee2360ba14ac857bdb47dfbba9b7d/sandbox/linux-sandbox/46/execroot/io_bazel_rules_go/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/string.h\x00\x00\x00\x00\x00\x00\x00\xff\xff\x00\x00'const go_sigaction_t"
    hermeticity_test.go:50: Found 9 absolute paths

[jvolkman]

I tried tracing this down and it seemed like even when passing relative paths to c++, zig upstream was inserting these absolute paths in outputs. They do go away with -O2, or -c opt to bazel.

[jvolkman]

I get the same reproduction when invoking via a relative path:

$ bazel-rules_cc/external/zig_sdk/tools/x86_64-linux-gnu.2.34/c++ -o out.o -c hello_world.cc -ffile-prefix-map=$(bazel info output_base)=redacted && strings out.o | grep zig_sdk | grep -v redacted
/home/jvolkman/.cache/bazel/_bazel_jvolkman/3aa3feaef46d2c24bea3a7828d24c330/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/locale
/home/jvolkman/.cache/bazel/_bazel_jvolkman/3aa3feaef46d2c24bea3a7828d24c330/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/string
/home/jvolkman/.cache/bazel/_bazel_jvolkman/3aa3feaef46d2c24bea3a7828d24c330/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/__memory/allocator.h
/home/jvolkman/.cache/bazel/_bazel_jvolkman/3aa3feaef46d2c24bea3a7828d24c330/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/ostream
/home/jvolkman/.cache/bazel/_bazel_jvolkman/3aa3feaef46d2c24bea3a7828d24c330/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/ios
/home/jvolkman/.cache/bazel/_bazel_jvolkman/3aa3feaef46d2c24bea3a7828d24c330/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/__locale
/home/jvolkman/.cache/bazel/_bazel_jvolkman/3aa3feaef46d2c24bea3a7828d24c330/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/__iterator/ostreambuf_iterator.h
/home/jvolkman/.cache/bazel/_bazel_jvolkman/3aa3feaef46d2c24bea3a7828d24c330/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/__memory/compressed_pair.h
/home/jvolkman/.cache/bazel/_bazel_jvolkman/3aa3feaef46d2c24bea3a7828d24c330/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/streambuf
'const union (anonymous union at /home/jvolkman/.cache/bazel/_bazel_jvolkman/3aa3feaef46d2c24bea3a7828d24c330/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/string:784:9)'
'const struct (anonymous struct at /home/jvolkman/.cache/bazel/_bazel_jvolkman/3aa3feaef46d2c24bea3a7828d24c330/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/string:761:9)'
'union (anonymous union at /home/jvolkman/.cache/bazel/_bazel_jvolkman/3aa3feaef46d2c24bea3a7828d24c330/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libcxx/include/string:784:9)'

[jvolkman]

Actually @fmeum I take that back. Zig uses its own cache (which defaults to /tmp/zig-cache here). If I remove that before switching to a relative path, the absolute paths are gone. And leaving the cache with relative paths but switching invocation to using an absolute path then also results in the relative paths remaining.

Lots of moving pieces here, unfortunately.

[motiejus]

This has nothing to do with zig's internal cache directory, as the headers are in zig's stdlib. If it were zig's cache, it would start with /tmp/zig-cache.

Optimization flag is a red herring; this is UBSAN:

$ /code/zig/build/stage3/bin/zig c++ -fno-sanitize=undefined hello_world.cc -o hello_world.o -c && strings *.o | grep ^/code/zig/
$ /code/zig/build/stage3/bin/zig c++                         hello_world.cc -o hello_world.o -c && strings *.o | grep ^/code/zig/
/code/zig/lib/libcxx/include/__locale
/code/zig/lib/libcxx/include/__memory/allocator.h
/code/zig/lib/libcxx/include/__iterator/ostreambuf_iterator.h
/code/zig/lib/libcxx/include/streambuf
/code/zig/lib/libcxx/include/string
/code/zig/lib/libcxx/include/ios
/code/zig/lib/libcxx/include/__memory/compressed_pair.h
/code/zig/lib/libcxx/include/ostream
/code/zig/lib/libcxx/include/locale

Optimizations (-c opt) turn off UBSAN, which leads to the same effect. I am unable to reproduce these paths when using plain clang++-16 -fsanitize=undefined yet.

[motiejus]

It does pas the -ffile-prefix-map parameter to the underlying clang. If you grep for redacted in the resulting object file, it will be there.

I have been trying to narrow down the difference between clang++-16 and zig c++ invocations; removing and changing the flags until they are the same. I am at a point where the underlying command-line invocation is the same, except the include paths are different

clang:

clang++-16 -v -fsanitize=undefined -ffile-prefix-map=/usr=redacted -g -I/usr/include/c++/11 -I/usr/include/x86_64-linux-gnu/c++/11 hello_world.cc -o hello_world.o -c && strings *.o | grep '^/\|redacted'

zig cc:

rm -fr ~/.cache/zig; ZIG_VERBOSE_CC=1 /code/zig/build/stage3/bin/zig c++  -fmacro-prefix-map=/code/zig=redacted -fcoverage-prefix-map=/code/zig=redacted -fdebug-prefix-map=/code/zig=redacted  hello_world.cc -o hello_world.o -c && strings *.o | grep '^/\|redacted'

Here are the invocations:
clang.txt
zig-cc.txt

And the diff of the commands:

Output of zig.txt:

/code/zig/lib/libcxx/include/ostream
/code/zig/lib/libcxx/include/locale
/code/zig/lib/libcxx/include/ios
/code/zig/lib/libcxx/include/streambuf
/code/zig/lib/libcxx/include/string
/code/zig/lib/libcxx/include/__memory/compressed_pair.h
/code/zig/lib/libcxx/include/__iterator/ostreambuf_iterator.h
/code/zig/lib/libcxx/include/__locale
/v      .
/v      /
/v      J
/v      K
/v      L
redacted/hermetic_cc_toolchain/examples/rules_cc
redacted/hermetic_cc_toolchain/examples/rules_cc
redacted/zig/lib/libcxx/include/__string
redacted/zig/lib/include
redacted/zig/lib/libcxx/include/__iterator
redacted/zig/lib/libcxx/include
redacted/zig/lib/libcxx/include/__memory
redacted/zig/lib/libcxx/include/__type_traits
/usr/include/x86_64-linux-gnu/bits/types
/usr/include
/usr/include/x86_64-linux-gnu/bits
redacted/zig/lib/libcxx/include/__chrono

Output of clang.txt:

/code/hermetic_cc_toolchain/examples/rules_cc
/code/hermetic_cc_toolchain/examples/rules_cc
redacted/include/c++/11
redacted/include/x86_64-linux-gnu/bits/types
redacted/include
redacted/lib/llvm-16/lib/clang/16/include
redacted/include/c++/11/bits
redacted/include/c++/11/debug
redacted/include/x86_64-linux-gnu/bits

As we can see from the redacted/ lines in the output of zig, it is applying the ffile-prefix-map. However, it is also adding the paths to libcxx headers for some reason, but only if -fsanitize is non-empty AND libcxx is in non-standard directory.

[motiejus]

_LIBCPP_HAS_NO_VENDOR_AVAILABILITY_ANNOTATIONS is suspicious. If I remove it, the zig version does not compile. This is the option's definition from llvm-project:

option(LIBCXX_ENABLE_VENDOR_AVAILABILITY_ANNOTATIONS
  "Whether to turn on vendor availability annotations on declarations that depend
   on definitions in a shared library. By default, we assume that we're not building
   libc++ for any specific vendor, and we disable those annotations. Vendors wishing
   to provide compile-time errors when using features unavailable on some version of
   the shared library they shipped should turn this on and see `include/__availability`
   for more details." OFF)

Edit: nope. I turned on this macro in clang and it still does not output the header paths.

[motiejus]

The next step would be to inspect the resulting object file hello_world.o and see where this string /code/zig/lib/libcxx/include/__iterator/ostreambuf_iterator.h is defined (probably strtab) and where is it referenced from (sounds hard).

hello_world.pptx

[jvolkman]

I found the -fsanitize-undefined-strip-path-components option which seems to affect these paths. Passing -fsanitize-undefined-strip-path-components=-1 strips all but the actual file name.

[motiejus]

I found the -fsanitize-undefined-strip-path-components option which seems to affect these paths. Passing -fsanitize-undefined-strip-path-components=-1 strips all but the actual file name.

This is a great find! I am still puzzled on why this does not reproduce on clang though (it does not have that flag).

I am open to put it to the default cflags in fastbuild. Thoughts?

Edit: how about -fsanitize-undefined-strip-path-components=-4?

[fmeum]

Nice find! Would this also be needed in the dbg configuration?

[jvolkman]

Edit: how about -fsanitize-undefined-strip-path-components=-4?

Is there anything special about -4? Certainly -1 isn't very useful, but depending on the use case (e.g., this rules_go use case where the cwd is changed), -4 could bleed into the random part of the path.

[fmeum]

It's going to be hard to set this properly in a generic way. It would be much better if something like -ffile-prefix-map applied to UBSan paths. Maybe we should file an issue at either zig or LLVM?

[motiejus]

Edit: how about -fsanitize-undefined-strip-path-components=-4?

Is there anything special about -4? Certainly -1 isn't very useful, but depending on the use case (e.g., this rules_go use case where the cwd is changed), -4 could bleed into the random part of the path.

Looking at the example paths, the 4 "final" ones have not bled, but that's just anecdotal and not guaranteed to remain such. I take it back, -1 is good.

It's going to be hard to set this properly in a generic way. It would be much better if something like -ffile-prefix-map applied to UBSan paths. Maybe we should file an issue at either zig or LLVM?

Feel free, but be prepared to dive deeper once you get some guidance from those folks. :)