Improve LibC interface in build.zig

[ikskuh]

Right now, the --libc [file] command line argument isn't really well exposed in std.Build.Step.Compile:

zig/lib/std/Build/Step/Compile.zig

Line 87 in 4558996

libc_file: ?LazyPath = null,

This implementation doesn't really give us the options to pass down ad-hoc compiled libcs like Foundation libc or newlib inside build.zig.

The fields available in a libc.txt file are these:

# The directory that contains `stdlib.h`.
# On POSIX-like systems, include directories be found with: `cc -E -Wp,-v -xc /dev/null`
include_dir=/nix/store/i58yz1rxjxpha40l17hgg7cz62jck9q3-glibc-2.38-77-dev/include

# The system-specific include directory. May be the same as `include_dir`.
# On Windows it's the directory that includes `vcruntime.h`.
# On POSIX it's the directory that includes `sys/errno.h`.
sys_include_dir=/nix/store/i58yz1rxjxpha40l17hgg7cz62jck9q3-glibc-2.38-77-dev/include

# The directory that contains `crt1.o` or `crt2.o`.
# On POSIX, can be found with `cc -print-file-name=crt1.o`.
# Not needed when targeting MacOS.
crt_dir=/nix/store/j0by58xwyc66f884x0q8rpzvgpwvjmf2-glibc-2.38-77/lib

# The directory that contains `vcruntime.lib`.
# Only needed when targeting MSVC on Windows.
msvc_lib_dir=

# The directory that contains `kernel32.lib`.
# Only needed when targeting MSVC on Windows.
kernel32_lib_dir=

# The directory that contains `crtbeginS.o` and `crtendS.o`
# Only needed when targeting Haiku.
gcc_dir=

These options could be exposed inside a struct std.Build.LibC:

pub const LibC = struct {
    /// The directory that contains `stdlib.h`.
    /// On POSIX-like systems, include directories be found with: `cc -E -Wp,-v -xc /dev/null`
    include_dirs: []const LazyPath,

    /// A list of additional object files or static libraries that might be linked with the final executable.
    /// These objects are required when using custom libc files.
    link_objects: []const LazyPath = &.{},

    /// The system-specific include directory. May be the same as `include_dir`.
    /// On Windows it's the directory that includes `vcruntime.h`.
    /// On POSIX it's the directory that includes `sys/errno.h`.
    sys_include_dirs: []const LazyPath = &.{},

    /// The directory that contains `crt1.o` or `crt2.o`.
    /// On POSIX, can be found with `cc -print-file-name=crt1.o`.
    /// Not needed when targeting MacOS.
    crt_dir: ?LazyPath = null,

    /// The directory that contains `vcruntime.lib`.
    /// Only needed when targeting MSVC on Windows.
    msvc_lib_dir: ?LazyPath = null,

    /// The directory that contains `kernel32.lib`.
    /// Only needed when targeting MSVC on Windows.
    kernel32_lib_dir: ?LazyPath = null,

    /// The directory that contains `crtbeginS.o` and `crtendS.o`
    /// Only needed when targeting Haiku.
    gcc_dir: ?LazyPath = null,
};

which could be used like this then:

- libc_file: ?LazyPath = null, 
+ libc: ?std.Build.LibC = null, 

Pre-defined libc:

pub fn build(b: *std.Build) void {
    // Emulate "--libc custom.txt":
    const libc = b.parseLibCFile(p.path("custom.txt"));

    const exe = b.addExecutable(.{
        .libc = libc,
        …
    });
}

Custom libc:

pub fn build(b: *std.Build) void {
    const foundation_libc = foundation_mod.artifact("foundation");

    const libc = std.Build.LibC {
        .include_dirs = &.{
            foundation_libc.getEmittedIncludeTree(),
        },
        .link_objects = &.{
            foundation_libc.getEmittedBin(),
        },
    };

    const exe = b.addExecutable(.{
        .libc = libc,
        …
    });
}

[pfgithub]

This would be nice for linking newlib, currently I link it directly with linkLibrary, but then zig doesn't know that libc is linked and stuff like std.heap.c_allocator doesn't work.

The way you described it with parseLibCFile wouldn't quite work because contents of the file can't be known LazyPath dependencies are resolved. It's fine with b.path(), but for a generated file it wouldn't work. It could work to make a step that generates libc.txt or to use pointers to std.Build.LibC and add dependencies into the struct that get resolved before data in the struct is used, like LazyPath.

[haydenridd]

Seems related and along the same lines as:
#19340

I currently manually link in the pre-compiled newlib-nano provided by the arm-none-eabi-gcc compiler as I'm targeting embedded targets. Being able to generate a custom libC struct would be nice and a lot more clear than what I'm currently doing, which is using arm-none-eabi-gcc itself to tell me where system paths are:
https://github.com/haydenridd/stm32-zig-porting-guide/blob/main/build.zig

Something else mentioned in the other issue I ran into is even when I do setup my libc file correctly targeting the bundled newlib-nano I get the following:

error: ld.lld: unable to find library -lm
error: ld.lld: unable to find library -lpthread
error: ld.lld: unable to find library -lc
error: ld.lld: unable to find library -ldl
error: ld.lld: unable to find library -lrt
error: ld.lld: unable to find library -lutil

Being able to control which libraries are linked in from libc would be nice, as in my case I'm targeting freestanding and don't have pthread (among others)

[linusg]

which could be used like this then:

I think you have a typo there?

- libc_file: ?LazyPath = null, 
+ libc: ?std.Build.LibC = null, 

[ikskuh]

@pfgithub:

The way you described it with parseLibCFile wouldn't quite work because contents of the file can't be known LazyPath dependencies are resolved. It's fine with b.path(), but for a generated file it wouldn't work. It could work to make a step that generates libc.txt or to use pointers to std.Build.LibC and add dependencies into the struct that get resolved before data in the struct is used, like LazyPath.

parseLibCFile is only for statically known libc files, otherwise you can conveniently construct the std.Build.LibC object as any other Zig structure :)

[GalaxyShard]

As a workaround, it is currently possible to use Zig-packaged libc's without hard-coding the paths by using a helper program to dynamically generate a libc.txt file from LazyPaths.

Example code:

// make-libc-file.zig
const std = @import("std");

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    defer _ = gpa.deinit();

    const allocator = gpa.allocator();

    const args = try std.process.argsAlloc(allocator);
    defer std.process.argsFree(allocator, args);

    var output = std.io.getStdOut();

    const valid_keys = .{ "include_dir", "sys_include_dir", "crt_dir", "msvc_lib_dir", "kernel32_lib_dir", "gcc_dir" };

    for (args[1..]) |arg| {
        var iter = std.mem.splitScalar(u8, arg, '=');
        const key = iter.next() orelse return error.InvalidOrMissingKey;
        const value = iter.next() orelse return error.InvalidOrMissingValue;

        inline for (valid_keys) |valid_key| {
            if (std.mem.eql(u8, valid_key, key)) {
                try output.writeAll(valid_key ++ "=");
                try output.writeAll(value);
                try output.writeAll("\n");
            }
        }
    }
}

Example usage:

    const libc_file_builder = b.addExecutable(.{
        .name = "libc_file_builder",
        .target = b.resolveTargetQuery(.{}), // native
        .optimize = .Debug,
        .root_source_file = b.path("make-libc-file.zig"),
    });
    const make_libc_file = b.addRunArtifact(libc_file_builder);
    make_libc_file.addArg("include_dir=/hardcoded/example");
    make_libc_file.addPrefixedDirectoryArg("sys_include_dir=", b.path("example_sys_include_dir"));
    make_libc_file.addPrefixedDirectoryArg("crt_dir=", crt_directory_lazy_path);
    b.default_step.dependOn(&make_libc_file.step);
    exe.setLibCFile(make_libc_file.captureStdOut());

It would be much simpler if this proposal were implemented. I could take a shot at implementing it if this proposal is accepted.

[pfgithub]

There's another problem with setLibCFile. There's no way to pass it down through dependency trees. If you're depending on freetype, how are you supposed to tell it to use a different libc file for its addStaticLibrary? I'm not sure what the solution to this is. The way it is now, every library that calls linkLibC would have to expose options for overriding the libc file which doesn't seem resonable to ask every library to do

[GalaxyShard]

There's another problem with setLibCFile. There's no way to pass it down through dependency trees. If you're depending on freetype, how are you supposed to tell it to use a different libc file for its addStaticLibrary? I'm not sure what the solution to this is. The way it is now, every library that calls linkLibC would have to expose options for overriding the libc file which doesn't seem resonable to ask every library to do

This problem seems more general than setLibCFile; after all, even optimization and target options have to be manually exposed by libraries. Personally I think it may be reasonable to make custom libc a target option though, passing it down with the usual standardTargetOptions, though that may be a seperate issue.

[jayschwa]

Portable libraries written in C often use something like a configure script or CMake to check what headers or symbols exist. If this proposal were implemented, there probably ought to be a way to perform these checks in the Zig build system too.