What is the process to 'vendor' third party dependencies?

[svermeulen]

When using bazel, I would often use http_archive or new_git_repository and then provide a git patch file to modify it, and also a build file to add to it. I see that buck2 also has http_archive but it's unclear how it's meant to be used. I have not been able to find any documentation for common practices to follow when 'vendoring' third party dependencies like this. Do any docs exist on this?

[dtolnay]

https://github.com/dtolnay/cxx/blob/f4453bc6bda5742a804732e3212c284272374623/third-party/BUCK has an example of http_archive. You can run git submodule update --init followed by buck2 build //third-party: in that repository to try it out.

Git dependencies are supported a very similar way, using https://buck2.build/docs/api/rules/#git_fetch. Buck2 itself uses them. You can run reindeer buckify in this directory and you'll see the generated BUCK file contains some, corresponding to this and this git dependency. (Reindeer: https://github.com/facebookincubator/reindeer.)

For vendoring, using https://github.com/dtolnay/cxx as an example, you can run cargo vendor --versioned-dirs --locked > .cargo/config.toml inside the third-party directory followed by reindeer buckify in the same directory, and in the generated BUCK file, you'll see instead of using http_archive to download the libraries from https://crates.io, now they'll be built using the vendored sources.

[svermeulen]

Thanks. Using http_archive has gotten me further towards migrating a bazel repo to buck2. I don't think reindeer helps in my case because I'm not vendoring rust/cargo dependencies specifically.

For non-rust/cargo dependencies, it looks like I can do something like use http_archive, and then run a python script (via a genrule for eg.) that modifies/builds the downloaded files to produce new files. But I do miss the bazel way of specifying a git patch file that has any number of changes you need to the repo, and also allowing a build file to be specified to use with that repo as well

[svermeulen]

For a more concrete example - I'm curious how a buck2 veteran would handle the following:

There is a C library at https://github.com/foo/bar that we want to vendor. When it is copy and pasted into our monorepo as a sub-directory, we can build it straightforwardly like this:

cxx_library(
    name = "foo",
    srcs = glob(["src/*.c"]),
    headers = glob(["**/*.h"]),
    visibility = ["PUBLIC"],
)

However, we'd like to avoid copy and past and instead use git_fetch or http_archive so we try the following instead:

git_fetch(
    name = 'foo-repo',
    repo = 'https://github.com/foo/bar',
    rev = '...',
)

// TODO - now how to define the library?

One idea I had was to use a genrule to select only all the *.c files, and then another one to only select all the *.h files, and then reference those genrules inside the 'srcs' and 'headers' of cxx_library, but I don't think this will work since cxx_library expects lists of files and when specified this way it gets a directory instead, and these genrules can only have one output as far as I know, so that approach won't work

For now the approach I am planning on taking is to use git sub modules or similar, then use a custom branch of the third party code that has my custom BUCK file, but I'd love to hear if there's better practice to follow around these kinds of problems

[thoughtpolice]

In the case of http_archive, at least, you can "export" specific files from the tarball as subtargets, and then explicitly refer to those subtargets in srcs fields of the relevant rules. This means that glob() doesn't work, but it's something (and sometimes you don't want glob() anyway, e.g. for platform specific files, so glob() isn't always guaranteed to work in any case). There might be an equivalent for git_fetch, I would hope...

For example, here is a completely standalone BUCK file that can build mimalloc directly from a downloaded tarball, with no sources copied into the repo. Just pure content-addressable storage. Maybe there's a better way to do this, but this does work for me on x86_64 Linux:

_VERSION = '2.1.2'

http_archive(
    name = 'src',
    sha256 = "2b1bff6f717f9725c70bf8d79e4786da13de8a270059e4ba0bdd262ae7be46eb",
    urls = [
        'https://github.com/microsoft/mimalloc/archive/refs/tags/v{}.tar.gz'.format(_VERSION)
    ],
    type = 'tar.gz',
    strip_prefix = 'mimalloc-{}'.format(_VERSION),
    visibility = [],
    sub_targets = [
        'include/mimalloc-override.h',
        'include/mimalloc.h',
        'include/mimalloc/internal.h',
        'include/mimalloc/prim.h',
        'include/mimalloc/types.h',
        'include/mimalloc/track.h',
        'include/mimalloc/atomic.h',
        'include/mimalloc-new-delete.h',

        'src/page.c',
        'src/segment.c',
        'src/os.c',
        'src/bitmap.h',
        'src/prim/prim.c',
        'src/prim/unix/prim.c',
        'src/prim/wasi/prim.c',
        'src/prim/osx/alloc-override-zone.c',
        'src/prim/osx/prim.c',
        'src/prim/windows/prim.c',
        'src/random.c',
        'src/stats.c',
        'src/options.c',
        'src/segment-map.c',
        'src/alloc.c',
        'src/bitmap.c',
        'src/heap.c',
        'src/alloc-aligned.c',
        'src/alloc-override.c',
        'src/static.c',
        'src/arena.c',
        'src/page-queue.c',
        'src/init.c',
        'src/alloc-posix.c',
    ]
)

cxx_library(
    name = "lib",
    visibility = [ 'PUBLIC' ],
    header_namespace = "",
    exported_headers = {
        'mimalloc.h':            ':src[include/mimalloc.h]',
        'mimalloc-override.h':   ':src[include/mimalloc-override.h]',
        'mimalloc/internal.h':   ':src[include/mimalloc/internal.h]',
        'mimalloc/prim.h':       ':src[include/mimalloc/prim.h]',
        'mimalloc/types.h':      ':src[include/mimalloc/types.h]',
        'mimalloc/track.h':      ':src[include/mimalloc/track.h]',
        'mimalloc/atomic.h':     ':src[include/mimalloc/atomic.h]',
        'mimalloc-new-delete.h': ':src[include/mimalloc-new-delete.h]',
    },
    srcs = [
        ':src[src/alloc.c]',
        ':src[src/alloc-aligned.c]',
        ':src[src/alloc-posix.c]',
        ':src[src/arena.c]',
        ':src[src/bitmap.c]',
        ':src[src/heap.c]',
        ':src[src/init.c]',
        ':src[src/options.c]',
        ':src[src/os.c]',
        ':src[src/page.c]',
        ':src[src/random.c]',
        ':src[src/segment.c]',
        ':src[src/segment-map.c]',
        ':src[src/stats.c]',
    ] + select({
        "config//os:linux": [':src[src/prim/unix/prim.c]'],
        "config//os:windows": [':src[src/prim/windows/prim.c]'],
        "config//os:macos": [
            ':src[src/prim/osx/alloc-override-zone.c]',
            ':src[src/prim/osx/prim.c]'
        ],
    }),
)

[c00t]

@svermeulen subdir_glob does that help?