Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement Bzlmod in Bazel #13316

Closed
meteorcloudy opened this issue Apr 8, 2021 · 5 comments
Closed

Implement Bzlmod in Bazel #13316

meteorcloudy opened this issue Apr 8, 2021 · 5 comments
Assignees
@Wyverald @meteorcloudy
Labels
area-Bzlmod Bzlmod-specific PRs, issues, and feature requests P1 I'll work on this now. (Assignee required) team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. type: feature request

Comments

@meteorcloudy
Copy link
Member

Implement the new dependency management system proposed in https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit?usp=sharing
@meteorcloudy meteorcloudy added team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. P2 We'll consider working on this in future. (Assignee optional) team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: feature request labels Apr 8, 2021
@meteorcloudy
Copy link
Member Author

We'll first build a MVP in the bzlmod branch.

bazel-io pushed a commit that referenced this issue Jun 18, 2021
@Wyverald @copybara-github
bzlmod: ModuleFileFunction
efa35e3 
(#13316)

This CL implements ModuleFileFunction, a SkyFunction that takes the name and version of a module, finds and evaluates its MODULE.bazel file and returns the evaluated result.

This is not hooked into the main SkyFrame route yet (RepositoryDelegatorFunction has yet to call this).

PiperOrigin-RevId: 380141062
bazel-io pushed a commit that referenced this issue Jun 18, 2021
@Wyverald @copybara-github
bzlmod: Remove override_dep in favor of having module_name on each ov…
d09077e 
…erride type

(#13316)

We're "denormalizing" the override API: whereas you'd write "override_dep(name='X', Y_override(...))" before, you'd now write "Y_override(module_name='X', ...)".

PiperOrigin-RevId: 380173439
bazel-io pushed a commit that referenced this issue Jun 21, 2021
@Wyverald @copybara-github
bzlmod: DiscoveryFunction
44fe650 
(#13316)

This SkyFunction performs module discovery by starting out from the root module and iteratively adding direct dependencies into the graph.

PiperOrigin-RevId: 380563418
bazel-io pushed a commit that referenced this issue Jun 22, 2021
@meteorcloudy @copybara-github
Bzlmod: Add RepoSpec.java
706e54c 
RepoSpec holds information of a repository rule definition, including where the rule class is defined and the attributes of a repository rule. It's the output of Bzlmod resolution and will be used to create a repository rule instance.

Related: #13316

RELNOTES: None
PiperOrigin-RevId: 380784644
bazel-io pushed a commit that referenced this issue Jun 24, 2021
@meteorcloudy @copybara-github
bzlmod: put RepoSpec.java in the common target
4c630f5 
Related: #13316

RELNOTES: None
PiperOrigin-RevId: 381261666
bazel-io pushed a commit that referenced this issue Jun 28, 2021
@Wyverald @copybara-github
bzlmod: SelectionFunction
a3fe02a 
(#13316)

This SkyFunction takes the result of DiscoveryFunction and runs MVS on it, resulting in a dep graph that contains only 1 version of each dependency (the highest requested one).

PiperOrigin-RevId: 381844359
bazel-io pushed a commit that referenced this issue Jun 30, 2021
@Wyverald @copybara-github
bzlmod: Add an HttpDownloader#downloadAndReadOneUrl method
d1120b6 
(#13316)

To implement the index registry outlined in the design doc, we need to make HTTP GET requests to certain URLs and read their contents immediately (instead of downloading and writing them to local disk).

Also added a way to distinguish a 404 from other unrecoverable errors, since 404 is semantically different from, say a 401, in terms of the registry list (earlier registries take precedence and delegate to later registries only when the requested module is not found).

PiperOrigin-RevId: 382304312
bazel-io pushed a commit that referenced this issue Jul 2, 2021
@Wyverald @copybara-github
bzlmod: use env.getValues() instead of getValue() in DiscoveryFunction
1464393 
(#13316)

Before we were requesting the ModuleFileValues one by one. This does not result in things being slow per se (since the individual ModuleFileFunction calls should be parallelized by Skyframe) but technically using getValues() can make SkyValue dependencies cleaner (since there is no dependency between the same "level" of SkyKeys being requested here).

PiperOrigin-RevId: 382700697
bazel-io pushed a commit that referenced this issue Jul 2, 2021
@Wyverald @copybara-github
bzlmod: IndexRegistry
a93baef 
(#13316)

This CL implements the basic "Index" registry handling (which is what the Bazel Central Registry will be using). A registry object does two things right now: one is fetching the module file (used during ModuleFileFunction), and the other is telling later stages how to instantiate a repo rule to fetch the contents of the module (used during BzlmodRepoRuleFunction).

PiperOrigin-RevId: 382739746
bazel-io pushed a commit that referenced this issue Jul 2, 2021
@meteorcloudy @copybara-github
bzlmod: add BzlmodRepoRuleFunction and friends
348b532 
- BzlmodRepoRuleHelper: calculates the RepoSpec for a given repository. The actual calculation will be implemented in following changes.

- BzlmodRepoRuleFunction: Take a given repository name, look up its RepoSpec via BzlmodRepoRuleHelper, and then create a repository rule out of it. The result will be used in RepositoryDelegatorFunction.

Related: #13316

RELNOTES: None.
PiperOrigin-RevId: 382745079
bazel-io pushed a commit that referenced this issue Jul 7, 2021
@meteorcloudy @copybara-github
bzlmod: Support creating Starlark repo rule
87325e5 
Related: #13316

RELNOTES: None.
PiperOrigin-RevId: 383380283
bazel-io pushed a commit that referenced this issue Jul 7, 2021
@Wyverald @copybara-github
bzlmod: Add --registry flag
05265a4 
(#13316)

This flag is repeatable and used by ModuleFileFunction to determine where to look for modules.

This is not yet testable with an integration test since this code path isn't hooked into production yet (we'll probably need a flag for that).

PiperOrigin-RevId: 383385942
bazel-io pushed a commit that referenced this issue Jul 7, 2021
@meteorcloudy @copybara-github
bzlmod: Add new features to http_archive rule
e9a4e93 
Those features are needed by the Bzlmod system to fetch Bazel module dependencies from Bazel registries.

- integrity: In addition to "sha256", users can specify the integrity value to verify the downloaded archive content.
- remote_patches, remote_patch_strip: apply patche files downloaded from given URLs. This will be used to apply patch files hosted in Bazel registries.

Related: #13316

RELNOTES: None
PiperOrigin-RevId: 383386903
bazel-io pushed a commit that referenced this issue Jul 12, 2021
@meteorcloudy @copybara-github
bzlmod: Implement BzlmodRepoRuleHelper
5acaf10 
Now we can get RepoSpec for non-registry overrides and Bazel module derived repositories,
module rule support will be added later.

Related: #13316

RELNOTES: None
PiperOrigin-RevId: 384178751
bazel-io pushed a commit that referenced this issue Jul 12, 2021
@Wyverald @copybara-github
bzlmod: compatibility_level support
f1f5b5d 
(#13316)

The compatibility_level is essentially the "major version" in semver, except that it's not embedded in the version string, but exists as a separate field. Modules with different compatibility levels participate in version resolution as if they're modules with different names, but the final dependency graph cannot contain multiple modules with the same name but different compatibility levels.

PiperOrigin-RevId: 384185854
bazel-io pushed a commit that referenced this issue Jul 12, 2021
@Wyverald @copybara-github
bzlmod: Remove overrides from DiscoveryValue and SelectionValue
6501f30 
(#13316)

The overrides cannot be nicely encapsulated in SelectionValue because we need the overrides when fetching modules with non-registry overrides, which happens during discovery. Any consumers of overrides should get them from the ModuleFileValue of the root module directly instead.

PiperOrigin-RevId: 384236547
bazel-io pushed a commit that referenced this issue Jul 12, 2021
@Wyverald @copybara-github
bzlmod: Use a Version object (née ParsedVersion) everywhere instead o…
32e91a7 
…f a string

(#13316)

See motivating TODO in ModuleFileGlobals.java. This avoids double-parsing (once at input verification time and once at selection time).

PiperOrigin-RevId: 384243747
bazel-io pushed a commit that referenced this issue Jul 13, 2021
@meteorcloudy @copybara-github
bzlmod: Load repositories from Bzlmod
7bc0199 
With --experimental_enable_bzlmod=true, Bazel can now load external repositories
from the new dependency management system -- Bzlmod.

Related: #13316

RELNOTES: None
PiperOrigin-RevId: 384424361
bazel-io pushed a commit that referenced this issue Jul 15, 2021
@Wyverald @copybara-github
bzlmod: Handle non-registry overrides in ModuleFileFunction
d382e91 
(#13316)

Now that we can properly fetch repos with non-registry overrides in RepositoryDelegatorFunction, we can also handle them correctly during discovery.

PiperOrigin-RevId: 384889161
bazel-io pushed a commit that referenced this issue Jul 15, 2021
@Wyverald @copybara-github
bzlmod: multiple_version_override
af17598 
(#13316)

Adds support for the "diamond" case of multiple_version_override: that is, multiple different modules can depend on different versions of the same module.

The "fork" case is still unchecked (so if you use `bazel_dep(B@1.0, repo_name='B1'); bazel_dep(B@2.0, repo_name='B2')` it won't throw an error). The check will be added in a next CL as I figured this one is already huge enough :)

PiperOrigin-RevId: 384894458
bazel-io pushed a commit that referenced this issue Jul 15, 2021
@Wyverald @copybara-github
bzlmod: Small cleanup for Version
44bd921 
(#13316)

Turns out nobody needs to deal with a null version anyway, so I removed the null handling. And the unused Version#mustParse method.

PiperOrigin-RevId: 384897795
bazel-io pushed a commit that referenced this issue Jan 20, 2022
@Wyverald @copybara-github
Bzlmod: Better canonical repo names for modules with overrides
b0d9d44 
(#13316)

Use "foo.override" instead of just "foo" as the canonical repo name for the module "foo" when it has a non-registry override. Context: https://groups.google.com/a/bazel.build/g/external-deps/c/GcV1cpRd-Ls/m/o446O3vGAAAJ

PiperOrigin-RevId: 423049588
bazel-io pushed a commit that referenced this issue Jan 27, 2022
@Wyverald @copybara-github
Bzlmod: Refactor repo rule creation code
90abed5 
(#13316)

By recognizing that the only difference between creating a native and starlark repo rule is the code to acquire the RuleClass object, we can remove a lot of duplicate code around repo rule creation.

* BzlmodRepoRuleCreator the interface is simply replaced with the existing interface RuleFunction, which exposes the underlying RuleClass.
* Moved repo rule creation code into BzlmodRepoRuleCreator, which is now a utility class. This logic can then be shared by ModuleExtensionEvalStarlarkThreadContext and BzlmodRepoRuleFunction, instead of existing in 3 different places.

This refactor helps us eventually support rules like `native.local_repository` in module extensions.

PiperOrigin-RevId: 424627127
@Wyverald Wyverald added area-Bzlmod Bzlmod-specific PRs, issues, and feature requests P1 I'll work on this now. (Assignee required) and removed P2 We'll consider working on this in future. (Assignee optional) labels Feb 22, 2022
@Wyverald
Copy link
Member

Closing this umbrella bug in favor of the individual issues with the "area-Bzlmod" label.

@cpsauer
Copy link
Contributor

cpsauer commented May 26, 2022

@Wyverald, for toolmakers/library maintainers that should be sure to adopt before bzlmod is enabled by default:
Is there a new issue we should follow to stay abreast of when bzlmod will fully launch?

@Wyverald
Copy link
Member

Bzlmod will "fully" launch with the release of Bazel 6.0, at which point the flag --experimental_enable_bzlmod will be renamed to just --enable_bzlmod.

Bzlmod will be enabled by default in Bazel 7.0 at the earliest. That would mean deprecating WORKSPACE support, which we're still a rather long way from.

Both of these will be featured prominently in the release notes of the major releases, so hopefully it's unlikely for you to miss them :)

@cpsauer
Copy link
Contributor

cpsauer commented May 26, 2022

Definitely. Super good to know. Thank you! Great to know the forward roadmap--and thanks for all your great efforts.

@cpsauer cpsauer mentioned this issue May 26, 2022
Closed
fmeum pushed a commit to fmeum/bazel that referenced this issue Jul 2, 2022
@Wyverald @fmeum
Bzlmod: Better canonical repo names for modules with overrides
768b686 
(bazelbuild#13316)

Use "foo.override" instead of just "foo" as the canonical repo name for the module "foo" when it has a non-registry override. Context: https://groups.google.com/a/bazel.build/g/external-deps/c/GcV1cpRd-Ls/m/o446O3vGAAAJ

PiperOrigin-RevId: 423049588
fmeum pushed a commit to fmeum/bazel that referenced this issue Jul 2, 2022
@Wyverald @fmeum
Bzlmod: Better canonical repo names for modules with overrides
7f13e4a 
(bazelbuild#13316)

Use "foo.override" instead of just "foo" as the canonical repo name for the module "foo" when it has a non-registry override. Context: https://groups.google.com/a/bazel.build/g/external-deps/c/GcV1cpRd-Ls/m/o446O3vGAAAJ

PiperOrigin-RevId: 423049588
@fmeum fmeum mentioned this issue Jul 2, 2022
Merged
fmeum pushed a commit to fmeum/bazel that referenced this issue Jul 2, 2022
@Wyverald @fmeum
Bzlmod: Better canonical repo names for modules with overrides
90c86e4 
(bazelbuild#13316)

Use "foo.override" instead of just "foo" as the canonical repo name for the module "foo" when it has a non-registry override. Context: https://groups.google.com/a/bazel.build/g/external-deps/c/GcV1cpRd-Ls/m/o446O3vGAAAJ

PiperOrigin-RevId: 423049588
ckolli5 pushed a commit that referenced this issue Jul 5, 2022
@fmeum @Wyverald
Bzlmod: Better canonical repo names for modules with overrides (#15793)
62be9ea 
(#13316)

Use "foo.override" instead of just "foo" as the canonical repo name for the module "foo" when it has a non-registry override. Context: https://groups.google.com/a/bazel.build/g/external-deps/c/GcV1cpRd-Ls/m/o446O3vGAAAJ

PiperOrigin-RevId: 423049588

Co-authored-by: wyv <wyv@google.com>
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
bzlmod: Store a Package instead of a Rule in BzlmodRepoRuleValue
4baebd5 
    (bazelbuild/bazel#13316)

    This is to work around a restriction in Google. The external packages we create in BzlmodRepoRuleFunction host just one rule each; so instead of storing a rule, we can just store the package and get the rule from there.

    This does require adding the rule to the package, which we weren't doing before. This involved a bit of a detour to clean up the RuleFactory#createAndAddRule method, which for some reason has had a much more sensible overload that's package private since time immemorial. This CL simply removes the overload that just gets two fields out of PackageContext.

    PiperOrigin-RevId: 387543141
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
bzlmod: Guarantee BFS iteration order for SelectionValue#getDepGraph
bdc31aa 
    (bazelbuild/bazel#13316)

    This makes it easier for later usage (for example, when collecting toolchains across the dep graph, BFS order makes the most sense).

    Also added some missing test assertions for the reverse lookups (which actually fixed a small bug).

    PiperOrigin-RevId: 387076182
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
bzlmod: Change FakeRegistry to return real, local_repository RepoSpecs
2b7d95f 
    (bazelbuild/bazel#13316)

    This allows us to more easily set up some tests that actually want to fetch modules, without relying on local_path_override all the time. (Useful for RegisteredExecutionPlatformsFunctionTest, for example.)

    PiperOrigin-RevId: 387064920
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
bzlmod: Store canonical repo names in SelectionValue
5ac09b3 
    (bazelbuild/bazel#13316)

    Up until now we assume the canonical repo name of a module is simply the module name. This doesn't work with multiple-version overrides. This CL addresses this issue; the canonical repo name will be ${moduleName}.${moduleVersion} for any module.

    This does mean that we need to put in some extra work to build reverse lookup maps, and especially add extra logic in RepositoryMappingFunction to make sure that WORKSPACE references to repos generated by modules can still work (i.e. if B is a module, something that points to "@b" gets rewritten to "@B.1.0", the canonical repo name)

    PiperOrigin-RevId: 385785118
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
bzlmod: Allow absence of module() call in MODULE.bazel
46d5c08 
    (bazelbuild/bazel#13316)

    Not calling module() should be the same as calling module() with no arguments. This was mistakenly disallowed before since the Module autovalue builder didn't have defaults for name and version.

    PiperOrigin-RevId: 385119757
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
bzlmod: "fork" case for multiple_version_override
86de4b9 
    (bazelbuild/bazel#13316)

    It's okay for a module depends on foo@1.2 and foo@1.3 with different repo names as long as there's a multiple_version_override for foo that allows both 1.2 and 1.3. If only 1.3 is allowed or if there's no multiple_version_override at all, then selection will succeed right now, and the module will depend on foo@1.3 twice. This CL adds a check that this doesn't happen.

    PiperOrigin-RevId: 384906574
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
bzlmod: Small cleanup for Version
7292a76 
    (bazelbuild/bazel#13316)

    Turns out nobody needs to deal with a null version anyway, so I removed the null handling. And the unused Version#mustParse method.

    PiperOrigin-RevId: 384897795
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
bzlmod: Remove overrides from DiscoveryValue and SelectionValue
4db3bab 
    (bazelbuild/bazel#13316)

    The overrides cannot be nicely encapsulated in SelectionValue because we need the overrides when fetching modules with non-registry overrides, which happens during discovery. Any consumers of overrides should get them from the ModuleFileValue of the root module directly instead.

    PiperOrigin-RevId: 384236547
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
bzlmod: compatibility_level support
db7453a 
    (bazelbuild/bazel#13316)

    The compatibility_level is essentially the "major version" in semver, except that it's not embedded in the version string, but exists as a separate field. Modules with different compatibility levels participate in version resolution as if they're modules with different names, but the final dependency graph cannot contain multiple modules with the same name but different compatibility levels.

    PiperOrigin-RevId: 384185854
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
bzlmod: use env.getValues() instead of getValue() in DiscoveryFunction
a654891 
    (bazelbuild/bazel#13316)

    Before we were requesting the ModuleFileValues one by one. This does not result in things being slow per se (since the individual ModuleFileFunction calls should be parallelized by Skyframe) but technically using getValues() can make SkyValue dependencies cleaner (since there is no dependency between the same "level" of SkyKeys being requested here).

    PiperOrigin-RevId: 382700697
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
bzlmod: Add an HttpDownloader#downloadAndReadOneUrl method
0575eac 
    (bazelbuild/bazel#13316)

    To implement the index registry outlined in the design doc, we need to make HTTP GET requests to certain URLs and read their contents immediately (instead of downloading and writing them to local disk).

    Also added a way to distinguish a 404 from other unrecoverable errors, since 404 is semantically different from, say a 401, in terms of the registry list (earlier registries take precedence and delegate to later registries only when the requested module is not found).

    PiperOrigin-RevId: 382304312
@cpsauer cpsauer mentioned this issue Oct 29, 2022
Open
@cpsauer cpsauer mentioned this issue Aug 1, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Wyverald Wyverald

@meteorcloudy meteorcloudy

Labels
area-Bzlmod Bzlmod-specific PRs, issues, and feature requests P1 I'll work on this now. (Assignee required) team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. type: feature request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Wyverald @philwo @meteorcloudy @cpsauer and others