What kinds of workflows do users have?

[acmcarther]

Summary of the discussion as of DEC 23 AM

We introduce two terms for mental models of how users might use cargo-raze. They are:

• "tomls-everywhere" model: Users define much or all of their crates through Cargo.tomls, including internal ones.
• "no-internal-tomls" model: Users define their internal crates (libraries, binaries, etc) using regular rules_rust and BUILD files, and use one or a few Cargo.tomls to define the full set of third_party dependencies, or some new invention like a crate_universe rule.

(this isn't necessarily meant to say one could not apply both approached, but that they're distinct ways of thinking about the cargo-bazel integration)

@UebelAndre and I agree that both mental models (which map down to code organization strategies) have valid usecases and should be supported. I add here now that we should view all new features as foundational (meaning internal or applies to both), or as applying to support one or the other usecase. I further suggest that we include a section very high in the documentation describing the two models, and helping users decide which works best for their Bazel workspace(s).

In general, the rest of the discussion consists of how we developed the previous vocabulary, and details in particular about the future direction of the "tomls-everywhere" model.

Original Post Below

---

Branched from discussion in #319

The workflow type that I'm familiar with (cribbed from how Google manages third party dependencies) is that there is a single repository of third party crates in the monorepo. In general, there is a rule there that there be only one version of each dependency. For each language, a set of special arbiters of third party dependencies exists. These folks gatekeep the inclusion of new third party code.

In that model, the organization in general would only have one cargo.toml, because the organization would not integrate deeply with cargo. That singular cargo.toml would not map to any single application. Instead application developers write applications and libraries more like one would write C++ applications and libraries, using the blessed versions of dependencies in //third_party or whatever, and never interact with cargo.toml.

The repo structure ends up looking like this

/third_party/crates/Cargo.toml
/third_party/crates/some_crate/
/some_sub_organization/some_project/some_component/code.rs 

---

In #319, @UebelAndre alludes to I think a different workflow which is more deeply integrated into cargo, and in general treats the cargo.toml, not the BUILD file, as the source of truth. I think this is an interesting model that should not necessarily be dismissed, in large part because it's significantly closer to the existing cargo user experience.

I think that workspace looks more like this

/some_sub_organization/some_project/Cargo.toml
/some_sub_organization/some_project/sub_crate/code.rs
/some_sub_organization/some_project/other_sub_crate/code.rs
/some_sub_organization/some_project/dependencies/some_crate
/some_sub_organization/some_other_project/Cargo.toml
/some_sub_organization/some_other_project/sub_crate/code.rs
/some_sub_organization/some_other_project/other_sub_crate/code.rs
/some_sub_organization/some_other_project/dependencies/some_crate

[acmcarther]

In general, I don't think I want to hash out here which model is objectively better. The google approach is detailed here. I think we can probably agree that (1) that approach might be better for large organizations, and (2) that approach might be onerous for small organizations. Furthermore, I think we can agree that the google approach forces code organization decisions that break assumptions made by most existing rust tools, which is a huge bummer.

---

In #319 (comment), @UebelAndre says

We have a few mono-ish repos that would immediately make use of cargo workspaces. We want to be able to take advantage of the lockfiles and avoid having a single volatile/hand authored place for all dependencies. And we don't want to have to be spending all our time writing toml and BUILD files. In an ideal world, we would near seamlessly replace the use of Cargo with Bazel and be able to do all the same things without a bunch of additional hassle.

It sounds like you envision developers on individual projects writing cargo.toml files for each crate, but instead of building with cargo, you'd build with bazel. Is that accurate? Under what circumstances would developers on the same or different project or same or different organization try to ensure that the same versions of dependencies are used?

[UebelAndre]

Not every organization is equipped to assert that organization wide, specific versions of dependencies will be used. I don't think it'd be appropriate for this tool to try and enforce that. Both cases should be supported.

[acmcarther]

Hence, why this topic exists.

[UebelAndre]

So, the only two workflows I personally keep in mind is:

1. What is the existing workflow for using standard tools of a particular language
2. What is the most Bazel-ish thing to do to keep everything fast and correct.

I think it's valuable to support the use case that someone has a cargo project and wants to leave it as is but add Bazel files to start building with Bazel.

Additionally, there are some freedoms you can take if you're only going to be using Bazel and have never had cargo in your project. I think this project has exclusively lived in this camp for the majority of it's life but being new to Rust but familiar with Bazel, I found it more natural and easier to be able to keep both Cargo and Bazel working side by side so I can prototype something new and weird in the environment where it was designed and then begin to port it over to Bazel. Without that, many interactions around some build.rs file or use of other crates that rely on Cargo metadata (cbindgen) might have driven me to simply abandon Bazel or come up with my own rules backed by cargo (similar to what happened in #193).

[acmcarther]

Couple of questions

• How should that user start using Bazel for their existing rust crate? Do they clone it into an existing monorepo? Where is it situated if there are preexisting crates? How does a third library depend upon both crates?
• Whats the expected level of backward and forward compatibility? A crate using rust_proto_library rules and then depending upon them in a rust library would not be backward compatible with cargo

[UebelAndre]

How should that user start using Bazel for their existing rust crate?

In an ideal world, cargo-raze could setup their Cargo project to start using Bazel similar to wasm-pack or cargo init.

Do they clone it into an existing monorepo? Where is it situated if there are preexisting crates?

The crates can exist in a mono repo or a not. The repo or location is not important, so long as it has a valid cargo workspace (purely speaking in terms structure), raze can generate the necessary files to support it in Bazel. A third

How does a third library depend upon both crates?

A third library could be added either in Bazel or Cargo.

Whats the expected level of backward and forward compatibility? A crate using rust_proto_library rules and then depending upon them in a rust library would not be backward compatible with cargo

If users decide to use other Bazel targets in as dependencies for their crates, then it should not be backwards compatible. Cargo would only be able to represent a dependency tree moving forward.

[UebelAndre]

I'll add more context here but:

In #319, @UebelAndre alludes to I think a different workflow which is more deeply integrated into cargo, and in general treats the cargo.toml, not the BUILD file as the source of truth

This is currently how workflows involving cargo-raze work the source of truth is not actually what's written in the BUILD files, it's whatever cargo-raze generated and aliased in the /third_party/crates package. In an ideal future, there would be a way to author rust_* target in a BUILD file and have Bazel correctly fetch the intended dependencies, but no such tooling exists today. The workflow I'm interested in is taking existing Cargo projects and converting them to use Bazel. The only future dependency on cargo is to define dependencies in a way that cargo-raze could generate a way for Bazel to fetch them.

[UebelAndre]

Are you asserting that my_crate which depends on serde consults serde's lockfile during dependency resolution?

No because serde is not a cargo workspace memeber. Only workspace members affect the lockfile, right? I'm very confused now. If you look at https://github.com/google/cargo-raze/tree/0026a7bde7487a61aac571ea992fee5cd8a442bd/examples/remote/cargo_workspace

Are you saying that the lockfile there is not what will be used for complication?

[acmcarther]

I am asserting one thing. Dependency's lockfiles are not consulted during resolution. If you are cargo-razing (or crate_universing) with against a root crate (or a root workspace), it may possess and use one lock file corresponding to the root toml. Any transitive crates or workspaces you depend on may have a lock file, but these are not used as part of resolution by cargo or cargo-raze, or crate_universe.

A hypothetical crate_universe which solves this problem must not do the dependency resolution -- it must defer it, because it might be used with other crate_universes. Similarly, in that situation, it could not use a lockfile, because the only lockfile relevant during resolution is the one corresponding to the root toml.

[UebelAndre]

Ok, that makes sense. So then the distinction I would make between cargo workspace and a model of one-list-of-dependencies-to-rule-them-all is that a cargo workspace may contain multiple versions of the same dependency. But they will all be resolved for the workspace members that depends on them.

In the no-internal-toml-model I would be interested in a way to have a way to model a cargo workspace. In your 3 crates example, each of those crates would be a part of the same universe but that universe can contain multiple versions of the same transitive dependency.

Another good example is again to look at the current state of the examples. I would consider each directory in examples/[remote|vendored] to be it's own crate_universe as they're already their own cargo workspaces. This makes the examples/[remote|vendored]/cargo_workspace example unique in that it contains multiple packages. It's still part of the same cargo workspace, and therefore should be representable in the same cargo_universe.

edit: Basically, the lockfiles that are there should be the same lockfiles used in cargo_universe, including the examples/[remote|vendored]/cargo_workspace lockfiles.

[acmcarther]

We're meandering into some additional territory here which is not directly related to the root issue (which is that crates from a given cargo_universe crate may not interact because they each represent an independent execution of crate resolution which might result in transitive deps which are impossible to compile together).

---

Ok, that makes sense. So then the distinction I would make between cargo workspace and a model of one-list-of-dependencies-to-rule-them-all is that a cargo workspace may contain multiple versions of the same dependency. But they will all be resolved for the workspace members that depends on them.

I interpret this to mean that the individual crates in the workspace can depend upon different versions of the same dependency, which is a capability unique to the workspace feature. As a rule, the aliases (the means by which third party crates are exposed for the no-internal-tomls folks) do not include multiple versions of the same crate.

In the no-internal-toml-model I would be interested in a way to have a way to model a cargo workspace. In your 3 crates example, each of those crates would be a part of the same universe but that universe can contain multiple versions of the same transitive dependency.

That depends on what you mean by "model a cargo workspace". If all things are allowed, meaning you're not constrained to having a "workspace toml", then you would just have three directories with three BUILD files. If you have some constraints (e.g. you want to retain a workspace toml), then we have to think about a hybrid approach, such as basically what we have now. This needs more thought, but my conclusion so far, at the risk of being tedious, is that you cannot compose two tomls-everywhere-model workspaces just the same as you can't compose two tomls-everywhere crates.

EDIT: this makes zero progress addressing the request of having the different workspace crates use different versions of crates. 🤷

Another good example is again to look at the current state of the examples. I would consider each directory in examples/[remote|vendored] to be it's own crate_universe as they're already their own cargo workspaces. This makes the examples/[remote|vendored]/cargo_workspace example unique in that it contains multiple packages. It's still part of the same cargo workspace, and therefore should be representable in the same cargo_universe.

That's true. The implication here is that none of these directories could ever interoperate. The workspace example has a limited type of interoperation internally (defined by the relationship between the crates in the workspace). Were on the same page that each directory represents one cargo_universe, and that the workspace version gets the privilege of having several crates operate in the context of that universe.

I didn't communicate this elsewhere, but the reason these examples are in independent dirs (and crate_universes) is just so that they can be isolated from one another. Some of the examples involve bringing in weird kinds of dependencies or specifying features, or whatever, and it's nice if they break independently and not all at once.

---

I recognize there's an unanswered question here, which is "ok, so what do we do about the lack of cargo_universe composability". I don't know the answer to that. I think the answer is probably either (1) treat your code as separate from the rest of the universe, meaning no-internal-tomls, because native bazel rules are always composable at least within the bazel workspace, or (2) treat all code, including your own code, as third party code. I think this a strange reframing or evolution of the "tomls-everywhere-model".

Incidentally, I also recognize that (1) also means "make your code not composable with any existing code in the ecosystem" which seems like throwing the baby out with the bathwater.

[UebelAndre]

I think this conversation would best be had over at bazelbuild/rules_rust#2 A lot of good questions and comments for that here. Over the holiday I'm hoping to take some time to do a deep dive into cargo_universe. Until then though, There are still a few PRs I'd like to get through which will unblock some efforts on cargo_universe.

[acmcarther]

When I filed this issue I had the impression that there existed an unverbalized mental model of rust-bazel-cargo-raze where developers are writing new code using new Cargo.tomls, even if such crate is never destined to be pushed to crates.io (or even built with Cargo). In this model, nobody would ever hand write a rust_library because every crate would consist of a toml and either be free floating (and unmoored from any common dependencies), or live in a single large cargo workspace and be "moored" by virtue of a shared lock file.

Is this indeed the goal we're after? If so, I think we should (informally) come up with a "user journey" for this use case, and a set of goals and non-goals. We can then tie some of these new features being implemented to that larger vision.

If "declare-in-cargo-build-in-bazel" is not the desired end-state, then what specifically is the goal? Is it just to make building a single arbitrary existing crate easier with cargo-raze? What role do rust_library and the other regular rules_rust rules play in this model?

[UebelAndre]

I'd love to see a "user journey" list! Happy to help build that out.

I think the ultimate goal of cargo-raze is to be the gateway to a world where users can author their dependencies in BUILD files and have some Bazel rule version check and fetch all those dependencies. I only want to define my dependencies once and not end up spending my time writing different variants of the same thing in different places.

However, going back to declare-in-cargo-build-in-bazel, I think there's actually a lot of value here. I wrote #305 (comment) on just that. Right now cargo-raze has some "special" behavior for fetching binary dependencies, a concept that isn't really supported by Cargo just yet. This is necessary for things like rust_wasm_bindgen but only due to an implementation detail. (I also think this relates to #327 (reply in thread)) If you recall #218 and it's relation to wasm-bindgen (the issue being cargo-raze is no longer able to generate the wasm-bindgen-cli binary), binary crates do not appear in Cargo metadata but it's still incredibly useful to be able to rely on those binaries and build other rules with them. I think it'd be appropriate for cargo-raze to support a declare-in-cargo-build-in-bazel model to solve that issue.

To me, cargo_universe is the the first big step towards making Bazel more first class as it could enable the source of truth to be build files. To do this, it's currently using cargo-raze as a library since there's a lot of work done here to build a dependency graph for Bazel. But I think the role of cargo-raze as a binary should focus on declare-in-cargo-build-in-bazel and ultimately be the glue for things like rust_wasm_bindgen. I'm very interested in writing a repository rule similar to new_git_repository that can take an arbitrary crate from an arbitrary repository (not vendored in a monorepo) and generate build targets for it.

[acmcarther]

I think the ultimate goal of cargo-raze is to be the gateway to a world where users can author their dependencies in BUILD files and have some Bazel rule version check and fetch all those dependencies. I only want to define my dependencies once and not end up spending my time writing different variants of the same thing in different places.

Do you want your code for your application to use hand written BUILD files or hand written Cargo.tomls?

I think it'd be appropriate for cargo-raze to support a declare-in-cargo-build-in-bazel model to solve that issue.

I've historically been against the "declare-in-cargo-build-in-bazel" model, hence my push back against features that make BUILD files less clear (as they'd drive us toward the opaque BUILD file model).

(I'm open to it now, for some definition of open, but i don't fully understand the ideas that are taking shape)

To me, cargo_universe is the the first big step towards making Bazel more first class as it could enable the source of truth to be build files

I agree with this, though not for the same reason. cargo_universe appears to allow someone to completely ditch the toml files and just declare a cargo_universe in //third_party and write regular BUILD files that depend on those third party deps. Sounds great to me.

[UebelAndre]

Do you want your code for your application to use hand written BUILD files or hand written Cargo.tomls?

There are use cases for both. I've wanted a rule that can setup any crate as a remote repository with a generated build file. This is effectively what's being done for binary_deps currently, more or less. But for my main/larger projects, I'll want to hand author my build files.

I agree with this, though not for the same reason. cargo_universe appears to allow someone to completely ditch the toml files and just declare a cargo_universe in //third_party and write regular BUILD files that depend on those third party deps. Sounds great to me.

I don't understand why you feel cargo_universe is any less of an abstraction. I would want to use some kind of all_crates macro for that as well so I don't redefine the same thing multiple times. I see this as being quite similar to python's requirement files in that you can pip_install a requirements file and have it be represented by a repository name.

pip_install(   # or pip3_import
   name = "my_deps",
   requirements = "//path/to:requirements.txt",
)

I use all_requirements because I already hand authored the requirements file to contain only the dependencies for the part of the project I want to use it for. I would fully expect cargo_universe to do something similar.

[acmcarther]

I don't understand why you feel cargo_universe is any less of an abstraction. I would want to use some kind of all_crates macro for that as well so I don't redefine the same thing multiple times

I'm not sure if this predates our shared understanding, but from recent comments, I think it might be prudent to clear this up.

People in the "no-internal-tomls" world view generally will have a monorepo consisting of a non-trivial number of rust_library and rust_binary floating around (among other non-rust build targets). In general, no individual BUILD target will likely depend on every single dependency, so it would be a rare circumstance where an "all_crates()" is useful.

(i recognize the utility in an environment where crate_universe maps one to one to internal cargo-tomls, etc, etc)

[UebelAndre]

Yeah, this predates a shared understanding. I did not have a world in mind where users had a colossal crate_universe where a small subset of dependencies were only relevant to a particular target.

[UebelAndre]

I think this might be relevant here, but I think this project is quite a unique mix of different components. For the life of this project, I think it's valuable to continue to publish cargo raze to crates.io both for use as a library and as a binary. But I also feel like there's huge potential in being able to build it with Bazel and begin to use it in Bazel rules.

[UebelAndre]

One other thing of note, I don't think any of the examples are in alignment with the Google model of cargo-raze.

[UebelAndre]

For months I've been making changes that grow support for the tomls-everywhere-model because to me it was the biggest benefit of this tool. If the no-internal-tomls-model is the end goal (which I'd like to work towards), then I feel cargo_universe should be rolled out and a larger design doc should be written up for exactly how that's going to work. But there's still absolutely a desire for tomls-everywhere-model which can easily live side by side with the no-internal-tomls-model. The examples are not the "Google"/no-internal-tomls-model but they are definitely valid use cases.

[acmcarther]

I think we're on the same page here now that we have the right vocabulary. There are valid usecases for both the "tomls-everywhere" and the "no-internal-tomls" approach and I think we should support both use cases. I just historically have had my brain in the "no-internal-tomls" mode, and haven't understood the struggles of someone working toward a "tomls-everywhere" workflow.

[UebelAndre]

So what happens now? 😅

[acmcarther]

Generally I intend to green light the relevant changes, but I'm going to continue to ask annoying questions like "how will these things interoperate", and "how exactly is this useful" despite the utility of the feature being incredibly obvious to you.

[UebelAndre]

I'm fine with that. I'm tying harder and harder to keep my PRs small and focused to make reviewing it and explaining it as easy as possible. I think the relevant PR here is an interesting one because it's was the thing that brought to light that we have not been on the same page for months.