Add cargo-add (from cargo-edit) to cargo proper

[killercup]

Cargo as well as cargo-edit have gone a long way since #4 was opened. It might now make sense to move (parts of) cargo-edit into cargo itself. Basically, we now have a format-preserving TOML library that -- at least for adding a dependency line -- work quite well (you can test this with cargo install cargo-edit --vers 0.3.0-beta.1 -f).

Speaking with @matklad, I believe I now have a somewhat solid understanding of what needs to happen to add a new built-in cargo subcommand. One important realization was that cargo-install already contains some of the parts we need.

Here are some steps to get started:

1. Create a src/bin/cargo/commands/add.rs (as duplicate of install.rs at first)
2. In src/bin/cargo/commands/mod.rs, add add to builtin/builtin_exec, and to the list of mods
3. Create a src/bin/cargo/ops/cargo_add.rs (as duplicate of cargo_install.rs at first)

One thing we should aim for though, is to now just copy-paste the whole of cargo-edit but to re-use as much of cargo's infrastructure as possible. For example, we should try to use cargo's way of querying the registry as well as its CLI args handling and output formatting. We can, for example, also use args.workspace.current_manifest to get the manifest and deal with workspace.

[killercup]

@rust-lang/cargo, is this something you'd be interested in being worked on right now? This might work as a mentored even if you can't spend much time to work on it ourselves.

[alexcrichton]

I'd personally be thrilled to see this enter Cargo itself! I would ideally prefer to avoid having two TOML parsers built in but if we need to have that for the time being I think it's not the end of the world. (I don't know much about the technical design of cargo-edit right now)

[steveklabnik]

This is the number one thing I want in Cargo; a simple 👍 is not enough. Tons and tons and tons of people are very excited for this 🎊

[ibaryshnikov]

I'd like to implement it, work in-progress is here
https://github.com/ibaryshnikov/cargo/tree/issue-5586-cargo-add

[japaric]

I know that lots of people (myself included) are plenty busy with stuff for the upcoming edition release and that we don't have much spare human Rustacean resources but I still want to say that it would be AMAZING to have cargo add in Cargo 2018. As mentioned by boats, cargo add fits perfectly with the module / path changes of the edition. And I fully agree, but to me the biggest win would be not having to cargo install cargo-edit, which involves compiling / linking to C code and can be a (very) frustrating experience.

@rust-lang/cargo @killercup @aturon could we make this happen, somehow? 🙏

[steveklabnik]

Huuuuuuuuuuuuge 👍 here

…

On Sep 6, 2018, at 7:59 PM, Jorge Aparicio ***@***.***> wrote: I know that lots of people (myself included) are plenty busy with stuff for the upcoming edition release and that we don't have much spare human Rustacean resources but I still want to say that it would be AMAZING to have cargo add in Cargo 2018. As mentioned by boats, cargo add fits perfectly with the module / path changes of the edition. And I fully agree, but to me the biggest win would be not having to cargo install cargo-edit, which involves compiling / linking to C code and can be a (very) frustrating experience. @rust-lang/cargo @killercup @aturon could we make this happen, somehow? 🙏 — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[joshtriplett]

👍 here.

[ibaryshnikov]

I did some preparations in this pr like filling the boilerplate and updating errors from cargo-edit to fit more for cargo, but farther work requires some patches to toml parser which is used in cargo (in case we don't want to have two parsers), and currently I don't have enough time to do required updates in the parser on my own. If you find some of the code in this pr fits for the solution, feel free to use it.

[thejpster]

I just wanted to add that I think cargo-add is a great tool for people new to Rust, as is cargo-generate. Unfortunately if I recommend them in a guide or tutorial, I have to explain how to install them and that they need to first install GCC and cmake and libssl-dev, which is non-trivial.

[withoutboats]

Definitely would love to see this added to cargo, since my workflow setting up a new dev environment is:

1. Install Rust
2. Install ripgrep
3. cargo install cargo-edit

As @ibaryshnikov points out though, its nontrivial work since the TOML parser cargo uses right now doesn't preserve layout, whitespace, or comments. We need to migrate to using the same parser cargo-edit uses before we can upstream any of its features into cargo.

[withoutboats]

We discussed this in the cargo meeting. There was strong support for upstreaming the functionality of cargo add into cargo. We doubt it can be stable in time for the 2018 edition (since the final cargo master for the edition needs to be merged roughly 5 weeks from today) but we do think its good to start making progress on adding this functionality.

We are comfortable having two TOML parsers in cargo for now (with a view to eventually merging them) so long as they share a common lexer, to be confident they aren't interpreting the TOML significantly differently. @alexcrichton says he's left a comment on the repo for the parser cargo-edit is using to this effect.

[sandangel]

Hi everyone, may I know the status of this issue? It's 2019 now and Happy New Year. ^^

[Eh2406]

From the discord Today at 5:48 PM:
"ehuss: (btw, my format-preserving toml parser just passed all tests 🎉 😄 )"

[yhaskell]

Hello! Does anyone know what is the status of this?

[ordian]

In #5611 (comment) a concern was raised that we should (partially) deduplicate toml parsing libraries (cargo currently uses toml-rs, but that's not a lossless toml parser, e.g. it will eat your comments, cargo-edit currently uses toml_edit) before using it in cargo. Unfortunately, I don't have time to work on it (link to issues), but would be happy to mentor if someone wants to make this happen (if you're interested, please leave a message in https://gitter.im/toml_edit/Lobby).

After the above issues are resolved, we can hopefully revive #5611.

[tomharmon]

@SamedhG and I have started to work on updating toml_edit to support toml v0.5 We wanted to hear the opinions of the rust team and see which is a better option, or if there is not a significant difference between the two:

1. Add lossless parsing to toml_rs and adapt cargo_edit to use that instead of toml_edit, then merge cargo-edit into cargo proper
2. Update toml_edit to support toml v0.5 and update cargo to use toml_edit, then merge cargo-edit into cargo proper

@killercup @alexcrichton @steveklabnik @ordian

[ehuss]

I'd be very wary of having a second TOML parser to Cargo. I think it was previously mentioned that if they could share a lexer, that might be a compromise. I'm still concerned about adding a large dependency that has doubt around how well it is maintained.

I did work on an experimental preserving API for toml-rs, but never finished it. I do have the intention of picking it back up, but probably not for a long while.

Creating a good, clean API that handles all the intricacies of toml is quite difficult, and it will require some moderately invasive changes to toml-rs. I unfortunately don't have a lot of time to help with this right now.

[sylann]

Hello there,

Apparently this feature has been awaited for quite some time. No doubt the toml writing part has played a role in that.

I was wondering if the idea of just "printing the line that would be added to Cargo.toml" has been considered.

Finding the latest version of a given crate without leaving the terminal, and have it pre-written in the relevant format should be a good first step that solves most of the developer's trouble.

And if I'm not mistaken, it seems waaay simpler than trying to handle all the possible shenanigans of how an existing file could have been written.

Or did I miss something?

[epage]

I think going that route depends on the answer to the questions like:

• Can we change behavior away from that once toml_edit is ready? My guess is "no".
• Will taking over the "cargo add" name with diminished functionality be acceptable by end users? My guess is "no"

With that said, I was considering the idea of creating a command for interacting with the registry that could possibly fill the role of what you mentioned. Right now, cargo has its own copy of a lot of logic but isn't very good for depending on for external tools. Some of the logic gets reused through cargo metadata command which is wrapped by the cargo_metadata crate. A lot of other functionality gets duplicated instead, like crates_index crate. What if we instead made a cargo command for updating the local index and querying it and wrapping that in a crate? My hope is this would be an acceptable way for cargo to expose the registry code for reuse. You could then use this command as well to look up crates.

Also, #10086 now passes tests. All that is left is to audit the user facing changes (and adjust as needed) and I need to do benchmarking.

[Alexendoo]

@romain20100 Hi!

You can use cargo search for this, the results it returns are valid to be put in a Cargo.toml

[sylann]

@Alexendoo Sure, it's relatively trivial to adapt the result. In fact I've done just that with a simple bash function (see below).
But it's not immediately usable.

My point was just that there may be an intermediate milestone that could be simpler to achieve.
Although it's possible we are now close to a full on solution. In that case it's ok.

---

function cargo-add {
    output=$(cargo search $1)
    tmp="${output//$1 = \"}"
    version="${tmp//\"*}"
    echo "$1 = { version = \"$version\" }"
}

[lnicola]

@romain20100 you can already copy-paste the output of cargo search to Cargo.toml. E.g. serde = "1.0" is enough, you don't have to write it as serde = { version = "1.0" }.

I don't think there's any need for an intermediate step here. cargo edit can and will be added to cargo proper when it's considered ready.

[simonsan]

serde = { version = "1.0" }

At least from cargo-smart-release I know from the limitations section, that the version needs to be inside a table though.

Edit: By which I mean that there is sometimes this requirement, which probably should be put behind an option flag or somethign? 🤔 Not sure.

CC: @Byron

[epage]

serde = { version = "1.0" }

At least from cargo-smart-release I know from the limitations section, that the version needs to be inside a table though.

Edit: By which I mean that there is sometimes this requirement, which probably should be put behind an option flag or somethign? thinking Not sure.

imo the focus should be on fixing that git-smart-release limitation. The code exists for it in cargo-release and in cargo-set-version (in this repo).

[Byron]

Thanks for bringing up the issue @simonsan , I totally agree with @epage and think smart-release should learn to deal with non-table versions.

[epage]

As an update, #10086 has been merged which was the major blocker for cargo-edit being upstreamed.

I am now preparing cargo-add for merging.

• I've upgraded cargo-edit to the latest toml_edit, past some breaking changes that had a large effect on it
• I am currently porting the tests to trycmd which, through its snapshot updating, will make it easier to update tests as we port cargo-add to use cargo's facilities
• Next, I will fork cargo-add into a branch and will slowly refactor it into a state for merging
  • Isolating all of the logic cargo-add needs from the cargo-edit [lib]
  • Port parts of cargo-add from what was in cargo-edit [lib] to cargo
• Then we'll be ready for a draft PR with the proposal to merge, like cargo tree
  • I think an important part of this will be calling out the current CLI and re-examining parts of it (I think --upgrade in particular should be re-examined for how to expose that functionality)
  • I feel like this issue has too much noise in it already to have the conversation. Maybe it could be had on internals?
• We can then start prioritizing and preparing the next cargo-edit command

[epage]

FYI I just posted on internals looking for feedback before we restrict cargo-add's evolution by merging it in cargo.

[nyurik]

Is porting cargo upgrade part of this ticket, or would that be a separate one? I couldn't find any mentions of it besides this issue. Thx!

[epage]

The plan is to focus on cargo add as its seen as the highest priority. After that, I'll be looking to cargo rm because of the symmetry with add and it being a small lift. cargo upgrade will be after that and there are a lot of things we need to figure out (e.g. should it be kept as a separate command or expressed as flags on cargo update)

[epage]

I've started work on the port of cargo-add to cargo. The cargo-edit branch can be found at https://github.com/killercup/cargo-edit/tree/merge-add

If anyone wants to help out, steps we have include

• Separate out an "op"
• Port to cargo's internal metadata code
• Port to cargo's registry code
• Port to cargo's clap code
• Look into how we should be handling user-reporting

Feel free to coordinate with me on Zulip, gitter, or Discord

[epage]

Once I merge this last PR, the merge-add branch is complete. Tomorrow, I expect to try to move the code over to cargo and make any adjustments needed for being in-tree. After that, I'll be making the PR that proposes merging cargo-add into cargo.

[epage]

Its live!