Port books to mdbook #39633

steveklabnik · 2017-02-08T00:24:01Z

blocked on #39431

As a first step towards the bookshelf, we ~~vendor mdbook in-tree and~~ port our books to it. Eventually, both of these books will be moved out-of-tree, but the nightly book will rely on doing the same thing. As such, this intermediate step is useful.

r? @alexcrichton @brson

/cc @azerupi

steveklabnik · 2017-02-08T00:27:48Z

Oh, and pour one out for rustbook; you had a good run. /cc @aturon

alexcrichton · 2017-02-08T00:40:06Z

Looking good! I take it that the src/tools/mdBook directory is literally the source code of mdBook? If so, could we perhaps use a submodule instead?

Also, I think that you'll need to re-run cargo vendor to pull in mdBook's dependencies as well. Ideally though we'd only pull in a small number (e.g. the non-default ones). By default though a workspace member pulls in all optional dependencies and we don't have a way to turn this off. Would it be easy to depend on mdbook as a library and write a small wrapper that emulates https://github.com/azerupi/mdBook/blob/master/src/bin/mdbook.rs perhaps?

steveklabnik · 2017-02-08T00:41:41Z

If so, could we perhaps use a submodule instead?

Ah ha! I always forget about submodules; totally.

By default though a workspace member pulls in all optional dependencies and we don't have a way to turn this off. Would it be easy to depend on mdbook as a library and write a small wrapper that emulates https://github.com/azerupi/mdBook/blob/master/src/bin/mdbook.rs perhaps?

yes, that shouldn't be too bad. It's a tiny file.

alexcrichton · 2017-02-08T00:42:41Z

Ok in that case let's scratch the submodule idea and instead just depend on it as a library, vendor a small bit of the book logic, and then we can rely on cargo-vendor to pull in the vendor'd deps

steveklabnik · 2017-02-08T00:44:48Z

@alexcrichton I guess I'll call this wrapper........... rustbook.

IT LIVES

steveklabnik · 2017-02-08T02:06:37Z

@alexcrichton I think I did this correctly...

frewsxcv · 2017-02-08T02:10:01Z

Are all these fingerprint files intentional? I don't really know what they are, so maybe they are intentional?

steveklabnik · 2017-02-08T02:15:56Z

@frewsxcv not sure, i just ran cargo vendor.

steveklabnik · 2017-02-08T02:16:42Z

Hrm, looks like maybe the vendoring didn't work? https://travis-ci.org/rust-lang/rust/jobs/199465089#L371

est31 · 2017-02-08T02:20:29Z

I'm not sure, but I feel like this file shouldn't be there: https://github.com/steveklabnik/rust/blob/041312e1909be2954852317339629715d314e045/src/target/release/.fingerprint/kernel32-sys-4995bb2d2dfb906c/dep-build-script-build_script_build-4995bb2d2dfb906c

maybe some bug in cargo vendor?

steveklabnik · 2017-02-08T02:24:07Z

@est31 @frewsxcv it looks like cargo new didn't give me a .gitignore; ignoring that now

steveklabnik · 2017-02-08T02:25:50Z

@alexcrichton i added a src/.gitignore that ignores target; lmk if that's wrong

steveklabnik · 2017-02-08T02:58:32Z

It is unclear to me why this works for me locally but fails on travis :(

steveklabnik · 2017-02-08T03:08:08Z

@est31 points out to me on IRC that the bots are using make still, it seems, so that's probably why.

bors · 2017-02-08T06:41:58Z

☔ The latest upstream changes (presumably #39638) made this pull request unmergeable. Please resolve the merge conflicts.

azerupi · 2017-02-08T07:31:57Z

Would it be easy to depend on mdbook as a library and write a small wrapper that emulates

mdBook is designed to be able to be used as a library, but I think you are the first ones actually using it like that 😉 So don't hesitate to give feedback so that we can improve that use case.

steveklabnik · 2017-02-08T17:20:19Z

I am extremely confused.

$ git push origin -f vendor-mdbook
Counting objects: 778, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (248/248), done.
Writing objects: 100% (543/543), 726.68 KiB | 0 bytes/s, done.
Total 543 (delta 353), reused 358 (delta 283)
remote: Resolving deltas: 100% (353/353), completed with 152 local objects.
To git@github.com:steveklabnik/rust
 + bb66fa9...3812b66 vendor-mdbook -> vendor-mdbook (forced update)
steve@BECOMING:~/src/rust$ git push origin -f vendor-mdbook
Everything up-to-date

yet the update doesn't show up here. hmmmmmmmmmmmmmmmmmmm

steveklabnik · 2017-02-08T17:25:13Z

It looks like github is having some issues; I assume this PR will be updated soon, then.

steveklabnik · 2017-02-08T17:52:11Z

Okay, I don't know why the bot is failing here; I can run make tidy on my local machine. @alexcrichton any ideas?

(It fails with license issues, but it doesn't fail like it does on travis)

steveklabnik · 2017-02-08T17:56:02Z

Here is the summary of license issues:

invalid license Unlicense/MIT in /home/steve/src/rust/src/vendor/aho-corasick/Cargo.toml
invalid license MIT in /home/steve/src/rust/src/vendor/ansi_term/Cargo.toml
invalid license MIT in /home/steve/src/rust/src/vendor/clap/Cargo.toml
invalid license MIT in /home/steve/src/rust/src/vendor/handlebars/Cargo.toml
invalid license MIT in /home/steve/src/rust/src/vendor/kernel32-sys/Cargo.toml
invalid license MIT in /home/steve/src/rust/src/vendor/lazy_static/Cargo.toml
invalid license Unlicense/MIT in /home/steve/src/rust/src/vendor/memchr/Cargo.toml
invalid license MIT in /home/steve/src/rust/src/vendor/open/Cargo.toml
invalid license MPL-2.0 in /home/steve/src/rust/src/vendor/pest/Cargo.toml
invalid license MIT in /home/steve/src/rust/src/vendor/pulldown-cmark/Cargo.toml
invalid license MIT OR Apache-2.0 in /home/steve/src/rust/src/vendor/quick-error/Cargo.toml
invalid license MIT in /home/steve/src/rust/src/vendor/strsim/Cargo.toml
invalid license Apache-2.0 in /home/steve/src/rust/src/vendor/thread-id/Cargo.toml
invalid license Apache-2.0/MIT in /home/steve/src/rust/src/vendor/thread_local/Cargo.toml
invalid license Unlicense/MIT in /home/steve/src/rust/src/vendor/utf8-ranges/Cargo.toml
invalid license MIT in /home/steve/src/rust/src/vendor/winapi/Cargo.toml
invalid license MIT in /home/steve/src/rust/src/vendor/winapi-build/Cargo.toml

So, that's

Unlicense
MIT
MPL-2.0
Apache-2.0

All of the Unlicense projects are dual with MIT, so the real issue is pest, I think. Ironic that MPL would be the issue here 😉

azerupi · 2017-02-08T18:23:25Z

so the real issue is pest, I think. Ironic that MPL would be the issue here

mdBook is also licensed under the MPL 2.0, is that a problem?

steveklabnik · 2017-02-08T18:29:30Z

@azerupi interesting, wonder why it wasn't caught by the lint.

We're going to discuss it at the core team meeting today; it's not inherently a problem, but right now, everything in the repo is licensed identically. Moving away from that would be a big change, possibly.

steveklabnik · 2017-02-08T18:38:00Z

Updating myself: it appears that I did not run ./configure --enable-vendor when building, so now the travis issue reproduces for me. hrm.

It's not clear why, with the vendor, it says that it's looking at crates.io to get the source and can't find it.

steveklabnik · 2017-02-08T22:04:34Z

@azerupi expect a post on internals sometime in the near-ish future; we're going to ask everyone what they think 😄

steveklabnik · 2017-02-08T23:28:33Z

Internals post is here: https://internals.rust-lang.org/t/licenses-of-vendored-tools-in-the-rust-repo/4758

bors · 2017-02-08T23:38:50Z

☔ The latest upstream changes (presumably #39523) made this pull request unmergeable. Please resolve the merge conflicts.

Automate vendoring by invoking cargo-vendor when building src dist tarballs. This avoids #39633 bringing the `src/vendor` checked into git by #37524, past 200,000 lines of code. I believe the strategy of having rustbuild run `cargo vendor` during the `dist src` step is sound. However, the only way to be sure `cargo-vendor` exists is to run `cargo install --force cargo-vendor`, which will recompile it every time (not passing `--force` means you can't tell between "already exists" and "build error"). ~~This is quite suboptimal and I'd like to somehow do it in each `Dockerfile` that would need it.~~ * [ ] Cache `CARGO_HOME` (i.e. `~/.cargo`) between CI runs * `bin/cargo-vendor` and the actual caches are the relevant bits * [x] Do not build `cargo-vendor` all the time * ~~Maybe detect `~/.cargo/bin/cargo-vendor` already exists?~~ * ~~Could also try to build it in a `Dockerfile` but do we have `cargo`/`rustc` there?~~ * Final solution: check `cargo install --list` for a line starting with `cargo-vendor ` cc @rust-lang/tools

steveklabnik · 2017-02-14T17:50:45Z

@eddyb's PR has landed, 🎊

@bors: r=alexcrichton

bors · 2017-02-14T17:50:46Z

📌 Commit cacb3bc has been approved by alexcrichton

steveklabnik · 2017-02-14T17:55:21Z

@bors: p=1

I'm giving this a priority because there are a few PRs to the book that will need to be rebased after this PR, and this PR has already been sitting in the queue for three days.

bors · 2017-02-14T18:00:17Z

⌛ Testing commit cacb3bc with merge f19eb11...

bors · 2017-02-14T20:06:05Z

💔 Test failed - status-travis

steveklabnik · 2017-02-14T20:12:43Z

@bors: retry

bors · 2017-02-14T21:46:13Z

⌛ Testing commit cacb3bc with merge c3ab706...

bors · 2017-02-14T22:46:07Z

💔 Test failed - status-travis

alexcrichton · 2017-02-14T22:50:22Z

@bors: retry * #39785 flaky test

…

On Tue, Feb 14, 2017 at 4:46 PM, bors ***@***.***> wrote: 💔 Test failed - status-travis <https://travis-ci.org/rust-lang/rust/builds/201670238> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39633 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD95BItJ-2YVK6hmiQhWGL6j2lY0Bqgks5rci6zgaJpZM4L6OOs> .

bors · 2017-02-15T01:22:20Z

⌛ Testing commit cacb3bc with merge 025c328...

@alexcrichton

Port books to mdbook Part of #39588 blocked on #39431 As a first step towards the bookshelf, we ~vendor mdbook in-tree and~ port our books to it. Eventually, both of these books will be moved out-of-tree, but the nightly book will rely on doing the same thing. As such, this intermediate step is useful. r? @alexcrichton @brson /cc @azerupi

bors · 2017-02-15T03:49:16Z

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 025c328 to master...

hoodie · 2017-02-23T20:14:18Z

could we perhaps switch back to the rustbook color scheme?

rust-highfive assigned alexcrichton Feb 8, 2017

steveklabnik changed the title ~~Bring mdbook in-tree and port our books to it~~ Vendor mdbook in-tree and port our books to it Feb 8, 2017

steveklabnik force-pushed the vendor-mdbook branch from bb66fa9 to 3812b66 Compare February 8, 2017 17:22

steveklabnik mentioned this pull request Feb 8, 2017

Tracking issue for RFC 1828: Rust Bookshelf #39588

Closed

10 tasks

bors merged commit cacb3bc into rust-lang:master Feb 15, 2017

steveklabnik deleted the vendor-mdbook branch February 15, 2017 15:58

jameslzhu mentioned this pull request Jul 20, 2017

Update README to use mdbook over rustbook rust-unofficial/too-many-lists#79

Merged

kennytm mentioned this pull request Aug 25, 2017

Update RLS and Cargo #44028

Merged

Port books to mdbook #39633

Port books to mdbook #39633

Conversation

steveklabnik commented Feb 8, 2017 • edited Loading

steveklabnik commented Feb 8, 2017

alexcrichton commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

alexcrichton commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

frewsxcv commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

est31 commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

bors commented Feb 8, 2017

azerupi commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

azerupi commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

steveklabnik commented Feb 8, 2017 • edited Loading

steveklabnik commented Feb 8, 2017

steveklabnik commented Feb 8, 2017

bors commented Feb 8, 2017

steveklabnik commented Feb 14, 2017

bors commented Feb 14, 2017

steveklabnik commented Feb 14, 2017

bors commented Feb 14, 2017

bors commented Feb 14, 2017

steveklabnik commented Feb 14, 2017

bors commented Feb 14, 2017

bors commented Feb 14, 2017

alexcrichton commented Feb 14, 2017 via email

bors commented Feb 15, 2017

bors commented Feb 15, 2017

hoodie commented Feb 23, 2017

steveklabnik commented Feb 8, 2017 •

edited

Loading

steveklabnik commented Feb 8, 2017 •

edited

Loading