Provide documentation hosting and integrate with cargo doc

[SimonSapin]

TL;DR: cargo publish should default to also running cargo doc and publishing the output online.

Having a crate’s documentation like http://doc.rust-lang.org/std/ available online is immensely valuable. Not only when using it, but also when deciding whether to use it (and so before cargo doc can be run locally).

Therefore, every crate on crates.io would ideally have its documentation online. For that to happen, it should be as easy as possible, with minimal effort from the developer.

The defaults should just work. cargo publish already involves building and testing crates, so documentation can be built at the same time. Even without any doc-comment, the rustdoc output can be useful. When the source package is uploaded to crates.io, the built documentation can be uploaded at the same time and be made available at https://example.net/[package-name]/ or https://example.net/[package-name]/[package-version]/, where example.net is a domain name managed together with crates.io. This URL should be linked from https://crates.io/crates/[package-name]

Although rustdoc is the default, if a project decides not to use rustdoc it should still be able to publish its documentation through this mechanism.

This effectively means providing free hosting of static files, which should be watched for abuse, but there is precedent: the Python Package Index does something similar with https://pythonhosted.org/. They have an HTTP API where one can upload a ZIP file, which the server exctracts to replace any previous content. Integration with Sphinx and distutils so that python setup.py upload_docs builds the documentation with Sphinx, zips it, and uploads it in one command.

Another case is Read The Docs, which pulls repositories and builds Sphinx documentation on its servers and can be trigerred on every commit by a GitHub hook. This is more risky as it involves running arbitrary code on the server.

If Cargo makes publishing/releasing a new version very easy, hopefully it can establish a convention in the Rust ecosystem of releasing often, so that updating the online docs for every commit (as opposed to releases) is not as necessary.

rust-lang/cargo#1016 would work well together with this. If rust-lang/cargo#1017 happens, it should be enabled for this.

Unresolved question: should the docs be made available for every version (at different URLs), or only the latest? If the former, should there be an unversioned URL for a default version? Is it a duplicate or a redirect? What navigation exists between versions? Read The Docs has a fixed-position overlay for this.

[steveklabnik]

I'm not sure if I prefer having it here or at rustdoc.org, which i believe @cmr owns.

(Ruby uses a separate site for external docs)

[SimonSapin]

rustdoc.org could be example.net above. I don’t care much how separate it’s considered to be, as long as cargo publish can integrate with it (and preferably does by default).

[nagisa]

Hackage hosts and builds the documentation for packages too, so there’s surely something that can be learnt from it.

I'm not sure if I prefer having it here or at rustdoc.org, which i believe @cmr owns.

IMO where exactly the docs are hosted is largely irrelevant as long as they are hosted and linked to automatically or at least semi-automatically.

[steveklabnik]

Oh, and crates can do http://www.steveklabnik.com/automatically_update_github_pages_with_travis_example/ in the meantime, too.

[SimonSapin]

@steveklabnik They can, but it involves jumping through a number of hoops and so only those who really want it are gonna do it. My point here is that it should be very easy, or even the default.

[steveklabnik]

Agreed. I only mention it as an interim solution :)

[lambda-fairy]

@steveklabnik I think it's better to host it on a separate domain, at least from a security POV. If we allow arbitrary HTML on crates.io, we'll have to worry about people stealing auth cookies and other scary things.

I believe GitHub Pages uses a separate domain (github.io) for the same reason.

[andrewrk]

Is security the only concern preventing us from implementing this?

[nagisa]

Lack of an implementation is the biggest blocker. Getting subdomain(s) or even a domain ready for this is a very minor hurdle.

[Manishearth]

As far as security goes, all we need to do is write the workflow so that cargo publish can push it through https://crates.io, but the actual HTML is shown on rustdoc.org or something. doc.crates.io works too, one can deny cookie-sharing. The website hosting the docs is entirely static, no authentication whatsoever.

[nagisa]

@Manishearth one can easily deploy arbitrary JavaScript, and another then could steal arbitrary cookies set by the first’s JS.

[Manishearth]

Yes, but that's not a security concern.

[SimonSapin]

IIRC cookie-related security concerns are why GitHub Pages and PyPI docs moved to github.io and pythonhosted.org. But that’s fine, we can use a separate domain as well.

[emberian]

rustdoc.org continues to be under my control and available.

[eternaleye]

There's also docs.rs - the owner has explicitly stated their willingness to pass it on to the core team.

[SimonSapin]

https://www.reddit.com/r/rust/comments/3fpo3r/youmu_online_rustdoc/ is an attempt at doing this with Travis-CI.

[ticki]

👍

[yberreby]

👍

[teh]

+1 - I frequently run into out-of-date docs on self hosted static sites (e.g. github.io)

[rgardner]

Is there any work going on in this area? I like the approach godoc takes where you can visit godoc.org/github.com/user/repo and get the specific documentation for that user.

What prevents the current travis-cargo doc-upload from being modified to upload docs to rustdoc.org instead of GitHub pages? Is it a security concern due to arbitrary JavaScript like: <script>alert(document.cookie)</script>, or is it just that no one has implemented this yet?

[huonw]

@rgardner this would preferably be integrated with the versions that are published to crates.io; godoc just shows the git HEAD (AFAIK) which may be arbitrarily different to the published versions, and travis-cargo certainly does.

[SimonSapin]

rustdoc.org doesn’t have any infrastructure yet.

[rgardner]

@huonw that makes sense that crates.io would have docs for each released version (instead of HEAD) and why using travis doesn't make sense for this problem.

So the blocker for this is the implementation hurdle of rolling our own GitHub Pages? (accepting static HTML/CSS/JS/font files?

[nagisa]

rustdoc output is not really suitable for proper documentation hosting service a-la readthedocs.org. rustdoc’s output only generates documentation for that particular package. In case of rustdoc.org/docs.rs/etc you’d probably want at least:

1. Cross-package search of some sort a-la hoogle;
2. Easy way to switch between packages and their versions;
3. Proper crosslinking between packages;
4. etc etc etc.

None of these are, or can be, provided by rustdoc as it is currently.

[Manishearth]

Cross-package search of some sort a-la hoogle;

Proper crosslinking between packages;

Actually rustdoc does do cross package search with proper crosslinking, provided you run rustdoc with the doc folder containing the rustdoc output for the deps (which cargo doc does, iirc). See http://doc.servo.org/.

[nagisa]

Actually rustdoc does do cross package search with proper crosslinking, provided you run rustdoc with the doc folder containing the rustdoc output for the deps (which cargo doc does, iirc). See http://doc.servo.org/.

It does seem to work, yes. However, it is not a very good idea to send every client the search index of the whole crates.io ecosystem.

As for cross-linking, does it still work properly when package A depends on C 0.1 and B depends on C 1.1?

[Manishearth]

You don't need to send it to each client for the crosslinking to work. You sort of do, for Hoogle-like search, but perhaps that can be done by concatenating search.js (or whatever it's called)

[ticki]

You sort of do, for Hoogle-like search

This could be an interesting project. Hoogle is definetly one of Haskell's strengths, I would love to see a similar thing for Rust. Rustdoc search still got much room for improvement.

[SimonSapin]

Rustdoc needs of lot of love, but that doesn't necessarily mean it should be dropped entirely.

And this is rather independent of what this issue is about: infrastructure to upload and host static files for a given crate and version.

[lambda-fairy]

Is anyone currently working on this?

If not then I'd like to pick up the issue. I used to work on the Hackage doc hosting machinery so I've got a good idea of the design space.

[Hywan]

👍

[alexcrichton]

@lfairy not currently, no, but we'd be excited to see movement here! We've talked about this in the past, and we likely don't want to commit to something too large just yet. Along those lines we've though:

• cargo publish will generate documentation for the host platform being published on
• The crate will be uploaded
• If successful, the documentation is also uploaded to a "default server"

We were thinking of using something like rustdoc.org for this where you'd just go to rustdoc.org/$crate/$version and you'd have all the rustdoc output. There's certainly lots more pieces to fill out in the long run, and even probably more pieces to fill out in the short term, but that's what we've thought will take us "most of the way there" with not a super high commitment.

[eternaleye]

@alexcrichton: One thing to consider, as documentation may vary with #[cfg], is whether docs should be uploaded to /$crate/$version/$hash_of_cfg_params, and /$crate/$version redirects to one based on some logic, but shows a banner at the top with a menu to select a different one if the default is wrong (probably saving them in a cookie so other crates use the same parameters).

(Aside: I'd do the hash as sorting keys by name, preceding each with + or - for enable/disablement, following with ="value" if a value is set, and separating with commas. So, +afoo,-bar,+baz="qux",-corge)

That would allow the dev, at their discretion, to upload docs for more configurations without clobbering any - and help with the Windows/Linux doc divide. (Maybe prompt "No documentation has been uploaded that matches your preferred config; displaying the nearest found" with another banner at the top, if needed.)

A first-take implementation could simply select the newest, and not bother with the banners until later.

(In the very long run, a way to integrate crates.io upload with a CI service, on a per-crate basis by the author, could help as well - then crate authors could simply hook up their Travis or Appveyor, and on crate upload it - using their resources, without crates.io needing a Travis or whatever setup itself - builds docs and hopefully runs tests. If the config-hash approach is taken, then the static upload infra could be reused.)

[alexcrichton]

@eternaleye yes a full blown "let's solve everyone's problems" solution to hosted docs would be significantly more complicated than what I outlined above. The idea is to that these kinds of features are the extra 10% which require a huge amount more work but don't bring that much benefit.

We would arguably massively benefit from just providing any solution to hosting docs by default (in the sense that by default crates on crates.io have documentation hosted somewhere).

The idea is to also allow ourselves some enough leeway to implement any number of extensions in the future (such as various features, platform specific docs, etc).

[eternaleye]

@alexcrichton My point is that the "first-take implementation" I laid out would be near-zero additional work on the server side (remember which one is newest for redirection), and very little overhead client-side (Cargo already has to know the cfg options for the conditional stuff that was added recently, so the only thing needed is formatting it and hashing, then sending the (cfg_string, docs) tuple to /$crate/$version/$hash).

In exchange, it can be upgraded to a better system with 100% compatibility, rather than breaking URLs or having to hack around simply not having the needed data. Because remember: Cool URIs don't change 😛

[alexcrichton]

Sure! So long as it's easy to implement seems reasonable to me

[onur]

Hi guys. I've been working on https://crates.fyi (repo) for a while and preparing a huge update. It's been fully functional since I announced it in reddit last month.

Current (outdated) version of crates.fyi is just pulling new releases from crates.io-index repository and generating documentations. I'll release an updated version of crates.fyi in few weeks, this update includes:

• Semver supported redirector to docs i.e: rustdoc.crates.fyi/rand will redirect to latest version, rustdoc.crates.fyi/rand/^0.1 will redirect to latest point release of 0.1 etc.
• Simple clean UI to see long description (documentation of root of crate, and also TOC) and README.md if readme is defined in Cargo.toml
• Examples and source browser.
• Github stars and issues if a crate is hosted in github.
• Graphviz based deep dependency graph.
• Recent releases (similar to metacpan).
• List of crates by authors and owners.
• Similar crates (based on keyword).

And my future plans:

• Test results (cargo test output).
• Stable, beta, nightly build status (this requires constant building and
  a build queue).
• Code coverage (when it becomes more usable).
• Community maintained categories (like web frameworks in areweweb or something similar to awesome-rust).
• Code search.
• A documentation builder for github repositories similar to godoc.org.

It's mainly influenced by my favorite repository browser: metacpan and I've finished most of the stuff and started working on UI. I was reluctant about developing something like this (expected to see something official from rust team). But looks like you guys are really determined about pushing self hosted docs.

And ofc, build specific docs is a huge problem, but I believe we need organized hosted docs to give an overview to user about what he is looking. And if a user decides to use this crate, he can always build and use his own cfg/platform specific doc in future.

@lfairy if you still wondering if someone is working on this issue, I am working on it.

I'd like to hear your ideas. And whole project is written in rust, but since I am learning rust along with this project, its taking a bit time :-)

[Manishearth]

Current (outdated) version of crates.fyi is just pulling new releases from crates.io-index repository and generating documentations

How do you handle sandboxing? Arbitrary code can get run during cargo doc.

I'd much prefer if cargo publish published docs built locally instead of having it done on the server. You don't have to worry about sandboxing and build times, then.

Godoc doesn't have this issue because go builds cannot run arbitrary code.

[onur]

I am building every crate in a chroot environment and using some pam_limits. It's pretty restricted environment and users in this build environment don't have any access to anything. I also prefer cargo publish if it's ever implemented. cratesfyi is just a tool to provide additional information (and currently missing hosted docs feature).

[nagisa]

Cross-platform documentation is a big pain point. You will almost never find documentation for e.g. Windows-specific modules. Whatever service spawns from this should support reading documentation from the perspective of at least all tier-1 platforms.

[alexcrichton]

@onur wow, what you've got planned sounds awesome! There's two possible things we may be able to do to help out as well:

1. Have docs uploaded on cargo publish directly to crates.fyi
2. Add webhooks on crates.io for when a crate is published.

That may be able to help out the multiple platform problem at least and also help keep docs up to date.

[steveklabnik]

Update: @onur has been doing a lot of work, and while it's not merged into mainline crates.fyi yet, it's shaping up really nicely.

[andrewdavidmackenzie]

+1 to having docs generated and published to crates.io as part of "cargo publish".

It would be very useful to be able to find a crate on crates.io and to browse it's documentation in the same site, before deciding to use it.

If there are other places it can be published, and either a command to do that ("cargo docs-publish"...), or "cargo docs" accepts an optional destination to publish to are all additional options.

[benaryorg]

@steveklabnik @onur is there any progress on this you can share?

Edit: or is there a site/place to look for updates on this?

[onur]

@benaryorg I forgot to update this thread, but check out recently released https://docs.rs.

[benaryorg]

Wow that looks beautiful.

If this fulfills all of the criteria then this issue could be closed I think.

[steveklabnik]

There's another issue discussing the integration, I would assume that if it
lands, it would close both this and that.

On Fri, Sep 2, 2016 at 4:27 AM, Katze notifications@github.com wrote:

Wow that looks beautiful.

If this fulfills all of the criteria then this issue could be closed I
think.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#91 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABsikLnM9WgpX0oBRqgOqBkm7tTws-Hks5ql93dgaJpZM4DFNBX
.

[carols10cents]

I'm going to close this since #676 has been merged and deployed!