From 542dcbb0d13083742451255c9b22544daa249a98 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Wed, 19 Jun 2024 02:52:00 +0000 Subject: [PATCH 01/28] add `mergable_rustdoc_cross_crate_info` --- ...0000-mergeable-rustdoc-cross-crate-info.md | 482 ++++++++++++++++++ 1 file changed, 482 insertions(+) create mode 100644 text/0000-mergeable-rustdoc-cross-crate-info.md diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md new file mode 100644 index 00000000000..5fc720efdca --- /dev/null +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -0,0 +1,482 @@ +- Feature Name: `mergable_rustdoc_cross_crate_info` +- Start Date: 2024-06-18 +- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) +- Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) + +# rustdoc Proposal - Merge Documentation From Multiple Crates + +# Work in progress + + + +# Summary + +Mergeable cross-crate information in rustdoc. Facilitates the generation of documentation indexes in environments with many crates by allowing each crate to write to an independent output directory. Final documentation is generated with a lightweight merge step. Adds a `.parts/` directory for pre-merge cross-crate information. Currently, rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating rustdoc in build systems that make build actions independent. + +# Motivation + +The main goal of this proposal is to facilitate users producing a documentation bundle of every crate in a large workspace. If a crate in the workspace is re-documented, only a relatively lightweight merge step should need to be performed to produce a complete documentation bundle. This proposal is to facilitate the creation and updating of these bundles. + +There are some files in the rustdoc output directory that are read and overwritten during every invocation of rustdoc. This proposal refers to these files as **cross-crate information**, or **CCI**, as in . + +Build systems may run build actions in a distributed environment across separate logical filesystems. It might also be desirable to run rustdoc in a lock-free parallel mode, where every rustdoc process writes to a disjoint set of files. + +Cargo fully supports cross-crate information, at the cost of requiring read-write access to a shared `target/doc/` directory. There are significant scalability issues with this approach. + +rustdoc needing global mutable access to the files that encode this cross-crate information has implications for caching, reproducible builds, and content hashing. By adding an option to avoid this mutation, rustdoc can serve as a first-class citizen in non-cargo build systems. + +These considerations motivate adding an option for outputting partial CCI (parts), which are merged (linked) with a later step. + + + + +# Guide-level explanation + +In this example, there is a crate `t` which defines a trait `T`, and a crate `s` which defines a struct `S` that implements `T`. Our goal in this demo is for `S` to appear as in implementer in `T`'s docs, even if `s` and `t` are documented independently. This guide will be assuming that we want a crate `i` that serves as our documentation index. See the Unresolved questions section for ideas that do not require an index crate. + +```shell +mkdir -p t/src s/src i/src +echo "pub trait T {}" > t/src/lib.rs +echo "pub struct S; impl t::T for S {}" > s/src/lib.rs +``` + +[Actively use](https://doc.rust-lang.org/rustc/command-line-arguments.html#--extern-specify-where-an-external-library-is-located) `t` and `s` in `i`. The `extern crate` declarations are not needed if you reference the crates in another way in the index; intra-doc links are enough. + +```shell +echo "extern crate t; extern crate s;" > i/src/lib.rs +``` + +Compile the crates. + +```shell +rustc --crate-name=t --crate-type=lib --edition=2021 --emit=metadata --out-dir=t/target t/src/lib.rs +rustc --crate-name=s --crate-type=lib --edition=2021 --emit=metadata --out-dir=s/target --extern t=t/target/libt.rmeta s/src/lib.rs +rustc --crate-name=i --crate-type=lib --edition=2021 --emit=metadata --out-dir=i/target --extern s=s/target/libs.rmeta --extern t=t/target/libt.rmeta -L t/target i/src/lib.rs +``` + +Document `s` and `t` independently, providing `--merge-parts=false`. + +```shell +rustdoc -Z unstable-options --crate-name=t --crate-type=lib --edition=2021 --out-dir=t/target/doc --extern-html-root-url t=merged/doc --merge-parts=false t/src/lib.rs +rustdoc -Z unstable-options --crate-name=s --crate-type=lib --edition=2021 --out-dir=s/target/doc --extern-html-root-url s=merged/doc --extern-html-root-url t=merged/doc --merge-parts=false --extern t=t/target/libt.rmeta s/src/lib.rs +``` + +Link everything with a final invocation of rustdoc on `i`. We will **not** provide `--merge-parts=false`, because we are merging the parts. We must also provide `--extern-parts-path`. See the Reference-level explanation about this flag. + +```shell +rustdoc -Z unstable-options --crate-name=i --crate-type=lib --edition=2021 --enable-index-page --out-dir=i/target/doc/ --extern-html-root-url s=merged/doc/ --extern-html-root-url t=merged/doc/ --extern-html-root-url i=merged/doc/ --extern-parts-path t=t/target/doc/ --extern-parts-path s=s/target/doc/ --extern t=t/target/libt.rmeta --extern s=s/target/libs.rmeta -L t/target i/src/lib.rs +``` + +Merge the docs with `cp`. This can be avoided if `--out-dir=merged/doc` is used for all of the rustdoc calls. We copy here to illustrate that documenting `s` is independent of documenting `t`, and could happen on separate machines. + +```shell +mkdir -p merged/doc +cp -r s/target/doc/* t/target/doc/* i/target/doc/* merged/doc +``` + +Browse `merged/doc/index.html` with cross-crate information. + +In general, instead of two crates in the environment (`s` and `t`) you could have thousands. Upon any changes, only the index and the crates that are changed have to be re-documented. + + +
+Click here for a directory listing after running the example above. + +
+$ tree . -a
+.
+├── i
+│   ├── src
+│   │   └── lib.rs
+│   └── target
+│       ├── doc
+│       │   ├── crates.js
+│       │   ├── help.html
+│       │   ├── i
+│       │   │   ├── all.html
+│       │   │   ├── fn.add.html
+│       │   │   ├── index.html
+│       │   │   └── sidebar-items.js
+│       │   ├── index.html
+│       │   ├── .lock
+│       │   ├── .parts
+│       │   │   └── i
+│       │   │       ├── crates-js
+│       │   │       ├── index-html
+│       │   │       ├── search-index-js
+│       │   │       ├── src-files-js
+│       │   │       ├── trait-impl
+│       │   │       └── type-impl
+│       │   ├── search.desc
+│       │   │   └── i
+│       │   │       └── i-desc-0-.js
+│       │   ├── settings.html
+│       │   ├── src
+│       │   │   └── i
+│       │   │       └── lib.rs.html
+│       │   ├── src-files.js
+│       │   └── static.files
+│       │       ├── COPYRIGHT-23e9bde6c69aea69.txt
+│       │       ├── favicon-2c020d218678b618.svg
+│       │       ├── favicon-32x32-422f7d1d52889060.png
+│       │       ├── FiraSans-LICENSE-db4b642586e02d97.txt
+│       │       ├── FiraSans-Medium-8f9a781e4970d388.woff2
+│       │       ├── FiraSans-Regular-018c141bf0843ffd.woff2
+│       │       ├── LICENSE-APACHE-b91fa81cba47b86a.txt
+│       │       ├── LICENSE-MIT-65090b722b3f6c56.txt
+│       │       ├── main-20a3ad099b048cf2.js
+│       │       ├── NanumBarunGothic-0f09457c7a19b7c6.ttf.woff2
+│       │       ├── NanumBarunGothic-LICENSE-18c5adf4b52b4041.txt
+│       │       ├── normalize-76eba96aa4d2e634.css
+│       │       ├── noscript-df360f571f6edeae.css
+│       │       ├── rustdoc-dd39b87e5fcfba68.css
+│       │       ├── rust-logo-151179464ae7ed46.svg
+│       │       ├── scrape-examples-ef1e698c1d417c0c.js
+│       │       ├── search-0fe7219eb170c82e.js
+│       │       ├── settings-4313503d2e1961c2.js
+│       │       ├── SourceCodePro-It-1cc31594bf4f1f79.ttf.woff2
+│       │       ├── SourceCodePro-LICENSE-d180d465a756484a.txt
+│       │       ├── SourceCodePro-Regular-562dcc5011b6de7d.ttf.woff2
+│       │       ├── SourceCodePro-Semibold-d899c5a5c4aeb14a.ttf.woff2
+│       │       ├── SourceSerif4-Bold-a2c9cd1067f8b328.ttf.woff2
+│       │       ├── SourceSerif4-It-acdfaf1a8af734b1.ttf.woff2
+│       │       ├── SourceSerif4-LICENSE-3bb119e13b1258b7.md
+│       │       ├── SourceSerif4-Regular-46f98efaafac5295.ttf.woff2
+│       │       ├── src-script-e66d777a5a92e9b2.js
+│       │       └── storage-118b08c4c78b968e.js
+│       └── libi.rmeta
+├── merged
+│   └── doc
+│       ├── crates.js
+│       ├── help.html
+│       ├── i
+│       │   ├── all.html
+│       │   ├── fn.add.html
+│       │   ├── index.html
+│       │   └── sidebar-items.js
+│       ├── index.html
+│       ├── s
+│       │   ├── all.html
+│       │   ├── index.html
+│       │   ├── sidebar-items.js
+│       │   └── struct.S.html
+│       ├── search.desc
+│       │   └── i
+│       │       └── i-desc-0-.js
+│       ├── settings.html
+│       ├── src
+│       │   ├── i
+│       │   │   └── lib.rs.html
+│       │   ├── s
+│       │   │   └── lib.rs.html
+│       │   └── t
+│       │       └── lib.rs.html
+│       ├── src-files.js
+│       ├── static.files
+│       │   ├── COPYRIGHT-23e9bde6c69aea69.txt
+│       │   ├── favicon-2c020d218678b618.svg
+│       │   ├── favicon-32x32-422f7d1d52889060.png
+│       │   ├── FiraSans-LICENSE-db4b642586e02d97.txt
+│       │   ├── FiraSans-Medium-8f9a781e4970d388.woff2
+│       │   ├── FiraSans-Regular-018c141bf0843ffd.woff2
+│       │   ├── LICENSE-APACHE-b91fa81cba47b86a.txt
+│       │   ├── LICENSE-MIT-65090b722b3f6c56.txt
+│       │   ├── main-20a3ad099b048cf2.js
+│       │   ├── NanumBarunGothic-0f09457c7a19b7c6.ttf.woff2
+│       │   ├── NanumBarunGothic-LICENSE-18c5adf4b52b4041.txt
+│       │   ├── normalize-76eba96aa4d2e634.css
+│       │   ├── noscript-df360f571f6edeae.css
+│       │   ├── rustdoc-dd39b87e5fcfba68.css
+│       │   ├── rust-logo-151179464ae7ed46.svg
+│       │   ├── scrape-examples-ef1e698c1d417c0c.js
+│       │   ├── search-0fe7219eb170c82e.js
+│       │   ├── settings-4313503d2e1961c2.js
+│       │   ├── SourceCodePro-It-1cc31594bf4f1f79.ttf.woff2
+│       │   ├── SourceCodePro-LICENSE-d180d465a756484a.txt
+│       │   ├── SourceCodePro-Regular-562dcc5011b6de7d.ttf.woff2
+│       │   ├── SourceCodePro-Semibold-d899c5a5c4aeb14a.ttf.woff2
+│       │   ├── SourceSerif4-Bold-a2c9cd1067f8b328.ttf.woff2
+│       │   ├── SourceSerif4-It-acdfaf1a8af734b1.ttf.woff2
+│       │   ├── SourceSerif4-LICENSE-3bb119e13b1258b7.md
+│       │   ├── SourceSerif4-Regular-46f98efaafac5295.ttf.woff2
+│       │   ├── src-script-e66d777a5a92e9b2.js
+│       │   └── storage-118b08c4c78b968e.js
+│       └── t
+│           ├── all.html
+│           ├── index.html
+│           ├── sidebar-items.js
+│           └── trait.T.html
+├── s
+│   ├── src
+│   │   └── lib.rs
+│   └── target
+│       ├── doc
+│       │   ├── help.html
+│       │   ├── .lock
+│       │   ├── .parts
+│       │   │   └── s
+│       │   │       ├── crates-js
+│       │   │       ├── index-html
+│       │   │       ├── search-index-js
+│       │   │       ├── src-files-js
+│       │   │       ├── trait-impl
+│       │   │       └── type-impl
+│       │   ├── s
+│       │   │   ├── all.html
+│       │   │   ├── index.html
+│       │   │   ├── sidebar-items.js
+│       │   │   └── struct.S.html
+│       │   ├── settings.html
+│       │   └── src
+│       │       └── s
+│       │           └── lib.rs.html
+│       └── libs.rmeta
+└── t
+    ├── src
+    │   └── lib.rs
+    └── target
+        ├── doc
+        │   ├── help.html
+        │   ├── .lock
+        │   ├── .parts
+        │   │   └── t
+        │   │       ├── crates-js
+        │   │       ├── index-html
+        │   │       ├── search-index-js
+        │   │       ├── src-files-js
+        │   │       ├── trait-impl
+        │   │       └── type-impl
+        │   ├── settings.html
+        │   ├── src
+        │   │   └── t
+        │   │       └── lib.rs.html
+        │   └── t
+        │       ├── all.html
+        │       ├── index.html
+        │       ├── sidebar-items.js
+        │       └── trait.T.html
+        └── libt.rmeta    
+
+ +
+ + +# Reference-level explanation + +Currently, CCI is written during the invocation of the `write_shared` function in [write_shared.rs](https://github.com/rust-lang/rust/blob/04ab7b2be0db3e6787f5303285c6b2ee6279868d/src/librustdoc/html/render/write_shared.rs#L47). This proposal does not add any new CCI or change their contents (modulo sorting order, whitespace). + +The existing cross-crate information files, like `search-index.js`, all happen to be lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc/`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of a the CCI. + +## New subdirectory: `doc/.parts//` + +The `doc/.parts//` files contain the unmerged contents of a single crates' version of their corresponding CCI. + +Every file in `doc/.parts//*` is a JSON array. Every element of the +array is a two-element array: a destination filename (relative to `doc/`), and +the representation of the part. The representation of that part depends on the type +of CCI that it describes. + +* `doc/.parts//src-files-js`: for `doc/src-files.js` + +This part is the JSON representation of the source index that is later stored in a `srcIndex` global variable. + +* `doc/.parts//search-index-js`: for `doc/search-index.js` + +This part is the JSON encoded search index, before it has been installed in `search-index.js`. + +* `doc/.parts//search-desc`: for `doc/search.desc/**/*.js` + +This part contains the JavaScript code to load a shard of the search descriptions. + +* `doc/.parts//all-crates`: for `doc/crates.js`, `/doc/index.html` + +This part is the crate name. + +* `doc/.parts//crates-index`: for `doc/crates.js`, `doc/index.html` + +This part is the also crate name. It represents a different kind of CCI because it is written to a `doc/index.html`, and rendered as an HTML document instead as JSON. In principal, we could use this part to add more information to the crates index `doc/index.html` (the first line of the top level crate documentation, for example). + +* `doc/.parts//type-impl`: for `doc/type.impl/**/*.js` + +This part is a two element array with the crate name and the JSON representation of a type implementation. + +* `doc/.parts//trait-impl`: for `doc/trait.impl/**/*.js` + +This part is a two element array with the crate name and the JSON representation of a trait implementation. + +## New flag: `--merge-parts[=true|false]` + +This flag defaults to true if not specified. + +With this flag set to false, rustdoc will generate `doc/.parts` and its contents, eagerly merge and render the parts, and write the rendered CCI to the output directory, on each invocation. The user-facing behavior without the flag is the same as the current behavior, with the addition of a new `doc/.parts` directory. + +When this flag is enabled, the `doc/.parts` directory and its contents will be generated, but the CCI will not be written. None of the CCI will be rendered. Another call to a merge step will be required to merge the parts and write the CCI. + +## New flag: `--extern-parts-path =` + +rustdoc considers a crate to be locally documented if its documentation appears in the current output directory. A crate is externally documented if its documentation cannot be found there. Externally documented crates may be documented online, or elsewhere in the filesystem. + +If rustdoc [identifies](https://github.com/rust-lang/rust/blob/dd104ef16315e2387fe94e8c43eb5a66e3dbd660/src/librustdoc/clean/types.rs#L184C7-L187C10) a crate as being documented locally, it will expect `doc/.parts/` to contain the parts. This flag is ignored in the case of locally documented crates. + +If rustdoc identifies that a crate is externally documented but a user wishes to add it to the CCI, rustdoc must be able to locate the parts. In this case, pass `--extern-parts-location =`. rustdoc will then assume that the parts are in `/.parts//`. + +This flag is similar to `--extern-html-root-url` in that it only applies to externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided is never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--extern-parts-path` tells rustdoc where to search for the `.parts` directory at documentation-time. It must not be a hyperlink. + +In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc/.parts/s`, so rustdoc is called with `--extern-parts-location s=s/target/doc/`. + +## Merge step + +This step is provided with a list of crates. It merges their documentation. This step involves copying parts (individual item, module documentation) from each of the provided crates. It merges the parts, renders, and writes the CCI to `doc/`. + +Discussion of the merge step is described in the Unresolved questions. + +# Drawbacks + +The WIP might change the sorting order of the elements in the CCI. It does not change the content of the documentation, and is intended to work without modifying Cargo and docs.rs. + +It also proposes to add a `doc/.parts` directory, unconditionally. This is discussed in Unresolved questions. + +# Rationale and alternatives + +Running rustdoc in parallel is essential in enabling the tool to scale to large projects. Cargo implements parallel rustdoc by locking the CCI files. There are some environments where having synchronized access to the CCI is impossible. This proposal implements a reasonable approach to shared rustdoc, because it cleanly enables the addition of new kinds of CCI. + +# Prior art + +Prior art for linking and merging independently generated documentation was **not** identified in Javadoc, Godoc, Doxygen, Sphinx (intersphinx), nor any documentation system for other languages. Analogs of cross-crate information were not found, but a more thorough investigation or experience with other systems may be needed. + +However, the issues presented here have been encountered in multiple build systems that interact with rustdoc. They limit the usefulness of rustdoc in large environments. + +## Bazel + +Bazel has `rules_rust` for building Rust targets and rustdoc documentation. + +* +* + +It does not document crates' dependencies. `search-index.js`, for example, is both a dependency and an output file for rustdoc in multi-crate documentation environments. If it is declared as a dependency in this way, Bazel could not build docs for the members of an environment in parallel with a single output directory, as it strictly enforces hermiticity. For a recursive, parallel rustdoc to ever serve as a first-class citizen in Bazel, changes similar to the ones described in this proposal would be needed. + +There is an [open issue](https://github.com/bazelbuild/rules_rust/issues/1837) raised about the fact that Bazel does not document crates dependencies. The comments in the issue discuss a pull request on Bazel that documents each crates dependencies in a separate output directory. It is noted in the discussion that this solution, being implemented under the current rustdoc, "doesn't scale well and it should be implemented in a different manner long term." In order to get CCI in a mode like this, rustdoc would need to adopt changes, like the ones in this proposal, for merging cross-crate information. + +## Buck2 + +The Buck2 build system has rules for building and testing rust binaries and libraries. + + + + +It has a subtarget, `[doc]`, for generating rustdoc for a crate. + +You can provide a coarse-grained `extern-html-root-url` for all dependencies. You could document all crates independently, but no cross-crate information would be shared. + +It does not document crates' dependencies for the same reason that Bazel does not. + + + + +## Ninja [(GN)](https://fuchsia.dev/fuchsia-src/development/build/build_system/intro) + Fuchsia + +Currently, the Fuchsia project runs rustdoc on all of their crates to generate a [documentation index](https://fuchsia-docs.firebaseapp.com/rust/rustdoc_index/). This index is effectively generated as an [atomic step](https://cs.opensource.google/fuchsia/fuchsia/+/main:tools/devshell/contrib/lib/rust/rustdoc.py) in the build system. It takes [3 hours](https://ci.chromium.org/ui/p/fuchsia/builders/global.ci/firebase-docs/b8744777376580022225/overview) to document the ~2700 crates in the environment. With this proposal, building each crate's documentation could be done as separate build actions, which would have a number of benefits. These include parallelism, caching (avoid rebuilding docs unnecessarily), and robustness (automatically reject pull requests that break documentation). + + + + + +# Unresolved questions + +## Index crate? + +Require users to generate documentation bundles via an index crate (current) vs. creating a new mode to allow rustdoc to run without a target crate (proposed) + +If one would like to merge the documentation of several crates, we could continue to require users to provide an index crate, like [the fuchsia index](https://fuchsia-docs.firebaseapp.com/rust/rustdoc_index/). This serves as the target of the rustdoc invocation, and the landing page for the collected documentation. Supporting only this style of index would require the fewest changes. This is the mode described in the Guide-level explanation. + +The proposition, to allow users of rustdoc the flexibility of not having to produce an index, is to allow rustdoc to be run in a mode where no target crate is provided. The source crates are provided to rustdoc, through a mechanism like `--extern`, rustdoc merges and writes the CCI, and copies the item and module links to `doc/`. This would require more extensive changes, as rustdoc assumes that it is invoked with a target crate. This mode is somewhat analogous to the [example scraping mode](https://github.com/rust-lang/rfcs/blob/master/text/3123-rustdoc-scrape-examples.md). Having to create an index crate, that actively uses all of the crates in the environment, might prohibit the use of this feature in settings where users do not intend to produce an index, or where exhaustively listing all dependencies (to `--extern` them) is difficult. + +## Unconditionally generating the `doc/.parts/*/*` files? + +Generate no extra files (current) vs. unconditionally creating `doc/.parts` to enable more complex future CCI (proposed) + +The current version of rustdoc performs merging by [collecting JSON](https://github.com/rust-lang/rust/blob/c25ac9d6cc285e57e1176dc2da6848b9d0163810/src/librustdoc/html/render/write_shared.rs#L166) blobs from the contents of the already-rendered CCI. This proposal performs merging by reading contents of the unmerged `doc/.parts/` files. This is simpler even if you don't care about deferring CCI generation, as it uses the existing `serde_json` dependency for extracting the blobs, as opposed to handwritten CCI-type specific parsing (current). + +The **alternative** to writing the parts to the `doc/.parts/` directory is to continue extracting them from the rendered versions of the CCI. There are several issues with this current practice +* If a user has a single `doc/` output directory, it is impossible to avoid shared mutation if every rustdoc process is writing to the same CCI +* It is difficult to extract the items in a diverse set of rendered HTML files. This is articipating of the CCI to include HTML files that, for example, statically include type+trait implementations directly +* Parts would need to be de-duplicated if a crate is a dependency of multiple crates, given that we now support merging. + +It is the authors' preference to merge by reading the `doc/.parts` files, even with `--merge-parts=false`. There would be no need for duplicate logic to read from both `doc/.parts` files and rendered CCI, because it would always use `doc/.parts` as the source for the merge. + +## Item links? + +Require users to pass `--extern-html-root-url` on all invocations of rustdoc to generate individually documented crates (current), vs. add a new flag to facilitate missing docs links being generated across all user crates (proposed) + +Cross-crate links are an important consideration when merging documentation. rustdoc provides a flag, `--extern-html-root-url`, which can provide fine-grain control over the generated URL for documentation. We believe this is already sufficient for generating cross-crate links, and we will make no changes to the generation of item links. Example usage of this flag is in the Guide-level explanation. + +A proposal is to provide more options to facilitate cross-crate links. For example, we could add a flag that implies `--extern-html-root-url` and `--extern-parts-path` on all crates passed through `--extern`. It could be something like `--extern-html-root-url-all-user-crates`. User crates would be taken to mean the transitive dependencies of the current crate, excluding the standard library. This is complicated by things like . + +## Reuse existing option? + +Create a new flag, `--no-merge-parts` (proposed), vs. use existing option `no_emit_shared` (current) + +There is a render option, `no_emit_shared`, which is used to conditionally generate the cross-crate information. It also controls the generation of static files, user CSS, etc. + +This option is not configurable from the command line, and appears to be enabled unless rustdoc is run in its example scraping mode. + +We could make it configurable from the command line, unconditionally generate `.parts/`, and use it to gate the merging of CCI. + +We could also make it configurable from the command line, and use it to gate the generation of `.parts/` and the generation of all of the shared files. + +We could also leave it as-is: always false unless we're scraping examples, and gate the generation of `.parts/` and the generation of all of the shared files. + + + +## Location of the `.parts` directory? + +Generate `.parts` as a child of the documentation root (proposed) vs. choose another, possibly user selected, location (other proposal) + +The documentation root houses the files in the generated site, while `.parts` holds intermediate output. Some may believe that `.parts` would bloat the generated site. + +There is no other obvious location for `.parts`. We might like for `.parts` to be a child of `target/`, not a child of `target/doc/`. However, rustdoc is never made aware of the `target/` directory. Instead, `--out-dir` always refers to the documentation root. + +One idea is to add an optional command-line argument to configure its location: `--parts-out-dir`, defaulting to `/.parts`. Users may configure it to be `target/.parts` if they have a target directory. + +# Future possibilities + +This change could begin to facilitate type aliases and the trait aliases being +statically compiled as part of the .html documentation, instead of being loaded +as separate JavaScript files. Each type and trait alias could be stored as an +HTML part, which are then merged into the regular documentation. + +Another possibility is for `.parts/` to be distributed on `docs.rs` along with the regular documentation. This would facilitate a mode where documentation of the dependencies could be downloaded externally, instead of being rebuilt locally. + +A future possibility related to the index crate idea is to have an option for embedding user-specified HTML into the `--enable-index-page`'s HTML. + +The changes in this proposal are intended to work with no changes to Cargo and docs.rs. However, there may be benefits to using `--merge-parts=false` with Cargo, as it would remove the need for locking the output directory. More of the documentation process could happen in parallel, which may speed up execution time.. From fdc1b70e3b104fac3675940f74203e440a7c3899 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Thu, 20 Jun 2024 16:58:58 +0000 Subject: [PATCH 02/28] generates no files by default --- ...0000-mergeable-rustdoc-cross-crate-info.md | 365 +++++------------- 1 file changed, 92 insertions(+), 273 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index 5fc720efdca..47307ab9aa0 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -1,9 +1,9 @@ -- Feature Name: `mergable_rustdoc_cross_crate_info` +- Feature Name: `mergable_Rustdoc_cross_crate_info` - Start Date: 2024-06-18 - RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) - Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) -# rustdoc Proposal - Merge Documentation From Multiple Crates +# Rustdoc Proposal - Merge Documentation From Multiple Crates # Work in progress @@ -11,19 +11,19 @@ # Summary -Mergeable cross-crate information in rustdoc. Facilitates the generation of documentation indexes in environments with many crates by allowing each crate to write to an independent output directory. Final documentation is generated with a lightweight merge step. Adds a `.parts/` directory for pre-merge cross-crate information. Currently, rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating rustdoc in build systems that make build actions independent. +Mergeable cross-crate information in Rustdoc. Facilitates the generation of documentation indexes in environments with many crates by allowing each crate to write to an independent output directory. Final documentation is generated with a lightweight merge step. Configurable with command-line flags, this proposal writes a `doc.parts` directory to hold pre-merge cross-crate information. Currently, Rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating Rustdoc in build systems that make build actions independent. # Motivation -The main goal of this proposal is to facilitate users producing a documentation bundle of every crate in a large workspace. If a crate in the workspace is re-documented, only a relatively lightweight merge step should need to be performed to produce a complete documentation bundle. This proposal is to facilitate the creation and updating of these bundles. +The main goal of this proposal is to facilitate users producing a documentation bundle of every crate in a large environment. When a crate needs to be re-documented, only a relatively lightweight merge step will be needed to produce a complete documentation bundle. This proposal is to facilitate the creation and updating of these bundles. -There are some files in the rustdoc output directory that are read and overwritten during every invocation of rustdoc. This proposal refers to these files as **cross-crate information**, or **CCI**, as in . +There are some files in the Rustdoc output directory that are read and overwritten during every invocation of Rustdoc. This proposal refers to these files as **cross-crate information**, or **CCI**, as in . -Build systems may run build actions in a distributed environment across separate logical filesystems. It might also be desirable to run rustdoc in a lock-free parallel mode, where every rustdoc process writes to a disjoint set of files. +Build systems may run build actions in a distributed environment across separate logical filesystems. It might also be desirable to run Rustdoc in a lock-free parallel mode, where every Rustdoc process writes to a disjoint set of files. -Cargo fully supports cross-crate information, at the cost of requiring read-write access to a shared `target/doc/` directory. There are significant scalability issues with this approach. +Cargo fully supports cross-crate information, at the cost of requiring read-write access to the documentation root (`target/doc/`). There are significant scalability issues with this approach. -rustdoc needing global mutable access to the files that encode this cross-crate information has implications for caching, reproducible builds, and content hashing. By adding an option to avoid this mutation, rustdoc can serve as a first-class citizen in non-cargo build systems. +Rustdoc needing global mutable access to the files that encode this cross-crate information has implications for caching, reproducible builds, and content hashing. By adding an option to avoid this mutation, Rustdoc will serve as a first-class citizen in non-cargo build systems. These considerations motivate adding an option for outputting partial CCI (parts), which are merged (linked) with a later step. @@ -37,9 +37,10 @@ This proposal also has the goal of enabling cross-crate links for items whose do In this example, there is a crate `t` which defines a trait `T`, and a crate `s` which defines a struct `S` that implements `T`. Our goal in this demo is for `S` to appear as in implementer in `T`'s docs, even if `s` and `t` are documented independently. This guide will be assuming that we want a crate `i` that serves as our documentation index. See the Unresolved questions section for ideas that do not require an index crate. ```shell -mkdir -p t/src s/src i/src +mkdir -p t/src s/src i/src merged/doc echo "pub trait T {}" > t/src/lib.rs echo "pub struct S; impl t::T for S {}" > s/src/lib.rs +MERGED=$(realpath merged/doc) ``` [Actively use](https://doc.rust-lang.org/rustc/command-line-arguments.html#--extern-specify-where-an-external-library-is-located) `t` and `s` in `i`. The `extern crate` declarations are not needed if you reference the crates in another way in the index; intra-doc links are enough. @@ -56,23 +57,22 @@ rustc --crate-name=s --crate-type=lib --edition=2021 --emit=metadata --out-dir=s rustc --crate-name=i --crate-type=lib --edition=2021 --emit=metadata --out-dir=i/target --extern s=s/target/libs.rmeta --extern t=t/target/libt.rmeta -L t/target i/src/lib.rs ``` -Document `s` and `t` independently, providing `--merge-parts=false`. +Document `s` and `t` independently, providing `--write-merged-cci=false`, `--read-merged-cci=false`, and `--parts-out-dir=/target/doc.parts` ```shell -rustdoc -Z unstable-options --crate-name=t --crate-type=lib --edition=2021 --out-dir=t/target/doc --extern-html-root-url t=merged/doc --merge-parts=false t/src/lib.rs -rustdoc -Z unstable-options --crate-name=s --crate-type=lib --edition=2021 --out-dir=s/target/doc --extern-html-root-url s=merged/doc --extern-html-root-url t=merged/doc --merge-parts=false --extern t=t/target/libt.rmeta s/src/lib.rs +rustdoc -Z unstable-options --crate-name=t --crate-type=lib --edition=2021 --out-dir=t/target/doc --extern-html-root-url t=$(MERGED) --write-merged-cci=false --read-merged-cci=false --parts-out-dir=t/target/doc.parts t/src/lib.rs +rustdoc -Z unstable-options --crate-name=s --crate-type=lib --edition=2021 --out-dir=s/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --write-merged-cci=false --read-merged-cci=false --parts-out-dir=t/target/doc.parts --extern t=t/target/libt.rmeta s/src/lib.rs ``` -Link everything with a final invocation of rustdoc on `i`. We will **not** provide `--merge-parts=false`, because we are merging the parts. We must also provide `--extern-parts-path`. See the Reference-level explanation about this flag. +Link everything with a final invocation of Rustdoc on `i`. We will **not** provide `--write-merged-cci=false`, because we are merging the parts. We will also provide `--read-merged-cci=false` and `--fetch-parts=`. See the Reference-level explanation about these flags. ```shell -rustdoc -Z unstable-options --crate-name=i --crate-type=lib --edition=2021 --enable-index-page --out-dir=i/target/doc/ --extern-html-root-url s=merged/doc/ --extern-html-root-url t=merged/doc/ --extern-html-root-url i=merged/doc/ --extern-parts-path t=t/target/doc/ --extern-parts-path s=s/target/doc/ --extern t=t/target/libt.rmeta --extern s=s/target/libs.rmeta -L t/target i/src/lib.rs +rustdoc -Z unstable-options --crate-name=i --crate-type=lib --edition=2021 --enable-index-page --out-dir=i/target/doc/ --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --extern-html-root-url i=$(MERGED) --read-merged-cci=false --fetch-parts t=t/target/doc.parts --fetch-parts s=s/target/doc.parts --extern t=t/target/libt.rmeta --extern s=s/target/libs.rmeta -L t/target i/src/lib.rs ``` -Merge the docs with `cp`. This can be avoided if `--out-dir=merged/doc` is used for all of the rustdoc calls. We copy here to illustrate that documenting `s` is independent of documenting `t`, and could happen on separate machines. +Merge the docs with `cp`. This can be avoided if `--out-dir=$(MERGED)` is used for all of the Rustdoc calls. We copy here to illustrate that documenting `s` is independent of documenting `t`, and could happen on separate machines. ```shell -mkdir -p merged/doc cp -r s/target/doc/* t/target/doc/* i/target/doc/* merged/doc ``` @@ -80,184 +80,11 @@ Browse `merged/doc/index.html` with cross-crate information. In general, instead of two crates in the environment (`s` and `t`) you could have thousands. Upon any changes, only the index and the crates that are changed have to be re-documented. -
Click here for a directory listing after running the example above.
-$ tree . -a
-.
-├── i
-│   ├── src
-│   │   └── lib.rs
-│   └── target
-│       ├── doc
-│       │   ├── crates.js
-│       │   ├── help.html
-│       │   ├── i
-│       │   │   ├── all.html
-│       │   │   ├── fn.add.html
-│       │   │   ├── index.html
-│       │   │   └── sidebar-items.js
-│       │   ├── index.html
-│       │   ├── .lock
-│       │   ├── .parts
-│       │   │   └── i
-│       │   │       ├── crates-js
-│       │   │       ├── index-html
-│       │   │       ├── search-index-js
-│       │   │       ├── src-files-js
-│       │   │       ├── trait-impl
-│       │   │       └── type-impl
-│       │   ├── search.desc
-│       │   │   └── i
-│       │   │       └── i-desc-0-.js
-│       │   ├── settings.html
-│       │   ├── src
-│       │   │   └── i
-│       │   │       └── lib.rs.html
-│       │   ├── src-files.js
-│       │   └── static.files
-│       │       ├── COPYRIGHT-23e9bde6c69aea69.txt
-│       │       ├── favicon-2c020d218678b618.svg
-│       │       ├── favicon-32x32-422f7d1d52889060.png
-│       │       ├── FiraSans-LICENSE-db4b642586e02d97.txt
-│       │       ├── FiraSans-Medium-8f9a781e4970d388.woff2
-│       │       ├── FiraSans-Regular-018c141bf0843ffd.woff2
-│       │       ├── LICENSE-APACHE-b91fa81cba47b86a.txt
-│       │       ├── LICENSE-MIT-65090b722b3f6c56.txt
-│       │       ├── main-20a3ad099b048cf2.js
-│       │       ├── NanumBarunGothic-0f09457c7a19b7c6.ttf.woff2
-│       │       ├── NanumBarunGothic-LICENSE-18c5adf4b52b4041.txt
-│       │       ├── normalize-76eba96aa4d2e634.css
-│       │       ├── noscript-df360f571f6edeae.css
-│       │       ├── rustdoc-dd39b87e5fcfba68.css
-│       │       ├── rust-logo-151179464ae7ed46.svg
-│       │       ├── scrape-examples-ef1e698c1d417c0c.js
-│       │       ├── search-0fe7219eb170c82e.js
-│       │       ├── settings-4313503d2e1961c2.js
-│       │       ├── SourceCodePro-It-1cc31594bf4f1f79.ttf.woff2
-│       │       ├── SourceCodePro-LICENSE-d180d465a756484a.txt
-│       │       ├── SourceCodePro-Regular-562dcc5011b6de7d.ttf.woff2
-│       │       ├── SourceCodePro-Semibold-d899c5a5c4aeb14a.ttf.woff2
-│       │       ├── SourceSerif4-Bold-a2c9cd1067f8b328.ttf.woff2
-│       │       ├── SourceSerif4-It-acdfaf1a8af734b1.ttf.woff2
-│       │       ├── SourceSerif4-LICENSE-3bb119e13b1258b7.md
-│       │       ├── SourceSerif4-Regular-46f98efaafac5295.ttf.woff2
-│       │       ├── src-script-e66d777a5a92e9b2.js
-│       │       └── storage-118b08c4c78b968e.js
-│       └── libi.rmeta
-├── merged
-│   └── doc
-│       ├── crates.js
-│       ├── help.html
-│       ├── i
-│       │   ├── all.html
-│       │   ├── fn.add.html
-│       │   ├── index.html
-│       │   └── sidebar-items.js
-│       ├── index.html
-│       ├── s
-│       │   ├── all.html
-│       │   ├── index.html
-│       │   ├── sidebar-items.js
-│       │   └── struct.S.html
-│       ├── search.desc
-│       │   └── i
-│       │       └── i-desc-0-.js
-│       ├── settings.html
-│       ├── src
-│       │   ├── i
-│       │   │   └── lib.rs.html
-│       │   ├── s
-│       │   │   └── lib.rs.html
-│       │   └── t
-│       │       └── lib.rs.html
-│       ├── src-files.js
-│       ├── static.files
-│       │   ├── COPYRIGHT-23e9bde6c69aea69.txt
-│       │   ├── favicon-2c020d218678b618.svg
-│       │   ├── favicon-32x32-422f7d1d52889060.png
-│       │   ├── FiraSans-LICENSE-db4b642586e02d97.txt
-│       │   ├── FiraSans-Medium-8f9a781e4970d388.woff2
-│       │   ├── FiraSans-Regular-018c141bf0843ffd.woff2
-│       │   ├── LICENSE-APACHE-b91fa81cba47b86a.txt
-│       │   ├── LICENSE-MIT-65090b722b3f6c56.txt
-│       │   ├── main-20a3ad099b048cf2.js
-│       │   ├── NanumBarunGothic-0f09457c7a19b7c6.ttf.woff2
-│       │   ├── NanumBarunGothic-LICENSE-18c5adf4b52b4041.txt
-│       │   ├── normalize-76eba96aa4d2e634.css
-│       │   ├── noscript-df360f571f6edeae.css
-│       │   ├── rustdoc-dd39b87e5fcfba68.css
-│       │   ├── rust-logo-151179464ae7ed46.svg
-│       │   ├── scrape-examples-ef1e698c1d417c0c.js
-│       │   ├── search-0fe7219eb170c82e.js
-│       │   ├── settings-4313503d2e1961c2.js
-│       │   ├── SourceCodePro-It-1cc31594bf4f1f79.ttf.woff2
-│       │   ├── SourceCodePro-LICENSE-d180d465a756484a.txt
-│       │   ├── SourceCodePro-Regular-562dcc5011b6de7d.ttf.woff2
-│       │   ├── SourceCodePro-Semibold-d899c5a5c4aeb14a.ttf.woff2
-│       │   ├── SourceSerif4-Bold-a2c9cd1067f8b328.ttf.woff2
-│       │   ├── SourceSerif4-It-acdfaf1a8af734b1.ttf.woff2
-│       │   ├── SourceSerif4-LICENSE-3bb119e13b1258b7.md
-│       │   ├── SourceSerif4-Regular-46f98efaafac5295.ttf.woff2
-│       │   ├── src-script-e66d777a5a92e9b2.js
-│       │   └── storage-118b08c4c78b968e.js
-│       └── t
-│           ├── all.html
-│           ├── index.html
-│           ├── sidebar-items.js
-│           └── trait.T.html
-├── s
-│   ├── src
-│   │   └── lib.rs
-│   └── target
-│       ├── doc
-│       │   ├── help.html
-│       │   ├── .lock
-│       │   ├── .parts
-│       │   │   └── s
-│       │   │       ├── crates-js
-│       │   │       ├── index-html
-│       │   │       ├── search-index-js
-│       │   │       ├── src-files-js
-│       │   │       ├── trait-impl
-│       │   │       └── type-impl
-│       │   ├── s
-│       │   │   ├── all.html
-│       │   │   ├── index.html
-│       │   │   ├── sidebar-items.js
-│       │   │   └── struct.S.html
-│       │   ├── settings.html
-│       │   └── src
-│       │       └── s
-│       │           └── lib.rs.html
-│       └── libs.rmeta
-└── t
-    ├── src
-    │   └── lib.rs
-    └── target
-        ├── doc
-        │   ├── help.html
-        │   ├── .lock
-        │   ├── .parts
-        │   │   └── t
-        │   │       ├── crates-js
-        │   │       ├── index-html
-        │   │       ├── search-index-js
-        │   │       ├── src-files-js
-        │   │       ├── trait-impl
-        │   │       └── type-impl
-        │   ├── settings.html
-        │   ├── src
-        │   │   └── t
-        │   │       └── lib.rs.html
-        │   └── t
-        │       ├── all.html
-        │       ├── index.html
-        │       ├── sidebar-items.js
-        │       └── trait.T.html
-        └── libt.rmeta    
+TODO
 
@@ -265,99 +92,118 @@ $ tree . -a # Reference-level explanation -Currently, CCI is written during the invocation of the `write_shared` function in [write_shared.rs](https://github.com/rust-lang/rust/blob/04ab7b2be0db3e6787f5303285c6b2ee6279868d/src/librustdoc/html/render/write_shared.rs#L47). This proposal does not add any new CCI or change their contents (modulo sorting order, whitespace). +Currently, cross-crate information is written during the invocation of the `write_shared` function in [write_shared.rs](https://github.com/rust-lang/rust/blob/04ab7b2be0db3e6787f5303285c6b2ee6279868d/src/libRustdoc/html/render/write_shared.rs#L47). This proposal does not add any new CCI or change their contents (modulo sorting order, whitespace). -The existing cross-crate information files, like `search-index.js`, all happen to be lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc/`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of a the CCI. +The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current Rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc/`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of a the CCI. -## New subdirectory: `doc/.parts//` +## New subdirectory: `doc.parts//` -The `doc/.parts//` files contain the unmerged contents of a single crates' version of their corresponding CCI. +The `doc.parts//` files contain the unmerged contents of a single crates' version of their corresponding CCI. It is written if the flag `--parts-out-dir=` is provided. -Every file in `doc/.parts//*` is a JSON array. Every element of the +Every file in `doc.parts//*` is a JSON array. Every element of the array is a two-element array: a destination filename (relative to `doc/`), and the representation of the part. The representation of that part depends on the type of CCI that it describes. -* `doc/.parts//src-files-js`: for `doc/src-files.js` +* `doc.parts//src-files-js`: for `doc/src-files.js` This part is the JSON representation of the source index that is later stored in a `srcIndex` global variable. -* `doc/.parts//search-index-js`: for `doc/search-index.js` +* `doc.parts//search-index-js`: for `doc/search-index.js` This part is the JSON encoded search index, before it has been installed in `search-index.js`. -* `doc/.parts//search-desc`: for `doc/search.desc/**/*.js` +* `doc.parts//search-desc`: for `doc/search.desc/**/*.js` This part contains the JavaScript code to load a shard of the search descriptions. -* `doc/.parts//all-crates`: for `doc/crates.js`, `/doc/index.html` +* `doc.parts//all-crates`: for `doc/crates.js`, `/doc/index.html` This part is the crate name. -* `doc/.parts//crates-index`: for `doc/crates.js`, `doc/index.html` +* `doc.parts//crates-index`: for `doc/crates.js`, `doc/index.html` This part is the also crate name. It represents a different kind of CCI because it is written to a `doc/index.html`, and rendered as an HTML document instead as JSON. In principal, we could use this part to add more information to the crates index `doc/index.html` (the first line of the top level crate documentation, for example). -* `doc/.parts//type-impl`: for `doc/type.impl/**/*.js` +* `doc.parts//type-impl`: for `doc/type.impl/**/*.js` This part is a two element array with the crate name and the JSON representation of a type implementation. -* `doc/.parts//trait-impl`: for `doc/trait.impl/**/*.js` +* `doc.parts//trait-impl`: for `doc/trait.impl/**/*.js` This part is a two element array with the crate name and the JSON representation of a trait implementation. -## New flag: `--merge-parts[=true|false]` +## New flag: `--parts-out-dir=` + +When this flag is provided, the unmerged parts for the current crate will be written to `//`. A typical `` is be `target/doc.parts`. + +If this flag is not provided, no `doc.parts` will be written. + +This flag is the complement to `--fetch-parts`. + +## New flag: `--write-merged-cci[=true|false]` + +This flag defaults to true if not specified. + +With this flag is true, Rustdoc will write the rendered CCI to the output directory, on each invocation. The user-facing behavior without the flag is the same as the current behavior, with the addition of a new `doc.parts` directory. + +When this flag is false, the `doc.parts` the CCI will not be written nor rendered. Another call to a merge step will be required to merge the parts and write the CCI. + +## New flag: `--read-merged-cci=[=true|false]` This flag defaults to true if not specified. -With this flag set to false, rustdoc will generate `doc/.parts` and its contents, eagerly merge and render the parts, and write the rendered CCI to the output directory, on each invocation. The user-facing behavior without the flag is the same as the current behavior, with the addition of a new `doc/.parts` directory. +When this flag is true, Rustdoc will read the the rendered cross crate information from the doc root. Additional parts read from parts included via `--fetch-parts` will be appended to these parts. + +If this flag is false, `--fetch-parts` may still be used. The parts will initialized empty. + +## New flag: `--fetch-parts =` -When this flag is enabled, the `doc/.parts` directory and its contents will be generated, but the CCI will not be written. None of the CCI will be rendered. Another call to a merge step will be required to merge the parts and write the CCI. + -If rustdoc [identifies](https://github.com/rust-lang/rust/blob/dd104ef16315e2387fe94e8c43eb5a66e3dbd660/src/librustdoc/clean/types.rs#L184C7-L187C10) a crate as being documented locally, it will expect `doc/.parts/` to contain the parts. This flag is ignored in the case of locally documented crates. +If this flag is provided, it will expect `//` to contain the parts for ``. It will append these parts to the ones it will output. -If rustdoc identifies that a crate is externally documented but a user wishes to add it to the CCI, rustdoc must be able to locate the parts. In this case, pass `--extern-parts-location =`. rustdoc will then assume that the parts are in `/.parts//`. +This flag is the complement to `--parts-out-dir` -This flag is similar to `--extern-html-root-url` in that it only applies to externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided is never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--extern-parts-path` tells rustdoc where to search for the `.parts` directory at documentation-time. It must not be a hyperlink. +This flag is similar to `--extern-html-root-url` in that it only applies to externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided is never accessed by Rustdoc, and represents the final destination of the documentation. The new flag `--fetch-parts` tells Rustdoc where to search for the `doc.parts` directory at documentation-time. It must not be a hyperlink. -In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc/.parts/s`, so rustdoc is called with `--extern-parts-location s=s/target/doc/`. +In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc.parts/s`, so Rustdoc is called with `--fetch-parts s=s/target/doc.parts`. ## Merge step -This step is provided with a list of crates. It merges their documentation. This step involves copying parts (individual item, module documentation) from each of the provided crates. It merges the parts, renders, and writes the CCI to `doc/`. +This step is provided with a list of crates. It merges their documentation. This step involves copying parts (individual item, module documentation) from each of the provided crates. It merges the parts, renders, and writes the CCI to the documentation root. Discussion of the merge step is described in the Unresolved questions. # Drawbacks -The WIP might change the sorting order of the elements in the CCI. It does not change the content of the documentation, and is intended to work without modifying Cargo and docs.rs. - -It also proposes to add a `doc/.parts` directory, unconditionally. This is discussed in Unresolved questions. +The WIP may change the sorting order of the elements in the CCI. It does not change the content of the documentation, and is intended to work without modifying Cargo and docs.rs. # Rationale and alternatives -Running rustdoc in parallel is essential in enabling the tool to scale to large projects. Cargo implements parallel rustdoc by locking the CCI files. There are some environments where having synchronized access to the CCI is impossible. This proposal implements a reasonable approach to shared rustdoc, because it cleanly enables the addition of new kinds of CCI. +Running Rustdoc in parallel is essential in enabling the tool to scale to large projects. Cargo implements parallel Rustdoc by locking the CCI files. There are some environments where having synchronized access to the CCI is impossible. This proposal implements a reasonable approach to shared Rustdoc, because it cleanly enables the addition of new kinds of CCI without any changes to existing documentation. # Prior art Prior art for linking and merging independently generated documentation was **not** identified in Javadoc, Godoc, Doxygen, Sphinx (intersphinx), nor any documentation system for other languages. Analogs of cross-crate information were not found, but a more thorough investigation or experience with other systems may be needed. -However, the issues presented here have been encountered in multiple build systems that interact with rustdoc. They limit the usefulness of rustdoc in large environments. +However, the issues presented here have been encountered in multiple build systems that interact with Rustdoc. They limit the usefulness of Rustdoc in large environments. ## Bazel -Bazel has `rules_rust` for building Rust targets and rustdoc documentation. +Bazel has `rules_rust` for building Rust targets and Rustdoc documentation. * -* +* -It does not document crates' dependencies. `search-index.js`, for example, is both a dependency and an output file for rustdoc in multi-crate documentation environments. If it is declared as a dependency in this way, Bazel could not build docs for the members of an environment in parallel with a single output directory, as it strictly enforces hermiticity. For a recursive, parallel rustdoc to ever serve as a first-class citizen in Bazel, changes similar to the ones described in this proposal would be needed. +It does not document crates' dependencies. `search-index.js`, for example, is both a dependency and an output file for Rustdoc in multi-crate documentation environments. If it is declared as a dependency in this way, Bazel could not build docs for the members of an environment in parallel with a single output directory, as it strictly enforces hermiticity. For a recursive, parallel Rustdoc to ever serve as a first-class citizen in Bazel, changes similar to the ones described in this proposal would be needed. -There is an [open issue](https://github.com/bazelbuild/rules_rust/issues/1837) raised about the fact that Bazel does not document crates dependencies. The comments in the issue discuss a pull request on Bazel that documents each crates dependencies in a separate output directory. It is noted in the discussion that this solution, being implemented under the current rustdoc, "doesn't scale well and it should be implemented in a different manner long term." In order to get CCI in a mode like this, rustdoc would need to adopt changes, like the ones in this proposal, for merging cross-crate information. +There is an [open issue](https://github.com/bazelbuild/rules_rust/issues/1837) raised about the fact that Bazel does not document crates dependencies. The comments in the issue discuss a pull request on Bazel that documents each crates dependencies in a separate output directory. It is noted in the discussion that this solution, being implemented under the current Rustdoc, "doesn't scale well and it should be implemented in a different manner long term." In order to get CCI in a mode like this, Rustdoc would need to adopt changes, like the ones in this proposal, for merging cross-crate information. ## Buck2 @@ -377,7 +223,7 @@ buck2 build //:library[doc] --verbose 5 ``` --> -It has a subtarget, `[doc]`, for generating rustdoc for a crate. +It has a subtarget, `[doc]`, for generating Rustdoc for a crate. You can provide a coarse-grained `extern-html-root-url` for all dependencies. You could document all crates independently, but no cross-crate information would be shared. @@ -388,86 +234,59 @@ It does not document crates' dependencies for the same reason that Bazel does no ## Ninja [(GN)](https://fuchsia.dev/fuchsia-src/development/build/build_system/intro) + Fuchsia -Currently, the Fuchsia project runs rustdoc on all of their crates to generate a [documentation index](https://fuchsia-docs.firebaseapp.com/rust/rustdoc_index/). This index is effectively generated as an [atomic step](https://cs.opensource.google/fuchsia/fuchsia/+/main:tools/devshell/contrib/lib/rust/rustdoc.py) in the build system. It takes [3 hours](https://ci.chromium.org/ui/p/fuchsia/builders/global.ci/firebase-docs/b8744777376580022225/overview) to document the ~2700 crates in the environment. With this proposal, building each crate's documentation could be done as separate build actions, which would have a number of benefits. These include parallelism, caching (avoid rebuilding docs unnecessarily), and robustness (automatically reject pull requests that break documentation). - - - - +Currently, the Fuchsia project runs Rustdoc on all of their crates to generate a [documentation index](https://fuchsia-docs.firebaseapp.com/rust/Rustdoc_index/). This index is effectively generated as an [atomic step](https://cs.opensource.google/fuchsia/fuchsia/+/main:tools/devshell/contrib/lib/rust/Rustdoc.py) in the build system. It takes [3 hours](https://ci.chromium.org/ui/p/fuchsia/builders/global.ci/firebase-docs/b8744777376580022225/overview) to document the ~2700 crates in the environment. With this proposal, building each crate's documentation could be done as separate build actions, which would have a number of benefits. These include parallelism, caching (avoid rebuilding docs unnecessarily), and robustness (automatically reject pull requests that break documentation). # Unresolved questions ## Index crate? -Require users to generate documentation bundles via an index crate (current) vs. creating a new mode to allow rustdoc to run without a target crate (proposed) +Require users to generate documentation bundles via an index crate (current) vs. creating a new mode to allow Rustdoc to run without a target crate (proposed) -If one would like to merge the documentation of several crates, we could continue to require users to provide an index crate, like [the fuchsia index](https://fuchsia-docs.firebaseapp.com/rust/rustdoc_index/). This serves as the target of the rustdoc invocation, and the landing page for the collected documentation. Supporting only this style of index would require the fewest changes. This is the mode described in the Guide-level explanation. +If one would like to merge the documentation of several crates, we could continue to require users to provide an index crate, like [the fuchsia index](https://fuchsia-docs.firebaseapp.com/rust/Rustdoc_index/). This serves as the target of the Rustdoc invocation, and the landing page for the collected documentation. Supporting only this style of index would require the fewest changes. This is the mode described in the Guide-level explanation. -The proposition, to allow users of rustdoc the flexibility of not having to produce an index, is to allow rustdoc to be run in a mode where no target crate is provided. The source crates are provided to rustdoc, through a mechanism like `--extern`, rustdoc merges and writes the CCI, and copies the item and module links to `doc/`. This would require more extensive changes, as rustdoc assumes that it is invoked with a target crate. This mode is somewhat analogous to the [example scraping mode](https://github.com/rust-lang/rfcs/blob/master/text/3123-rustdoc-scrape-examples.md). Having to create an index crate, that actively uses all of the crates in the environment, might prohibit the use of this feature in settings where users do not intend to produce an index, or where exhaustively listing all dependencies (to `--extern` them) is difficult. +The proposition, to allow users of Rustdoc the flexibility of not having to produce an index, is to allow Rustdoc to be run in a mode where no target crate is provided. The source crates are provided to Rustdoc, through a mechanism like `--extern`, Rustdoc merges and writes the CCI, and copies the item and module links to `doc/`. This would require more extensive changes, as Rustdoc assumes that it is invoked with a target crate. This mode is somewhat analogous to the [example scraping mode](https://github.com/rust-lang/rfcs/blob/master/text/3123-Rustdoc-scrape-examples.md). Having to create an index crate, that actively uses all of the crates in the environment, might prohibit the use of this feature in settings where users do not intend to produce an index, or where exhaustively listing all dependencies (to `--extern` them) is difficult. -## Unconditionally generating the `doc/.parts/*/*` files? +## Unconditionally generating the `doc.parts` files? -Generate no extra files (current) vs. unconditionally creating `doc/.parts` to enable more complex future CCI (proposed) +Generate no extra files (current) vs. unconditionally creating `doc.parts` to enable more complex future CCI (should consider) -The current version of rustdoc performs merging by [collecting JSON](https://github.com/rust-lang/rust/blob/c25ac9d6cc285e57e1176dc2da6848b9d0163810/src/librustdoc/html/render/write_shared.rs#L166) blobs from the contents of the already-rendered CCI. This proposal performs merging by reading contents of the unmerged `doc/.parts/` files. This is simpler even if you don't care about deferring CCI generation, as it uses the existing `serde_json` dependency for extracting the blobs, as opposed to handwritten CCI-type specific parsing (current). - -The **alternative** to writing the parts to the `doc/.parts/` directory is to continue extracting them from the rendered versions of the CCI. There are several issues with this current practice -* If a user has a single `doc/` output directory, it is impossible to avoid shared mutation if every rustdoc process is writing to the same CCI -* It is difficult to extract the items in a diverse set of rendered HTML files. This is articipating of the CCI to include HTML files that, for example, statically include type+trait implementations directly -* Parts would need to be de-duplicated if a crate is a dependency of multiple crates, given that we now support merging. - -It is the authors' preference to merge by reading the `doc/.parts` files, even with `--merge-parts=false`. There would be no need for duplicate logic to read from both `doc/.parts` files and rendered CCI, because it would always use `doc/.parts` as the source for the merge. +The current version of Rustdoc performs merging by [collecting JSON](https://github.com/rust-lang/rust/blob/c25ac9d6cc285e57e1176dc2da6848b9d0163810/src/libRustdoc/html/render/write_shared.rs#L166) blobs from the contents of the already-rendered CCI. +This proposal proposes to continue reading from the rendered cross-crate information under the default `--read-merged-cci=true`. It can also read `doc.parts` files, under `--fetch-parts`. However, there are several issues with reading from the rendered CCI that must be stated: +* If a user has a single `doc/` output directory, it is impossible to avoid shared mutation if every Rustdoc process is writing to the same CCI +* It is difficult to extract the items in a diverse set of rendered HTML files. This is anticipating of the CCI to include HTML files that, for example, statically include type+trait implementations directly +* Reading exclusively from `doc.parts` is simpler than the existing `serde_json` dependency for extracting the blobs, as opposed to handwritten CCI-type specific parsing (current) +* With this proposal, there will be duplicate logic to read from both `doc.parts` files and rendered CCI. ## Item links? -Require users to pass `--extern-html-root-url` on all invocations of rustdoc to generate individually documented crates (current), vs. add a new flag to facilitate missing docs links being generated across all user crates (proposed) +Require users to pass `--extern-html-root-url` on all external dependencies (current), vs. add a new flag to facilitate missing docs links being generated across all user crates (should consider) -Cross-crate links are an important consideration when merging documentation. rustdoc provides a flag, `--extern-html-root-url`, which can provide fine-grain control over the generated URL for documentation. We believe this is already sufficient for generating cross-crate links, and we will make no changes to the generation of item links. Example usage of this flag is in the Guide-level explanation. +Cross-crate links are an important consideration when merging documentation. Rustdoc provides a flag, `--extern-html-root-url`, which can provide fine-grain control over the generated URL for documentation. We believe this is already sufficient for generating cross-crate links, and we will make no changes to the generation of item links. Example usage of this flag is in the Guide-level explanation. -A proposal is to provide more options to facilitate cross-crate links. For example, we could add a flag that implies `--extern-html-root-url` and `--extern-parts-path` on all crates passed through `--extern`. It could be something like `--extern-html-root-url-all-user-crates`. User crates would be taken to mean the transitive dependencies of the current crate, excluding the standard library. This is complicated by things like . +A proposal is to provide more options to facilitate cross-crate links. For example, we could add a flag that implies `--extern-html-root-url` and `--fetch-parts` on all crates passed through `--extern`. It could be something like `--extern-html-root-url-all-user-crates`. User crates would be taken to mean the transitive dependencies of the current crate, excluding the standard library. This is complicated by things like . ## Reuse existing option? -Create a new flag, `--no-merge-parts` (proposed), vs. use existing option `no_emit_shared` (current) +Create a new flag, `--write-merged-cci` (proposed), vs. use existing option `no_emit_shared` There is a render option, `no_emit_shared`, which is used to conditionally generate the cross-crate information. It also controls the generation of static files, user CSS, etc. -This option is not configurable from the command line, and appears to be enabled unless rustdoc is run in its example scraping mode. +This option is not configurable from the command line, and appears to be enabled unless Rustdoc is run in its example scraping mode. -We could make it configurable from the command line, unconditionally generate `.parts/`, and use it to gate the merging of CCI. +We could make it configurable from the command line, unconditionally generate `doc.parts/`, and use it to gate the merging of CCI. -We could also make it configurable from the command line, and use it to gate the generation of `.parts/` and the generation of all of the shared files. +We could also make it configurable from the command line, and use it to gate the generation of `doc.parts/` and the generation of all of the shared files. -We could also leave it as-is: always false unless we're scraping examples, and gate the generation of `.parts/` and the generation of all of the shared files. +We could also leave it as-is: always false unless we're scraping examples, and gate the generation of `doc.parts/` and the generation of all of the shared files. -## Location of the `.parts` directory? - -Generate `.parts` as a child of the documentation root (proposed) vs. choose another, possibly user selected, location (other proposal) - -The documentation root houses the files in the generated site, while `.parts` holds intermediate output. Some may believe that `.parts` would bloat the generated site. - -There is no other obvious location for `.parts`. We might like for `.parts` to be a child of `target/`, not a child of `target/doc/`. However, rustdoc is never made aware of the `target/` directory. Instead, `--out-dir` always refers to the documentation root. - -One idea is to add an optional command-line argument to configure its location: `--parts-out-dir`, defaulting to `/.parts`. Users may configure it to be `target/.parts` if they have a target directory. - # Future possibilities This change could begin to facilitate type aliases and the trait aliases being @@ -475,8 +294,8 @@ statically compiled as part of the .html documentation, instead of being loaded as separate JavaScript files. Each type and trait alias could be stored as an HTML part, which are then merged into the regular documentation. -Another possibility is for `.parts/` to be distributed on `docs.rs` along with the regular documentation. This would facilitate a mode where documentation of the dependencies could be downloaded externally, instead of being rebuilt locally. +Another possibility is for `doc.parts` to be distributed on `docs.rs` along with the regular documentation. This would facilitate a mode where documentation of the dependencies could be downloaded externally, instead of being rebuilt locally. A future possibility related to the index crate idea is to have an option for embedding user-specified HTML into the `--enable-index-page`'s HTML. -The changes in this proposal are intended to work with no changes to Cargo and docs.rs. However, there may be benefits to using `--merge-parts=false` with Cargo, as it would remove the need for locking the output directory. More of the documentation process could happen in parallel, which may speed up execution time.. +The changes in this proposal are intended to work with no changes to Cargo and docs.rs. However, there may be benefits to using `--write-merged-cci=false` with Cargo, as it would remove the need for locking the output directory. More of the documentation process could happen in parallel, which may speed up execution time.. From 8c78a93f0e3074cf4d0c64376174da0193731539 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Thu, 20 Jun 2024 17:29:01 +0000 Subject: [PATCH 03/28] fix typos --- ...0000-mergeable-rustdoc-cross-crate-info.md | 308 ++++++++++++++---- 1 file changed, 241 insertions(+), 67 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index 47307ab9aa0..bd6536e80ee 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -1,4 +1,4 @@ -- Feature Name: `mergable_Rustdoc_cross_crate_info` +- Feature Name: `mergable_rustdoc_cross_crate_info` - Start Date: 2024-06-18 - RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) - Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) @@ -11,27 +11,22 @@ # Summary -Mergeable cross-crate information in Rustdoc. Facilitates the generation of documentation indexes in environments with many crates by allowing each crate to write to an independent output directory. Final documentation is generated with a lightweight merge step. Configurable with command-line flags, this proposal writes a `doc.parts` directory to hold pre-merge cross-crate information. Currently, Rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating Rustdoc in build systems that make build actions independent. +Mergeable cross-crate information in rustdoc. Facilitates the generation of documentation indexes in environments with many crates by allowing each crate to write to an independent output directory. Final documentation is rendered with a lightweight merge step. Configurable with command-line flags, this proposal writes a `doc.parts` directory to hold pre-merge cross-crate information. Currently, rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating rustdoc in build systems that make build actions independent. # Motivation The main goal of this proposal is to facilitate users producing a documentation bundle of every crate in a large environment. When a crate needs to be re-documented, only a relatively lightweight merge step will be needed to produce a complete documentation bundle. This proposal is to facilitate the creation and updating of these bundles. -There are some files in the Rustdoc output directory that are read and overwritten during every invocation of Rustdoc. This proposal refers to these files as **cross-crate information**, or **CCI**, as in . +There are some files in the rustdoc output directory that are read and overwritten during every invocation of rustdoc. This proposal refers to these files as **cross-crate information**, or **CCI**, as in . -Build systems may run build actions in a distributed environment across separate logical filesystems. It might also be desirable to run Rustdoc in a lock-free parallel mode, where every Rustdoc process writes to a disjoint set of files. +Build systems may run build actions in a distributed environment across separate logical filesystems. It might also be desirable to run rustdoc in a lock-free parallel mode, where every rustdoc process writes to a disjoint set of files. Cargo fully supports cross-crate information, at the cost of requiring read-write access to the documentation root (`target/doc/`). There are significant scalability issues with this approach. -Rustdoc needing global mutable access to the files that encode this cross-crate information has implications for caching, reproducible builds, and content hashing. By adding an option to avoid this mutation, Rustdoc will serve as a first-class citizen in non-cargo build systems. +Rustdoc needing global mutable access to the files that encode this cross-crate information has implications for caching, reproducible builds, and content hashing. By adding an option to avoid this mutation, rustdoc will serve as a first-class citizen in non-cargo build systems. These considerations motivate adding an option for outputting partial CCI (parts), which are merged (linked) with a later step. - - - # Guide-level explanation In this example, there is a crate `t` which defines a trait `T`, and a crate `s` which defines a struct `S` that implements `T`. Our goal in this demo is for `S` to appear as in implementer in `T`'s docs, even if `s` and `t` are documented independently. This guide will be assuming that we want a crate `i` that serves as our documentation index. See the Unresolved questions section for ideas that do not require an index crate. @@ -64,13 +59,13 @@ rustdoc -Z unstable-options --crate-name=t --crate-type=lib --edition=2021 --out rustdoc -Z unstable-options --crate-name=s --crate-type=lib --edition=2021 --out-dir=s/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --write-merged-cci=false --read-merged-cci=false --parts-out-dir=t/target/doc.parts --extern t=t/target/libt.rmeta s/src/lib.rs ``` -Link everything with a final invocation of Rustdoc on `i`. We will **not** provide `--write-merged-cci=false`, because we are merging the parts. We will also provide `--read-merged-cci=false` and `--fetch-parts=`. See the Reference-level explanation about these flags. +Link everything with a final invocation of rustdoc on `i`. We will **not** provide `--write-merged-cci=false`, because we are merging the parts. We will also provide `--read-merged-cci=false` and `--fetch-parts=`. See the Reference-level explanation about these flags. ```shell -rustdoc -Z unstable-options --crate-name=i --crate-type=lib --edition=2021 --enable-index-page --out-dir=i/target/doc/ --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --extern-html-root-url i=$(MERGED) --read-merged-cci=false --fetch-parts t=t/target/doc.parts --fetch-parts s=s/target/doc.parts --extern t=t/target/libt.rmeta --extern s=s/target/libs.rmeta -L t/target i/src/lib.rs +rustdoc -Z unstable-options --crate-name=i --crate-type=lib --edition=2021 --enable-index-page --out-dir=i/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --extern-html-root-url i=$(MERGED) --read-merged-cci=false --fetch-parts t=t/target/doc.parts --fetch-parts s=s/target/doc.parts --extern t=t/target/libt.rmeta --extern s=s/target/libs.rmeta -L t/target i/src/lib.rs ``` -Merge the docs with `cp`. This can be avoided if `--out-dir=$(MERGED)` is used for all of the Rustdoc calls. We copy here to illustrate that documenting `s` is independent of documenting `t`, and could happen on separate machines. +Merge the docs with `cp`. This can be avoided if `--out-dir=$(MERGED)` is used for all of the rustdoc calls. We copy here to illustrate that documenting `s` is independent of documenting `t`, and could happen on separate machines. ```shell cp -r s/target/doc/* t/target/doc/* i/target/doc/* merged/doc @@ -84,7 +79,205 @@ In general, instead of two crates in the environment (`s` and `t`) you could hav Click here for a directory listing after running the example above.
-TODO
+$ tree . -a
+.
+├── i
+│   ├── Cargo.lock
+│   ├── Cargo.toml
+│   ├── .gitignore
+│   ├── src
+│   │   └── lib.rs
+│   └── target
+│       ├── doc
+│       │   ├── crates.js
+│       │   ├── help.html
+│       │   ├── i
+│       │   │   ├── all.html
+│       │   │   ├── fn.add.html
+│       │   │   ├── index.html
+│       │   │   └── sidebar-items.js
+│       │   ├── index.html
+│       │   ├── .lock
+│       │   ├── .parts
+│       │   │   └── i
+│       │   │       ├── crates-js
+│       │   │       ├── index-html
+│       │   │       ├── search-index-js
+│       │   │       ├── src-files-js
+│       │   │       ├── trait-impl
+│       │   │       └── type-impl
+│       │   ├── search.desc
+│       │   │   ├── i
+│       │   │   │   └── i-desc-0-.js
+│       │   │   ├── s
+│       │   │   │   └── s-desc-0-.js
+│       │   │   └── t
+│       │   │       └── t-desc-0-.js
+│       │   ├── settings.html
+│       │   ├── src
+│       │   │   └── i
+│       │   │       └── lib.rs.html
+│       │   ├── src-files.js
+│       │   ├── static.files
+│       │   │   ├── COPYRIGHT-23e9bde6c69aea69.txt
+│       │   │   ├── favicon-2c020d218678b618.svg
+│       │   │   ├── favicon-32x32-422f7d1d52889060.png
+│       │   │   ├── FiraSans-LICENSE-db4b642586e02d97.txt
+│       │   │   ├── FiraSans-Medium-8f9a781e4970d388.woff2
+│       │   │   ├── FiraSans-Regular-018c141bf0843ffd.woff2
+│       │   │   ├── LICENSE-APACHE-b91fa81cba47b86a.txt
+│       │   │   ├── LICENSE-MIT-65090b722b3f6c56.txt
+│       │   │   ├── main-20a3ad099b048cf2.js
+│       │   │   ├── NanumBarunGothic-0f09457c7a19b7c6.ttf.woff2
+│       │   │   ├── NanumBarunGothic-LICENSE-18c5adf4b52b4041.txt
+│       │   │   ├── normalize-76eba96aa4d2e634.css
+│       │   │   ├── noscript-df360f571f6edeae.css
+│       │   │   ├── rustdoc-dd39b87e5fcfba68.css
+│       │   │   ├── rust-logo-151179464ae7ed46.svg
+│       │   │   ├── scrape-examples-ef1e698c1d417c0c.js
+│       │   │   ├── search-0fe7219eb170c82e.js
+│       │   │   ├── settings-4313503d2e1961c2.js
+│       │   │   ├── SourceCodePro-It-1cc31594bf4f1f79.ttf.woff2
+│       │   │   ├── SourceCodePro-LICENSE-d180d465a756484a.txt
+│       │   │   ├── SourceCodePro-Regular-562dcc5011b6de7d.ttf.woff2
+│       │   │   ├── SourceCodePro-Semibold-d899c5a5c4aeb14a.ttf.woff2
+│       │   │   ├── SourceSerif4-Bold-a2c9cd1067f8b328.ttf.woff2
+│       │   │   ├── SourceSerif4-It-acdfaf1a8af734b1.ttf.woff2
+│       │   │   ├── SourceSerif4-LICENSE-3bb119e13b1258b7.md
+│       │   │   ├── SourceSerif4-Regular-46f98efaafac5295.ttf.woff2
+│       │   │   ├── src-script-e66d777a5a92e9b2.js
+│       │   │   └── storage-118b08c4c78b968e.js
+│       │   └── trait.impl
+│       │       ├── core
+│       │       │   ├── marker
+│       │       │   │   ├── trait.Freeze.js
+│       │       │   │   ├── trait.Send.js
+│       │       │   │   ├── trait.Sync.js
+│       │       │   │   └── trait.Unpin.js
+│       │       │   └── panic
+│       │       │       └── unwind_safe
+│       │       │           ├── trait.RefUnwindSafe.js
+│       │       │           └── trait.UnwindSafe.js
+│       │       └── t
+│       │           └── trait.T.js
+│       └── libi.rmeta
+├── merged
+│   └── doc
+│       ├── crates.js
+│       ├── help.html
+│       ├── i
+│       │   ├── all.html
+│       │   ├── fn.add.html
+│       │   ├── index.html
+│       │   └── sidebar-items.js
+│       ├── index.html
+│       ├── s
+│       │   ├── all.html
+│       │   ├── index.html
+│       │   ├── sidebar-items.js
+│       │   └── struct.S.html
+│       ├── search.desc
+│       │   └── i
+│       │       └── i-desc-0-.js
+│       ├── settings.html
+│       ├── src
+│       │   ├── i
+│       │   │   └── lib.rs.html
+│       │   ├── s
+│       │   │   └── lib.rs.html
+│       │   └── t
+│       │       └── lib.rs.html
+│       ├── src-files.js
+│       ├── static.files
+│       │   ├── COPYRIGHT-23e9bde6c69aea69.txt
+│       │   ├── favicon-2c020d218678b618.svg
+│       │   ├── favicon-32x32-422f7d1d52889060.png
+│       │   ├── FiraSans-LICENSE-db4b642586e02d97.txt
+│       │   ├── FiraSans-Medium-8f9a781e4970d388.woff2
+│       │   ├── FiraSans-Regular-018c141bf0843ffd.woff2
+│       │   ├── LICENSE-APACHE-b91fa81cba47b86a.txt
+│       │   ├── LICENSE-MIT-65090b722b3f6c56.txt
+│       │   ├── main-20a3ad099b048cf2.js
+│       │   ├── NanumBarunGothic-0f09457c7a19b7c6.ttf.woff2
+│       │   ├── NanumBarunGothic-LICENSE-18c5adf4b52b4041.txt
+│       │   ├── normalize-76eba96aa4d2e634.css
+│       │   ├── noscript-df360f571f6edeae.css
+│       │   ├── rustdoc-dd39b87e5fcfba68.css
+│       │   ├── rust-logo-151179464ae7ed46.svg
+│       │   ├── scrape-examples-ef1e698c1d417c0c.js
+│       │   ├── search-0fe7219eb170c82e.js
+│       │   ├── settings-4313503d2e1961c2.js
+│       │   ├── SourceCodePro-It-1cc31594bf4f1f79.ttf.woff2
+│       │   ├── SourceCodePro-LICENSE-d180d465a756484a.txt
+│       │   ├── SourceCodePro-Regular-562dcc5011b6de7d.ttf.woff2
+│       │   ├── SourceCodePro-Semibold-d899c5a5c4aeb14a.ttf.woff2
+│       │   ├── SourceSerif4-Bold-a2c9cd1067f8b328.ttf.woff2
+│       │   ├── SourceSerif4-It-acdfaf1a8af734b1.ttf.woff2
+│       │   ├── SourceSerif4-LICENSE-3bb119e13b1258b7.md
+│       │   ├── SourceSerif4-Regular-46f98efaafac5295.ttf.woff2
+│       │   ├── src-script-e66d777a5a92e9b2.js
+│       │   └── storage-118b08c4c78b968e.js
+│       └── t
+│           ├── all.html
+│           ├── index.html
+│           ├── sidebar-items.js
+│           └── trait.T.html
+├── s
+│   ├── Cargo.lock
+│   ├── Cargo.toml
+│   ├── .gitignore
+│   ├── src
+│   │   └── lib.rs
+│   └── target
+│       ├── doc
+│       │   ├── help.html
+│       │   ├── .lock
+│       │   ├── s
+│       │   │   ├── all.html
+│       │   │   ├── index.html
+│       │   │   ├── sidebar-items.js
+│       │   │   └── struct.S.html
+│       │   ├── settings.html
+│       │   └── src
+│       │       └── s
+│       │           └── lib.rs.html
+│       ├── doc.parts
+│       │   └── s
+│       │       ├── crates-js
+│       │       ├── index-html
+│       │       ├── search-index-js
+│       │       ├── src-files-js
+│       │       ├── trait-impl
+│       │       └── type-impl
+│       └── libs.rmeta
+└── t
+    ├── Cargo.lock
+    ├── Cargo.toml
+    ├── .gitignore
+    ├── src
+    │   └── lib.rs
+    └── target
+        ├── doc
+        │   ├── help.html
+        │   ├── .lock
+        │   ├── t
+        │   │   ├── all.html
+        │   │   ├── index.html
+        │   │   ├── sidebar-items.js
+        │   │   └── trait.T.html
+        │   ├── settings.html
+        │   └── src
+        │       └── t
+        │           └── lib.rs.html
+        ├── doc.parts
+        │   └── t
+        │       ├── crates-js
+        │       ├── index-html
+        │       ├── search-index-js
+        │       ├── src-files-js
+        │       ├── trait-impl
+        │       └── type-impl
+        └── libs.rmeta
 
@@ -92,50 +285,50 @@ TODO # Reference-level explanation -Currently, cross-crate information is written during the invocation of the `write_shared` function in [write_shared.rs](https://github.com/rust-lang/rust/blob/04ab7b2be0db3e6787f5303285c6b2ee6279868d/src/libRustdoc/html/render/write_shared.rs#L47). This proposal does not add any new CCI or change their contents (modulo sorting order, whitespace). +Currently, cross-crate information is written during the invocation of the `write_shared` function in [write_shared.rs](https://github.com/rust-lang/rust/blob/04ab7b2be0db3e6787f5303285c6b2ee6279868d/src/librustdoc/html/render/write_shared.rs#L47). This proposal does not add any new CCI or change their contents (modulo sorting order, whitespace). -The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current Rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc/`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of a the CCI. +The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc/`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of a the CCI. -## New subdirectory: `doc.parts//` +## New subdirectory: `//` -The `doc.parts//` files contain the unmerged contents of a single crates' version of their corresponding CCI. It is written if the flag `--parts-out-dir=` is provided. +Typically, `` is selected as `./target/doc.parts`. The `//` files contain the unmerged contents of a single crates' version of their corresponding CCI. It is written if the flag `--parts-out-dir=` is provided. -Every file in `doc.parts//*` is a JSON array. Every element of the +Every file in `//*` is a JSON array. Every element of the array is a two-element array: a destination filename (relative to `doc/`), and the representation of the part. The representation of that part depends on the type of CCI that it describes. -* `doc.parts//src-files-js`: for `doc/src-files.js` +* `//src-files-js`: for `doc/src-files.js` This part is the JSON representation of the source index that is later stored in a `srcIndex` global variable. -* `doc.parts//search-index-js`: for `doc/search-index.js` +* `//search-index-js`: for `doc/search-index.js` This part is the JSON encoded search index, before it has been installed in `search-index.js`. -* `doc.parts//search-desc`: for `doc/search.desc/**/*.js` +* `//search-desc`: for `doc/search.desc/**/*.js` This part contains the JavaScript code to load a shard of the search descriptions. -* `doc.parts//all-crates`: for `doc/crates.js`, `/doc/index.html` +* `//all-crates`: for `doc/crates.js`, `/doc/index.html` This part is the crate name. -* `doc.parts//crates-index`: for `doc/crates.js`, `doc/index.html` +* `//crates-index`: for `doc/crates.js`, `doc/index.html` This part is the also crate name. It represents a different kind of CCI because it is written to a `doc/index.html`, and rendered as an HTML document instead as JSON. In principal, we could use this part to add more information to the crates index `doc/index.html` (the first line of the top level crate documentation, for example). -* `doc.parts//type-impl`: for `doc/type.impl/**/*.js` +* `//type-impl`: for `doc/type.impl/**/*.js` This part is a two element array with the crate name and the JSON representation of a type implementation. -* `doc.parts//trait-impl`: for `doc/trait.impl/**/*.js` +* `//trait-impl`: for `doc/trait.impl/**/*.js` This part is a two element array with the crate name and the JSON representation of a trait implementation. ## New flag: `--parts-out-dir=` -When this flag is provided, the unmerged parts for the current crate will be written to `//`. A typical `` is be `target/doc.parts`. +When this flag is provided, the unmerged parts for the current crate will be written to `//`. A typical `` is `./target/doc.parts`. If this flag is not provided, no `doc.parts` will be written. @@ -145,34 +338,23 @@ This flag is the complement to `--fetch-parts`. This flag defaults to true if not specified. -With this flag is true, Rustdoc will write the rendered CCI to the output directory, on each invocation. The user-facing behavior without the flag is the same as the current behavior, with the addition of a new `doc.parts` directory. - -When this flag is false, the `doc.parts` the CCI will not be written nor rendered. Another call to a merge step will be required to merge the parts and write the CCI. +With this flag is true, rustdoc will write the rendered CCI to the output directory, on each invocation. The user-facing behavior without the flag is the same as the current behavior. When this flag is false, the CCI will not be written nor rendered. Another call to a merge step will be required to merge the parts and write the CCI. ## New flag: `--read-merged-cci=[=true|false]` This flag defaults to true if not specified. -When this flag is true, Rustdoc will read the the rendered cross crate information from the doc root. Additional parts read from parts included via `--fetch-parts` will be appended to these parts. - -If this flag is false, `--fetch-parts` may still be used. The parts will initialized empty. +When this flag is true, rustdoc will read the rendered cross crate information from the doc root. Additional parts included via `--fetch-parts` will be appended to these parts. When this flag is false, the parts will initialized empty. ## New flag: `--fetch-parts =` - - -If this flag is provided, it will expect `//` to contain the parts for ``. It will append these parts to the ones it will output. +This flag is the complement to `--parts-out-dir`. -This flag is the complement to `--parts-out-dir` +This flag is similar to `--extern-html-root-url` in that it only applies to externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided is never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--fetch-parts` tells rustdoc where to search for the `doc.parts` directory at documentation-time. It must not be a hyperlink. -This flag is similar to `--extern-html-root-url` in that it only applies to externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided is never accessed by Rustdoc, and represents the final destination of the documentation. The new flag `--fetch-parts` tells Rustdoc where to search for the `doc.parts` directory at documentation-time. It must not be a hyperlink. - -In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc.parts/s`, so Rustdoc is called with `--fetch-parts s=s/target/doc.parts`. +In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc.parts/s`, so rustdoc is called with `--fetch-parts s=s/target/doc.parts`. ## Merge step @@ -186,24 +368,24 @@ The WIP may change the sorting order of the elements in the CCI. It does not cha # Rationale and alternatives -Running Rustdoc in parallel is essential in enabling the tool to scale to large projects. Cargo implements parallel Rustdoc by locking the CCI files. There are some environments where having synchronized access to the CCI is impossible. This proposal implements a reasonable approach to shared Rustdoc, because it cleanly enables the addition of new kinds of CCI without any changes to existing documentation. +Running rustdoc in parallel is essential in enabling the tool to scale to large projects. The alternative, implemented by Cargo, is to run rustdoc in parallel by locking the CCI files. There are some environments where having synchronized access to the CCI is impossible. This proposal implements a reasonable approach to shared rustdoc, because it cleanly enables the addition of new kinds of CCI without changing existing documentation. # Prior art Prior art for linking and merging independently generated documentation was **not** identified in Javadoc, Godoc, Doxygen, Sphinx (intersphinx), nor any documentation system for other languages. Analogs of cross-crate information were not found, but a more thorough investigation or experience with other systems may be needed. -However, the issues presented here have been encountered in multiple build systems that interact with Rustdoc. They limit the usefulness of Rustdoc in large environments. +However, the issues presented here have been encountered in multiple build systems that interact with rustdoc. They limit the usefulness of rustdoc in large environments. ## Bazel -Bazel has `rules_rust` for building Rust targets and Rustdoc documentation. +Bazel has `rules_rust` for building Rust targets and rustdoc documentation. * -* +* -It does not document crates' dependencies. `search-index.js`, for example, is both a dependency and an output file for Rustdoc in multi-crate documentation environments. If it is declared as a dependency in this way, Bazel could not build docs for the members of an environment in parallel with a single output directory, as it strictly enforces hermiticity. For a recursive, parallel Rustdoc to ever serve as a first-class citizen in Bazel, changes similar to the ones described in this proposal would be needed. +It does not document crates' dependencies. `search-index.js`, for example, is both a dependency and an output file for rustdoc in multi-crate documentation environments. If it is declared as a dependency in this way, Bazel could not build docs for the members of an environment in parallel with a single output directory, as it strictly enforces hermiticity. For a recursive, parallel rustdoc to ever serve as a first-class citizen in Bazel, changes similar to the ones described in this proposal would be needed. -There is an [open issue](https://github.com/bazelbuild/rules_rust/issues/1837) raised about the fact that Bazel does not document crates dependencies. The comments in the issue discuss a pull request on Bazel that documents each crates dependencies in a separate output directory. It is noted in the discussion that this solution, being implemented under the current Rustdoc, "doesn't scale well and it should be implemented in a different manner long term." In order to get CCI in a mode like this, Rustdoc would need to adopt changes, like the ones in this proposal, for merging cross-crate information. +There is an [open issue](https://github.com/bazelbuild/rules_rust/issues/1837) raised about the fact that Bazel does not document crates dependencies. The comments in the issue discuss a pull request on Bazel that documents each crates dependencies in a separate output directory. It is noted in the discussion that this solution, being implemented under the current rustdoc, "doesn't scale well and it should be implemented in a different manner long term." In order to get CCI in a mode like this, rustdoc would need to adopt changes, like the ones in this proposal, for merging cross-crate information. ## Buck2 @@ -223,7 +405,7 @@ buck2 build //:library[doc] --verbose 5 ``` --> -It has a subtarget, `[doc]`, for generating Rustdoc for a crate. +It has a subtarget, `[doc]`, for generating rustdoc for a crate. You can provide a coarse-grained `extern-html-root-url` for all dependencies. You could document all crates independently, but no cross-crate information would be shared. @@ -234,25 +416,25 @@ It does not document crates' dependencies for the same reason that Bazel does no ## Ninja [(GN)](https://fuchsia.dev/fuchsia-src/development/build/build_system/intro) + Fuchsia -Currently, the Fuchsia project runs Rustdoc on all of their crates to generate a [documentation index](https://fuchsia-docs.firebaseapp.com/rust/Rustdoc_index/). This index is effectively generated as an [atomic step](https://cs.opensource.google/fuchsia/fuchsia/+/main:tools/devshell/contrib/lib/rust/Rustdoc.py) in the build system. It takes [3 hours](https://ci.chromium.org/ui/p/fuchsia/builders/global.ci/firebase-docs/b8744777376580022225/overview) to document the ~2700 crates in the environment. With this proposal, building each crate's documentation could be done as separate build actions, which would have a number of benefits. These include parallelism, caching (avoid rebuilding docs unnecessarily), and robustness (automatically reject pull requests that break documentation). +Currently, the Fuchsia project runs rustdoc on all of their crates to generate a [documentation index](https://fuchsia-docs.firebaseapp.com/rust/rustdoc_index/). This index is effectively generated as an [atomic step](https://cs.opensource.google/fuchsia/fuchsia/+/main:tools/devshell/contrib/lib/rust/rustdoc.py) in the build system. It takes [3 hours](https://ci.chromium.org/ui/p/fuchsia/builders/global.ci/firebase-docs/b8744777376580022225/overview) to document the ~2700 crates in the environment. With this proposal, building each crate's documentation could be done as separate build actions, which would have a number of benefits. These include parallelism, caching (avoid rebuilding docs unnecessarily), and robustness (automatically reject pull requests that break documentation). # Unresolved questions ## Index crate? -Require users to generate documentation bundles via an index crate (current) vs. creating a new mode to allow Rustdoc to run without a target crate (proposed) +Require users to generate documentation bundles via an index crate (current) vs. creating a new mode to allow rustdoc to run without a target crate (proposed) -If one would like to merge the documentation of several crates, we could continue to require users to provide an index crate, like [the fuchsia index](https://fuchsia-docs.firebaseapp.com/rust/Rustdoc_index/). This serves as the target of the Rustdoc invocation, and the landing page for the collected documentation. Supporting only this style of index would require the fewest changes. This is the mode described in the Guide-level explanation. +If one would like to merge the documentation of several crates, we could continue to require users to provide an index crate, like [the fuchsia index](https://fuchsia-docs.firebaseapp.com/rust/rustdoc_index/). This serves as the target of the rustdoc invocation, and the landing page for the collected documentation. Supporting only this style of index would require the fewest changes. This is the mode described in the Guide-level explanation. -The proposition, to allow users of Rustdoc the flexibility of not having to produce an index, is to allow Rustdoc to be run in a mode where no target crate is provided. The source crates are provided to Rustdoc, through a mechanism like `--extern`, Rustdoc merges and writes the CCI, and copies the item and module links to `doc/`. This would require more extensive changes, as Rustdoc assumes that it is invoked with a target crate. This mode is somewhat analogous to the [example scraping mode](https://github.com/rust-lang/rfcs/blob/master/text/3123-Rustdoc-scrape-examples.md). Having to create an index crate, that actively uses all of the crates in the environment, might prohibit the use of this feature in settings where users do not intend to produce an index, or where exhaustively listing all dependencies (to `--extern` them) is difficult. +The proposition, to allow users of rustdoc the flexibility of not having to produce an index, is to allow rustdoc to be run in a mode where no target crate is provided. The source crates are provided to rustdoc, through a mechanism like `--extern`, rustdoc merges and writes the CCI, and copies the item and module links to `doc/`. This would require more extensive changes, as rustdoc assumes that it is invoked with a target crate. This mode is somewhat analogous to the [example scraping mode](https://github.com/rust-lang/rfcs/blob/master/text/3123-rustdoc-scrape-examples.md). Having to create an index crate, that actively uses all of the crates in the environment, might prohibit the use of this feature in settings where users do not intend to produce an index, or where exhaustively listing all dependencies (to `--extern` them) is difficult. ## Unconditionally generating the `doc.parts` files? Generate no extra files (current) vs. unconditionally creating `doc.parts` to enable more complex future CCI (should consider) -The current version of Rustdoc performs merging by [collecting JSON](https://github.com/rust-lang/rust/blob/c25ac9d6cc285e57e1176dc2da6848b9d0163810/src/libRustdoc/html/render/write_shared.rs#L166) blobs from the contents of the already-rendered CCI. +The current version of rustdoc performs merging by [collecting JSON](https://github.com/rust-lang/rust/blob/c25ac9d6cc285e57e1176dc2da6848b9d0163810/src/librustdoc/html/render/write_shared.rs#L166) blobs from the contents of the already-rendered CCI. This proposal proposes to continue reading from the rendered cross-crate information under the default `--read-merged-cci=true`. It can also read `doc.parts` files, under `--fetch-parts`. However, there are several issues with reading from the rendered CCI that must be stated: -* If a user has a single `doc/` output directory, it is impossible to avoid shared mutation if every Rustdoc process is writing to the same CCI +* If a user has a single `doc/` output directory, it is impossible to avoid shared mutation if every rustdoc process is writing to the same CCI * It is difficult to extract the items in a diverse set of rendered HTML files. This is anticipating of the CCI to include HTML files that, for example, statically include type+trait implementations directly * Reading exclusively from `doc.parts` is simpler than the existing `serde_json` dependency for extracting the blobs, as opposed to handwritten CCI-type specific parsing (current) * With this proposal, there will be duplicate logic to read from both `doc.parts` files and rendered CCI. @@ -261,7 +443,7 @@ This proposal proposes to continue reading from the rendered cross-crate informa Require users to pass `--extern-html-root-url` on all external dependencies (current), vs. add a new flag to facilitate missing docs links being generated across all user crates (should consider) -Cross-crate links are an important consideration when merging documentation. Rustdoc provides a flag, `--extern-html-root-url`, which can provide fine-grain control over the generated URL for documentation. We believe this is already sufficient for generating cross-crate links, and we will make no changes to the generation of item links. Example usage of this flag is in the Guide-level explanation. +Cross-crate links are an important consideration when merging documentation. rustdoc provides a flag, `--extern-html-root-url`, which can provide fine-grain control over the generated URL for documentation. We believe this is already sufficient for generating cross-crate links, and we will make no changes to the generation of item links. Example usage of this flag is in the Guide-level explanation. A proposal is to provide more options to facilitate cross-crate links. For example, we could add a flag that implies `--extern-html-root-url` and `--fetch-parts` on all crates passed through `--extern`. It could be something like `--extern-html-root-url-all-user-crates`. User crates would be taken to mean the transitive dependencies of the current crate, excluding the standard library. This is complicated by things like . @@ -271,7 +453,7 @@ Create a new flag, `--write-merged-cci` (proposed), vs. use existing option `no_ There is a render option, `no_emit_shared`, which is used to conditionally generate the cross-crate information. It also controls the generation of static files, user CSS, etc. -This option is not configurable from the command line, and appears to be enabled unless Rustdoc is run in its example scraping mode. +This option is not configurable from the command line, and appears to be enabled unless rustdoc is run in its example scraping mode. We could make it configurable from the command line, unconditionally generate `doc.parts/`, and use it to gate the merging of CCI. @@ -279,14 +461,6 @@ We could also make it configurable from the command line, and use it to gate the We could also leave it as-is: always false unless we're scraping examples, and gate the generation of `doc.parts/` and the generation of all of the shared files. - - # Future possibilities This change could begin to facilitate type aliases and the trait aliases being From 08ac8a2b55fac4c79814c4643639fcf038793460 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Thu, 20 Jun 2024 17:35:41 +0000 Subject: [PATCH 04/28] merged -> rendered in flags --- ...0000-mergeable-rustdoc-cross-crate-info.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index bd6536e80ee..ccce4976a9a 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -38,7 +38,7 @@ echo "pub struct S; impl t::T for S {}" > s/src/lib.rs MERGED=$(realpath merged/doc) ``` -[Actively use](https://doc.rust-lang.org/rustc/command-line-arguments.html#--extern-specify-where-an-external-library-is-located) `t` and `s` in `i`. The `extern crate` declarations are not needed if you reference the crates in another way in the index; intra-doc links are enough. +[Actively use](https://doc.rust-lang.org/rustc/command-line-arguments.html#--extern-specify-where-an-external-library-is-located) `t` and `s` in `i`. The `extern crate` declarations are not needed if the crates are otherwise referenced in the index; intra-doc links are enough. ```shell echo "extern crate t; extern crate s;" > i/src/lib.rs @@ -52,17 +52,17 @@ rustc --crate-name=s --crate-type=lib --edition=2021 --emit=metadata --out-dir=s rustc --crate-name=i --crate-type=lib --edition=2021 --emit=metadata --out-dir=i/target --extern s=s/target/libs.rmeta --extern t=t/target/libt.rmeta -L t/target i/src/lib.rs ``` -Document `s` and `t` independently, providing `--write-merged-cci=false`, `--read-merged-cci=false`, and `--parts-out-dir=/target/doc.parts` +Document `s` and `t` independently, providing `--write-rendered-cci=false`, `--read-rendered-cci=false`, and `--parts-out-dir=/target/doc.parts` ```shell -rustdoc -Z unstable-options --crate-name=t --crate-type=lib --edition=2021 --out-dir=t/target/doc --extern-html-root-url t=$(MERGED) --write-merged-cci=false --read-merged-cci=false --parts-out-dir=t/target/doc.parts t/src/lib.rs -rustdoc -Z unstable-options --crate-name=s --crate-type=lib --edition=2021 --out-dir=s/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --write-merged-cci=false --read-merged-cci=false --parts-out-dir=t/target/doc.parts --extern t=t/target/libt.rmeta s/src/lib.rs +rustdoc -Z unstable-options --crate-name=t --crate-type=lib --edition=2021 --out-dir=t/target/doc --extern-html-root-url t=$(MERGED) --write-rendered-cci=false --read-rendered-cci=false --parts-out-dir=t/target/doc.parts t/src/lib.rs +rustdoc -Z unstable-options --crate-name=s --crate-type=lib --edition=2021 --out-dir=s/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --write-rendered-cci=false --read-rendered-cci=false --parts-out-dir=t/target/doc.parts --extern t=t/target/libt.rmeta s/src/lib.rs ``` -Link everything with a final invocation of rustdoc on `i`. We will **not** provide `--write-merged-cci=false`, because we are merging the parts. We will also provide `--read-merged-cci=false` and `--fetch-parts=`. See the Reference-level explanation about these flags. +Link everything with a final invocation of rustdoc on `i`. We will **not** provide `--write-rendered-cci=false`, because we are merging the parts. We will also provide `--read-rendered-cci=false` and `--fetch-parts=`. See the Reference-level explanation about these flags. ```shell -rustdoc -Z unstable-options --crate-name=i --crate-type=lib --edition=2021 --enable-index-page --out-dir=i/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --extern-html-root-url i=$(MERGED) --read-merged-cci=false --fetch-parts t=t/target/doc.parts --fetch-parts s=s/target/doc.parts --extern t=t/target/libt.rmeta --extern s=s/target/libs.rmeta -L t/target i/src/lib.rs +rustdoc -Z unstable-options --crate-name=i --crate-type=lib --edition=2021 --enable-index-page --out-dir=i/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --extern-html-root-url i=$(MERGED) --read-rendered-cci=false --fetch-parts t=t/target/doc.parts --fetch-parts s=s/target/doc.parts --extern t=t/target/libt.rmeta --extern s=s/target/libs.rmeta -L t/target i/src/lib.rs ``` Merge the docs with `cp`. This can be avoided if `--out-dir=$(MERGED)` is used for all of the rustdoc calls. We copy here to illustrate that documenting `s` is independent of documenting `t`, and could happen on separate machines. @@ -334,13 +334,13 @@ If this flag is not provided, no `doc.parts` will be written. This flag is the complement to `--fetch-parts`. -## New flag: `--write-merged-cci[=true|false]` +## New flag: `--write-rendered-cci[=true|false]` This flag defaults to true if not specified. With this flag is true, rustdoc will write the rendered CCI to the output directory, on each invocation. The user-facing behavior without the flag is the same as the current behavior. When this flag is false, the CCI will not be written nor rendered. Another call to a merge step will be required to merge the parts and write the CCI. -## New flag: `--read-merged-cci=[=true|false]` +## New flag: `--read-rendered-cci=[=true|false]` This flag defaults to true if not specified. @@ -433,7 +433,7 @@ The proposition, to allow users of rustdoc the flexibility of not having to prod Generate no extra files (current) vs. unconditionally creating `doc.parts` to enable more complex future CCI (should consider) The current version of rustdoc performs merging by [collecting JSON](https://github.com/rust-lang/rust/blob/c25ac9d6cc285e57e1176dc2da6848b9d0163810/src/librustdoc/html/render/write_shared.rs#L166) blobs from the contents of the already-rendered CCI. -This proposal proposes to continue reading from the rendered cross-crate information under the default `--read-merged-cci=true`. It can also read `doc.parts` files, under `--fetch-parts`. However, there are several issues with reading from the rendered CCI that must be stated: +This proposal proposes to continue reading from the rendered cross-crate information under the default `--read-rendered-cci=true`. It can also read `doc.parts` files, under `--fetch-parts`. However, there are several issues with reading from the rendered CCI that must be stated: * If a user has a single `doc/` output directory, it is impossible to avoid shared mutation if every rustdoc process is writing to the same CCI * It is difficult to extract the items in a diverse set of rendered HTML files. This is anticipating of the CCI to include HTML files that, for example, statically include type+trait implementations directly * Reading exclusively from `doc.parts` is simpler than the existing `serde_json` dependency for extracting the blobs, as opposed to handwritten CCI-type specific parsing (current) @@ -449,7 +449,7 @@ A proposal is to provide more options to facilitate cross-crate links. For examp ## Reuse existing option? -Create a new flag, `--write-merged-cci` (proposed), vs. use existing option `no_emit_shared` +Create a new flag, `--write-rendered-cci` (proposed), vs. use existing option `no_emit_shared` There is a render option, `no_emit_shared`, which is used to conditionally generate the cross-crate information. It also controls the generation of static files, user CSS, etc. @@ -472,4 +472,4 @@ Another possibility is for `doc.parts` to be distributed on `docs.rs` along with A future possibility related to the index crate idea is to have an option for embedding user-specified HTML into the `--enable-index-page`'s HTML. -The changes in this proposal are intended to work with no changes to Cargo and docs.rs. However, there may be benefits to using `--write-merged-cci=false` with Cargo, as it would remove the need for locking the output directory. More of the documentation process could happen in parallel, which may speed up execution time.. +The changes in this proposal are intended to work with no changes to Cargo and docs.rs. However, there may be benefits to using `--write-rendered-cci=false` with Cargo, as it would remove the need for locking the output directory. More of the documentation process could happen in parallel, which may speed up execution time.. From ef10ed53534f896b36edc2f148710762b93936ca Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Thu, 20 Jun 2024 17:44:15 +0000 Subject: [PATCH 05/28] fix typos --- ...0000-mergeable-rustdoc-cross-crate-info.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index ccce4976a9a..de3d37646ab 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -21,7 +21,7 @@ There are some files in the rustdoc output directory that are read and overwritt Build systems may run build actions in a distributed environment across separate logical filesystems. It might also be desirable to run rustdoc in a lock-free parallel mode, where every rustdoc process writes to a disjoint set of files. -Cargo fully supports cross-crate information, at the cost of requiring read-write access to the documentation root (`target/doc/`). There are significant scalability issues with this approach. +Cargo fully supports cross-crate information, at the cost of requiring global read-write access to the doc root (`target/doc`). There are significant scalability issues with this approach. Rustdoc needing global mutable access to the files that encode this cross-crate information has implications for caching, reproducible builds, and content hashing. By adding an option to avoid this mutation, rustdoc will serve as a first-class citizen in non-cargo build systems. @@ -287,14 +287,14 @@ $ tree . -a Currently, cross-crate information is written during the invocation of the `write_shared` function in [write_shared.rs](https://github.com/rust-lang/rust/blob/04ab7b2be0db3e6787f5303285c6b2ee6279868d/src/librustdoc/html/render/write_shared.rs#L47). This proposal does not add any new CCI or change their contents (modulo sorting order, whitespace). -The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc/`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of a the CCI. +The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of a the CCI. ## New subdirectory: `//` Typically, `` is selected as `./target/doc.parts`. The `//` files contain the unmerged contents of a single crates' version of their corresponding CCI. It is written if the flag `--parts-out-dir=` is provided. Every file in `//*` is a JSON array. Every element of the -array is a two-element array: a destination filename (relative to `doc/`), and +array is a two-element array: a destination filename (relative to the doc root), and the representation of the part. The representation of that part depends on the type of CCI that it describes. @@ -358,7 +358,7 @@ In the Guide-level explanation, for example, crate `i` needs to identify the loc ## Merge step -This step is provided with a list of crates. It merges their documentation. This step involves copying parts (individual item, module documentation) from each of the provided crates. It merges the parts, renders, and writes the CCI to the documentation root. +This step is provided with a list of crates. It merges their documentation. This step involves copying parts (individual item, module documentation) from each of the provided crates. It merges the parts, renders, and writes the CCI to the doc root. Discussion of the merge step is described in the Unresolved questions. @@ -368,7 +368,7 @@ The WIP may change the sorting order of the elements in the CCI. It does not cha # Rationale and alternatives -Running rustdoc in parallel is essential in enabling the tool to scale to large projects. The alternative, implemented by Cargo, is to run rustdoc in parallel by locking the CCI files. There are some environments where having synchronized access to the CCI is impossible. This proposal implements a reasonable approach to shared rustdoc, because it cleanly enables the addition of new kinds of CCI without changing existing documentation. +Running rustdoc in parallel is essential in enabling the tool to scale to large projects. The approach implemented by Cargo is to run rustdoc in parallel by locking the CCI files. There are some environments where having synchronized access to the CCI is impossible. This proposal implements a reasonable approach to shared rustdoc, because it cleanly enables the addition of new kinds of CCI without changing existing documentation. # Prior art @@ -426,7 +426,7 @@ Require users to generate documentation bundles via an index crate (current) vs. If one would like to merge the documentation of several crates, we could continue to require users to provide an index crate, like [the fuchsia index](https://fuchsia-docs.firebaseapp.com/rust/rustdoc_index/). This serves as the target of the rustdoc invocation, and the landing page for the collected documentation. Supporting only this style of index would require the fewest changes. This is the mode described in the Guide-level explanation. -The proposition, to allow users of rustdoc the flexibility of not having to produce an index, is to allow rustdoc to be run in a mode where no target crate is provided. The source crates are provided to rustdoc, through a mechanism like `--extern`, rustdoc merges and writes the CCI, and copies the item and module links to `doc/`. This would require more extensive changes, as rustdoc assumes that it is invoked with a target crate. This mode is somewhat analogous to the [example scraping mode](https://github.com/rust-lang/rfcs/blob/master/text/3123-rustdoc-scrape-examples.md). Having to create an index crate, that actively uses all of the crates in the environment, might prohibit the use of this feature in settings where users do not intend to produce an index, or where exhaustively listing all dependencies (to `--extern` them) is difficult. +The proposition, to allow users of rustdoc the flexibility of not having to produce an index, is to allow rustdoc to be run in a mode where no target crate is provided. The source crates are provided to rustdoc, through a mechanism like `--extern`, rustdoc merges and writes the CCI, and copies the item and module links to the doc root. This would require more extensive changes, as rustdoc assumes that it is invoked with a target crate. This mode is somewhat analogous to the [example scraping mode](https://github.com/rust-lang/rfcs/blob/master/text/3123-rustdoc-scrape-examples.md). Having to create an index crate, that actively uses all of the crates in the environment, might prohibit the use of this feature in settings where users do not intend to produce an index, or where exhaustively listing all dependencies (to `--extern` them) is difficult. ## Unconditionally generating the `doc.parts` files? @@ -434,7 +434,7 @@ Generate no extra files (current) vs. unconditionally creating `doc.parts` to en The current version of rustdoc performs merging by [collecting JSON](https://github.com/rust-lang/rust/blob/c25ac9d6cc285e57e1176dc2da6848b9d0163810/src/librustdoc/html/render/write_shared.rs#L166) blobs from the contents of the already-rendered CCI. This proposal proposes to continue reading from the rendered cross-crate information under the default `--read-rendered-cci=true`. It can also read `doc.parts` files, under `--fetch-parts`. However, there are several issues with reading from the rendered CCI that must be stated: -* If a user has a single `doc/` output directory, it is impossible to avoid shared mutation if every rustdoc process is writing to the same CCI +* Every rustdoc process outputs the CCI to the same doc root by default * It is difficult to extract the items in a diverse set of rendered HTML files. This is anticipating of the CCI to include HTML files that, for example, statically include type+trait implementations directly * Reading exclusively from `doc.parts` is simpler than the existing `serde_json` dependency for extracting the blobs, as opposed to handwritten CCI-type specific parsing (current) * With this proposal, there will be duplicate logic to read from both `doc.parts` files and rendered CCI. @@ -455,11 +455,11 @@ There is a render option, `no_emit_shared`, which is used to conditionally gener This option is not configurable from the command line, and appears to be enabled unless rustdoc is run in its example scraping mode. -We could make it configurable from the command line, unconditionally generate `doc.parts/`, and use it to gate the merging of CCI. +We could make it configurable from the command line, unconditionally generate `doc.parts`, and use it to gate the merging of CCI. -We could also make it configurable from the command line, and use it to gate the generation of `doc.parts/` and the generation of all of the shared files. +We could also make it configurable from the command line, and use it to gate the generation of `doc.parts` and the generation of all of the shared files. -We could also leave it as-is: always false unless we're scraping examples, and gate the generation of `doc.parts/` and the generation of all of the shared files. +We could also leave it as-is: always false unless we're scraping examples, and gate the generation of `doc.parts` and the generation of all of the shared files. # Future possibilities From 499425169ba25b599b16acb216eea72d89e12e99 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Thu, 20 Jun 2024 17:46:30 +0000 Subject: [PATCH 06/28] whitespace and clear lines --- ...0000-mergeable-rustdoc-cross-crate-info.md | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index de3d37646ab..d04b0d74caf 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -15,9 +15,9 @@ Mergeable cross-crate information in rustdoc. Facilitates the generation of docu # Motivation -The main goal of this proposal is to facilitate users producing a documentation bundle of every crate in a large environment. When a crate needs to be re-documented, only a relatively lightweight merge step will be needed to produce a complete documentation bundle. This proposal is to facilitate the creation and updating of these bundles. +The main goal of this proposal is to facilitate users producing a documentation bundle of every crate in a large environment. When a crate needs to be re-documented, only a relatively lightweight merge step will be needed to produce a complete documentation bundle. This proposal is to facilitate the creation and updating of these bundles. -There are some files in the rustdoc output directory that are read and overwritten during every invocation of rustdoc. This proposal refers to these files as **cross-crate information**, or **CCI**, as in . +There are some files in the rustdoc output directory that are read and overwritten during every invocation of rustdoc. This proposal refers to these files as **cross-crate information**, or **CCI**, as in . Build systems may run build actions in a distributed environment across separate logical filesystems. It might also be desirable to run rustdoc in a lock-free parallel mode, where every rustdoc process writes to a disjoint set of files. @@ -29,7 +29,7 @@ These considerations motivate adding an option for outputting partial CCI (parts # Guide-level explanation -In this example, there is a crate `t` which defines a trait `T`, and a crate `s` which defines a struct `S` that implements `T`. Our goal in this demo is for `S` to appear as in implementer in `T`'s docs, even if `s` and `t` are documented independently. This guide will be assuming that we want a crate `i` that serves as our documentation index. See the Unresolved questions section for ideas that do not require an index crate. +In this example, there is a crate `t` which defines a trait `T`, and a crate `s` which defines a struct `S` that implements `T`. Our goal in this demo is for `S` to appear as in implementer in `T`'s docs, even if `s` and `t` are documented independently. This guide will be assuming that we want a crate `i` that serves as our documentation index. See the Unresolved questions section for ideas that do not require an index crate. ```shell mkdir -p t/src s/src i/src merged/doc @@ -68,7 +68,7 @@ rustdoc -Z unstable-options --crate-name=i --crate-type=lib --edition=2021 --en Merge the docs with `cp`. This can be avoided if `--out-dir=$(MERGED)` is used for all of the rustdoc calls. We copy here to illustrate that documenting `s` is independent of documenting `t`, and could happen on separate machines. ```shell -cp -r s/target/doc/* t/target/doc/* i/target/doc/* merged/doc +cp -r s/target/doc/* t/target/doc/* i/target/doc/* ``` Browse `merged/doc/index.html` with cross-crate information. @@ -279,7 +279,7 @@ $ tree . -a │ └── type-impl └── libs.rmeta - + @@ -320,7 +320,7 @@ This part is the also crate name. It represents a different kind of CCI because * `//type-impl`: for `doc/type.impl/**/*.js` -This part is a two element array with the crate name and the JSON representation of a type implementation. +This part is a two element array with the crate name and the JSON representation of a type implementation. * `//trait-impl`: for `doc/trait.impl/**/*.js` @@ -385,7 +385,7 @@ Bazel has `rules_rust` for building Rust targets and rustdoc documentation. It does not document crates' dependencies. `search-index.js`, for example, is both a dependency and an output file for rustdoc in multi-crate documentation environments. If it is declared as a dependency in this way, Bazel could not build docs for the members of an environment in parallel with a single output directory, as it strictly enforces hermiticity. For a recursive, parallel rustdoc to ever serve as a first-class citizen in Bazel, changes similar to the ones described in this proposal would be needed. -There is an [open issue](https://github.com/bazelbuild/rules_rust/issues/1837) raised about the fact that Bazel does not document crates dependencies. The comments in the issue discuss a pull request on Bazel that documents each crates dependencies in a separate output directory. It is noted in the discussion that this solution, being implemented under the current rustdoc, "doesn't scale well and it should be implemented in a different manner long term." In order to get CCI in a mode like this, rustdoc would need to adopt changes, like the ones in this proposal, for merging cross-crate information. +There is an [open issue](https://github.com/bazelbuild/rules_rust/issues/1837) raised about the fact that Bazel does not document crates dependencies. The comments in the issue discuss a pull request on Bazel that documents each crates dependencies in a separate output directory. It is noted in the discussion that this solution, being implemented under the current rustdoc, "doesn't scale well and it should be implemented in a different manner long term." In order to get CCI in a mode like this, rustdoc would need to adopt changes, like the ones in this proposal, for merging cross-crate information. ## Buck2 @@ -405,7 +405,7 @@ buck2 build //:library[doc] --verbose 5 ``` --> -It has a subtarget, `[doc]`, for generating rustdoc for a crate. +It has a subtarget, `[doc]`, for generating rustdoc for a crate. You can provide a coarse-grained `extern-html-root-url` for all dependencies. You could document all crates independently, but no cross-crate information would be shared. @@ -432,12 +432,12 @@ The proposition, to allow users of rustdoc the flexibility of not having to prod Generate no extra files (current) vs. unconditionally creating `doc.parts` to enable more complex future CCI (should consider) -The current version of rustdoc performs merging by [collecting JSON](https://github.com/rust-lang/rust/blob/c25ac9d6cc285e57e1176dc2da6848b9d0163810/src/librustdoc/html/render/write_shared.rs#L166) blobs from the contents of the already-rendered CCI. +The current version of rustdoc performs merging by [collecting JSON](https://github.com/rust-lang/rust/blob/c25ac9d6cc285e57e1176dc2da6848b9d0163810/src/librustdoc/html/render/write_shared.rs#L166) blobs from the contents of the already-rendered CCI. This proposal proposes to continue reading from the rendered cross-crate information under the default `--read-rendered-cci=true`. It can also read `doc.parts` files, under `--fetch-parts`. However, there are several issues with reading from the rendered CCI that must be stated: * Every rustdoc process outputs the CCI to the same doc root by default * It is difficult to extract the items in a diverse set of rendered HTML files. This is anticipating of the CCI to include HTML files that, for example, statically include type+trait implementations directly * Reading exclusively from `doc.parts` is simpler than the existing `serde_json` dependency for extracting the blobs, as opposed to handwritten CCI-type specific parsing (current) -* With this proposal, there will be duplicate logic to read from both `doc.parts` files and rendered CCI. +* With this proposal, there will be duplicate logic to read from both `doc.parts` files and rendered CCI. ## Item links? @@ -446,14 +446,14 @@ Require users to pass `--extern-html-root-url` on all external dependencies (cur Cross-crate links are an important consideration when merging documentation. rustdoc provides a flag, `--extern-html-root-url`, which can provide fine-grain control over the generated URL for documentation. We believe this is already sufficient for generating cross-crate links, and we will make no changes to the generation of item links. Example usage of this flag is in the Guide-level explanation. A proposal is to provide more options to facilitate cross-crate links. For example, we could add a flag that implies `--extern-html-root-url` and `--fetch-parts` on all crates passed through `--extern`. It could be something like `--extern-html-root-url-all-user-crates`. User crates would be taken to mean the transitive dependencies of the current crate, excluding the standard library. This is complicated by things like . - + ## Reuse existing option? -Create a new flag, `--write-rendered-cci` (proposed), vs. use existing option `no_emit_shared` - +Create a new flag, `--write-rendered-cci` (proposed), vs. use existing option `no_emit_shared` + There is a render option, `no_emit_shared`, which is used to conditionally generate the cross-crate information. It also controls the generation of static files, user CSS, etc. -This option is not configurable from the command line, and appears to be enabled unless rustdoc is run in its example scraping mode. +This option is not configurable from the command line, and appears to be enabled unless rustdoc is run in its example scraping mode. We could make it configurable from the command line, unconditionally generate `doc.parts`, and use it to gate the merging of CCI. @@ -468,7 +468,7 @@ statically compiled as part of the .html documentation, instead of being loaded as separate JavaScript files. Each type and trait alias could be stored as an HTML part, which are then merged into the regular documentation. -Another possibility is for `doc.parts` to be distributed on `docs.rs` along with the regular documentation. This would facilitate a mode where documentation of the dependencies could be downloaded externally, instead of being rebuilt locally. +Another possibility is for `doc.parts` to be distributed on `docs.rs` along with the regular documentation. This would facilitate a mode where documentation of the dependencies could be downloaded externally, instead of being rebuilt locally. A future possibility related to the index crate idea is to have an option for embedding user-specified HTML into the `--enable-index-page`'s HTML. From 0f01c9b713d605a9c7546ef896a5a43e0e829226 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Thu, 20 Jun 2024 17:54:48 +0000 Subject: [PATCH 07/28] typo --- text/0000-mergeable-rustdoc-cross-crate-info.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index d04b0d74caf..3e4572f6047 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -68,7 +68,7 @@ rustdoc -Z unstable-options --crate-name=i --crate-type=lib --edition=2021 --en Merge the docs with `cp`. This can be avoided if `--out-dir=$(MERGED)` is used for all of the rustdoc calls. We copy here to illustrate that documenting `s` is independent of documenting `t`, and could happen on separate machines. ```shell -cp -r s/target/doc/* t/target/doc/* i/target/doc/* +cp -r s/target/doc/* t/target/doc/* i/target/doc/* merged/doc ``` Browse `merged/doc/index.html` with cross-crate information. From 7db0a4a8e2c18a46d4951340e0e9bfe332203d5b Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Thu, 20 Jun 2024 18:49:42 +0000 Subject: [PATCH 08/28] remove work in progress --- text/0000-mergeable-rustdoc-cross-crate-info.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index 3e4572f6047..914731a6933 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -3,12 +3,6 @@ - RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) - Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) -# Rustdoc Proposal - Merge Documentation From Multiple Crates - -# Work in progress - - - # Summary Mergeable cross-crate information in rustdoc. Facilitates the generation of documentation indexes in environments with many crates by allowing each crate to write to an independent output directory. Final documentation is rendered with a lightweight merge step. Configurable with command-line flags, this proposal writes a `doc.parts` directory to hold pre-merge cross-crate information. Currently, rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating rustdoc in build systems that make build actions independent. From 23e34e081bf2b88fcbfd2f182049a8e584214cb3 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Fri, 21 Jun 2024 21:22:28 +0000 Subject: [PATCH 09/28] address comments from P1n3appl3 --- ...0000-mergeable-rustdoc-cross-crate-info.md | 25 +++++++------------ 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index 914731a6933..00aed1f2970 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -5,12 +5,14 @@ # Summary -Mergeable cross-crate information in rustdoc. Facilitates the generation of documentation indexes in environments with many crates by allowing each crate to write to an independent output directory. Final documentation is rendered with a lightweight merge step. Configurable with command-line flags, this proposal writes a `doc.parts` directory to hold pre-merge cross-crate information. Currently, rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating rustdoc in build systems that make build actions independent. +Mergeable cross-crate information in rustdoc. Facilitates the generation of documentation indexes in workspaces with many crates by allowing each crate to write to an independent output directory. Final documentation is rendered with a lightweight merge step. Configurable with command-line flags, this proposal writes a `doc.parts` directory to hold pre-merge cross-crate information. Currently, rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating rustdoc in build systems that make build actions independent. # Motivation The main goal of this proposal is to facilitate users producing a documentation bundle of every crate in a large environment. When a crate needs to be re-documented, only a relatively lightweight merge step will be needed to produce a complete documentation bundle. This proposal is to facilitate the creation and updating of these bundles. +This proposal also targets documenting individual crates and their dependencies in non-cargo build systems. As will be explained, doc targets in non-cargo build systems often do not support cross-crate information. + There are some files in the rustdoc output directory that are read and overwritten during every invocation of rustdoc. This proposal refers to these files as **cross-crate information**, or **CCI**, as in . Build systems may run build actions in a distributed environment across separate logical filesystems. It might also be desirable to run rustdoc in a lock-free parallel mode, where every rustdoc process writes to a disjoint set of files. @@ -44,7 +46,7 @@ Compile the crates. rustc --crate-name=t --crate-type=lib --edition=2021 --emit=metadata --out-dir=t/target t/src/lib.rs rustc --crate-name=s --crate-type=lib --edition=2021 --emit=metadata --out-dir=s/target --extern t=t/target/libt.rmeta s/src/lib.rs rustc --crate-name=i --crate-type=lib --edition=2021 --emit=metadata --out-dir=i/target --extern s=s/target/libs.rmeta --extern t=t/target/libt.rmeta -L t/target i/src/lib.rs -``` + Document `s` and `t` independently, providing `--write-rendered-cci=false`, `--read-rendered-cci=false`, and `--parts-out-dir=/target/doc.parts` @@ -76,9 +78,6 @@ In general, instead of two crates in the environment (`s` and `t`) you could hav $ tree . -a . ├── i -│ ├── Cargo.lock -│ ├── Cargo.toml -│ ├── .gitignore │ ├── src │ │ └── lib.rs │ └── target @@ -217,9 +216,6 @@ $ tree . -a │ ├── sidebar-items.js │ └── trait.T.html ├── s -│ ├── Cargo.lock -│ ├── Cargo.toml -│ ├── .gitignore │ ├── src │ │ └── lib.rs │ └── target @@ -245,9 +241,6 @@ $ tree . -a │ │ └── type-impl │ └── libs.rmeta └── t - ├── Cargo.lock - ├── Cargo.toml - ├── .gitignore ├── src │ └── lib.rs └── target @@ -310,7 +303,7 @@ This part is the crate name. * `//crates-index`: for `doc/crates.js`, `doc/index.html` -This part is the also crate name. It represents a different kind of CCI because it is written to a `doc/index.html`, and rendered as an HTML document instead as JSON. In principal, we could use this part to add more information to the crates index `doc/index.html` (the first line of the top level crate documentation, for example). +This part is also the crate name. It represents a different kind of CCI because it is written to a `doc/index.html`, and rendered as an HTML document instead as JSON. In principal, we could use this part to add more information to the crates index `doc/index.html` (the first line of the top level crate documentation, for example). * `//type-impl`: for `doc/type.impl/**/*.js` @@ -362,13 +355,13 @@ The WIP may change the sorting order of the elements in the CCI. It does not cha # Rationale and alternatives -Running rustdoc in parallel is essential in enabling the tool to scale to large projects. The approach implemented by Cargo is to run rustdoc in parallel by locking the CCI files. There are some environments where having synchronized access to the CCI is impossible. This proposal implements a reasonable approach to shared rustdoc, because it cleanly enables the addition of new kinds of CCI without changing existing documentation. +Running rustdoc in parallel is essential in enabling the tool to scale to large projects. The approach implemented by Cargo is to run rustdoc in parallel by locking the CCI files. There are some workspaces where having synchronized access to the CCI is impossible. This proposal implements a reasonable approach to shared rustdoc, because it cleanly enables the addition of new kinds of CCI without changing existing documentation. # Prior art Prior art for linking and merging independently generated documentation was **not** identified in Javadoc, Godoc, Doxygen, Sphinx (intersphinx), nor any documentation system for other languages. Analogs of cross-crate information were not found, but a more thorough investigation or experience with other systems may be needed. -However, the issues presented here have been encountered in multiple build systems that interact with rustdoc. They limit the usefulness of rustdoc in large environments. +However, the issues presented here have been encountered in multiple build systems that interact with rustdoc. They limit the usefulness of rustdoc in large workspaces. ## Bazel @@ -377,7 +370,7 @@ Bazel has `rules_rust` for building Rust targets and rustdoc documentation. * * -It does not document crates' dependencies. `search-index.js`, for example, is both a dependency and an output file for rustdoc in multi-crate documentation environments. If it is declared as a dependency in this way, Bazel could not build docs for the members of an environment in parallel with a single output directory, as it strictly enforces hermiticity. For a recursive, parallel rustdoc to ever serve as a first-class citizen in Bazel, changes similar to the ones described in this proposal would be needed. +It does not document crates' dependencies. `search-index.js`, for example, is both a dependency and an output file for rustdoc in multi-crate documentation workspaces. If it is declared as a dependency in this way, Bazel could not build docs for the members of an environment in parallel with a single output directory, as it strictly enforces hermiticity. For a recursive, parallel rustdoc to ever serve as a first-class citizen in Bazel, changes similar to the ones described in this proposal would be needed. There is an [open issue](https://github.com/bazelbuild/rules_rust/issues/1837) raised about the fact that Bazel does not document crates dependencies. The comments in the issue discuss a pull request on Bazel that documents each crates dependencies in a separate output directory. It is noted in the discussion that this solution, being implemented under the current rustdoc, "doesn't scale well and it should be implemented in a different manner long term." In order to get CCI in a mode like this, rustdoc would need to adopt changes, like the ones in this proposal, for merging cross-crate information. @@ -403,7 +396,7 @@ It has a subtarget, `[doc]`, for generating rustdoc for a crate. You can provide a coarse-grained `extern-html-root-url` for all dependencies. You could document all crates independently, but no cross-crate information would be shared. -It does not document crates' dependencies for the same reason that Bazel does not. +buck2 does not natively merge rustdoc from separate targets. The buck2 maintainers have a [proprietary search backend](https://rust-lang.zulipchat.com/#narrow/stream/266220-t-rustdoc/topic/mergable.20rustdoc.20proposal/near/445952204) that merges and parses `search-index.js` files from separately documented crates. Their proprietary tooling does not handle cross-crate trait implementations from upstream crates. By implementing this merging directly in rustdoc, we could avoid fragmentation and bring cross-crate information to more consumers. From 97cf6c6630b2475442ae55b930a7a95f001b2d35 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Mon, 24 Jun 2024 18:15:37 +0000 Subject: [PATCH 10/28] changes in response to jsha and notriddle --- ...0000-mergeable-rustdoc-cross-crate-info.md | 142 ++++++++---------- 1 file changed, 59 insertions(+), 83 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index 00aed1f2970..183c742a395 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -5,7 +5,7 @@ # Summary -Mergeable cross-crate information in rustdoc. Facilitates the generation of documentation indexes in workspaces with many crates by allowing each crate to write to an independent output directory. Final documentation is rendered with a lightweight merge step. Configurable with command-line flags, this proposal writes a `doc.parts` directory to hold pre-merge cross-crate information. Currently, rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating rustdoc in build systems that make build actions independent. +Mergeable cross-crate information in rustdoc. Facilitates the generation of documentation indexes in workspaces with many crates by allowing each crate to write to an independent output directory. Final documentation is rendered with a lightweight merge step. Configurable with command-line flags, this proposal writes a `crate-info.json` directory to hold pre-merge cross-crate information. Currently, rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating rustdoc in build systems that make build actions independent. # Motivation @@ -31,7 +31,7 @@ In this example, there is a crate `t` which defines a trait `T`, and a crate `s` mkdir -p t/src s/src i/src merged/doc echo "pub trait T {}" > t/src/lib.rs echo "pub struct S; impl t::T for S {}" > s/src/lib.rs -MERGED=$(realpath merged/doc) +MERGED=file://$(realpath merged/doc) ``` [Actively use](https://doc.rust-lang.org/rustc/command-line-arguments.html#--extern-specify-where-an-external-library-is-located) `t` and `s` in `i`. The `extern crate` declarations are not needed if the crates are otherwise referenced in the index; intra-doc links are enough. @@ -46,19 +46,19 @@ Compile the crates. rustc --crate-name=t --crate-type=lib --edition=2021 --emit=metadata --out-dir=t/target t/src/lib.rs rustc --crate-name=s --crate-type=lib --edition=2021 --emit=metadata --out-dir=s/target --extern t=t/target/libt.rmeta s/src/lib.rs rustc --crate-name=i --crate-type=lib --edition=2021 --emit=metadata --out-dir=i/target --extern s=s/target/libs.rmeta --extern t=t/target/libt.rmeta -L t/target i/src/lib.rs +``` - -Document `s` and `t` independently, providing `--write-rendered-cci=false`, `--read-rendered-cci=false`, and `--parts-out-dir=/target/doc.parts` +Document `s` and `t` independently, providing `--merge=none` and `--write-info-json=/target/doc.parts` ```shell -rustdoc -Z unstable-options --crate-name=t --crate-type=lib --edition=2021 --out-dir=t/target/doc --extern-html-root-url t=$(MERGED) --write-rendered-cci=false --read-rendered-cci=false --parts-out-dir=t/target/doc.parts t/src/lib.rs -rustdoc -Z unstable-options --crate-name=s --crate-type=lib --edition=2021 --out-dir=s/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --write-rendered-cci=false --read-rendered-cci=false --parts-out-dir=t/target/doc.parts --extern t=t/target/libt.rmeta s/src/lib.rs +rustdoc -Z unstable-options --crate-name=t --crate-type=lib --edition=2021 --out-dir=t/target/doc --extern-html-root-url t=$(MERGED) --merge=none --write-info-json=t/target/doc.parts t/src/lib.rs +rustdoc -Z unstable-options --crate-name=s --crate-type=lib --edition=2021 --out-dir=s/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --merge=none --write-info-json=t/target/doc.parts --extern t=t/target/libt.rmeta s/src/lib.rs ``` -Link everything with a final invocation of rustdoc on `i`. We will **not** provide `--write-rendered-cci=false`, because we are merging the parts. We will also provide `--read-rendered-cci=false` and `--fetch-parts=`. See the Reference-level explanation about these flags. +Link everything with a final invocation of rustdoc on `i`. We will provide `--merge=write-only` and `--include-info-json=`. See the Reference-level explanation about these flags. ```shell -rustdoc -Z unstable-options --crate-name=i --crate-type=lib --edition=2021 --enable-index-page --out-dir=i/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --extern-html-root-url i=$(MERGED) --read-rendered-cci=false --fetch-parts t=t/target/doc.parts --fetch-parts s=s/target/doc.parts --extern t=t/target/libt.rmeta --extern s=s/target/libs.rmeta -L t/target i/src/lib.rs +rustdoc -Z unstable-options --crate-name=i --crate-type=lib --edition=2021 --enable-index-page --out-dir=i/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --extern-html-root-url i=$(MERGED) --merge=write-only --include-info-json t=t/target/doc.parts --include-info-json s=s/target/doc.parts --extern t=t/target/libt.rmeta --extern s=s/target/libs.rmeta -L t/target i/src/lib.rs ``` Merge the docs with `cp`. This can be avoided if `--out-dir=$(MERGED)` is used for all of the rustdoc calls. We copy here to illustrate that documenting `s` is independent of documenting `t`, and could happen on separate machines. @@ -91,14 +91,6 @@ $ tree . -a │ │ │ └── sidebar-items.js │ │ ├── index.html │ │ ├── .lock -│ │ ├── .parts -│ │ │ └── i -│ │ │ ├── crates-js -│ │ │ ├── index-html -│ │ │ ├── search-index-js -│ │ │ ├── src-files-js -│ │ │ ├── trait-impl -│ │ │ └── type-impl │ │ ├── search.desc │ │ │ ├── i │ │ │ │ └── i-desc-0-.js @@ -233,12 +225,7 @@ $ tree . -a │ │ └── lib.rs.html │ ├── doc.parts │ │ └── s -│ │ ├── crates-js -│ │ ├── index-html -│ │ ├── search-index-js -│ │ ├── src-files-js -│ │ ├── trait-impl -│ │ └── type-impl +│ │ └── crate-info.json │ └── libs.rmeta └── t ├── src @@ -258,12 +245,7 @@ $ tree . -a │ └── lib.rs.html ├── doc.parts │ └── t - │ ├── crates-js - │ ├── index-html - │ ├── search-index-js - │ ├── src-files-js - │ ├── trait-impl - │ └── type-impl +│ │ └── crate-info.json └── libs.rmeta @@ -276,78 +258,84 @@ Currently, cross-crate information is written during the invocation of the `writ The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of a the CCI. -## New subdirectory: `//` +## New subdirectory: `//crate-info.json` -Typically, `` is selected as `./target/doc.parts`. The `//` files contain the unmerged contents of a single crates' version of their corresponding CCI. It is written if the flag `--parts-out-dir=` is provided. +The `//crate-info.json` file contains the unmerged contents of a single crates' version of their corresponding CCI. Typically, `` is selected as `./target/doc.parts`. This file is written if the flag `--write-info-json=` is provided. -Every file in `//*` is a JSON array. Every element of the -array is a two-element array: a destination filename (relative to the doc root), and -the representation of the part. The representation of that part depends on the type -of CCI that it describes. +`crate-info.json` is a JSON object with several key-value pairs. Every value is a JSON array. Every element of the JSON array is a two-element array: a destination filename relative to the doc root, and the representation of the part. The representation of the part depends on the type of CCI that it describes. -* `//src-files-js`: for `doc/src-files.js` +* key: `src-files-js`, for `doc/src-files.js` This part is the JSON representation of the source index that is later stored in a `srcIndex` global variable. -* `//search-index-js`: for `doc/search-index.js` +* key: `search-index-js`, for `doc/search-index.js` This part is the JSON encoded search index, before it has been installed in `search-index.js`. -* `//search-desc`: for `doc/search.desc/**/*.js` - -This part contains the JavaScript code to load a shard of the search descriptions. - -* `//all-crates`: for `doc/crates.js`, `/doc/index.html` +* key: `all-crates`, for `doc/crates.js` This part is the crate name. -* `//crates-index`: for `doc/crates.js`, `doc/index.html` +* key: `crates-index`, for `doc/index.html` This part is also the crate name. It represents a different kind of CCI because it is written to a `doc/index.html`, and rendered as an HTML document instead as JSON. In principal, we could use this part to add more information to the crates index `doc/index.html` (the first line of the top level crate documentation, for example). -* `//type-impl`: for `doc/type.impl/**/*.js` +* key: `type-impl`, for `doc/type.impl/**/*.js` This part is a two element array with the crate name and the JSON representation of a type implementation. -* `//trait-impl`: for `doc/trait.impl/**/*.js` +* key: `trait-impl`, for `doc/trait.impl/**/*.js` This part is a two element array with the crate name and the JSON representation of a trait implementation. -## New flag: `--parts-out-dir=` +## New flag: `--write-info-json=` -When this flag is provided, the unmerged parts for the current crate will be written to `//`. A typical `` is `./target/doc.parts`. +When this flag is provided, the unmerged parts for the current crate will be written to `//crate-info.json`. A typical `` is `./target/doc.parts`. -If this flag is not provided, no `doc.parts` will be written. +Crates `--include-info-json`ed will not appear in `crate-info.json`, as `crate-info.json` only includes the CCI parts for the current crate. -This flag is the complement to `--fetch-parts`. +If this flag is not provided, no `crate-info.json` will be written. -## New flag: `--write-rendered-cci[=true|false]` +## New flag: `--include-info-json =` -This flag defaults to true if not specified. +If this flag is provided, rustdoc will expect `//crate-info.json` to contain the parts for ``. It will append these parts to the ones it will render in the doc root (`--out-dir`). This info is not written to `crate-info.json`, as `crate-info.json` only holds the CCI parts for the current crate. -With this flag is true, rustdoc will write the rendered CCI to the output directory, on each invocation. The user-facing behavior without the flag is the same as the current behavior. When this flag is false, the CCI will not be written nor rendered. Another call to a merge step will be required to merge the parts and write the CCI. +This flag is similar to `--extern-html-root-url` in that it only applies to externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided in `--extern-html-root-url` never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--include-info-json` tells rustdoc where to search for the `crate-info.json` directory at documentation-time. It must not be a hyperlink. -## New flag: `--read-rendered-cci=[=true|false]` +In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc.parts/s`, so rustdoc is called with `--include-info-json s=s/target/doc.parts`. -This flag defaults to true if not specified. +## New flag: `--merge=auto|none|write-only` -When this flag is true, rustdoc will read the rendered cross crate information from the doc root. Additional parts included via `--fetch-parts` will be appended to these parts. When this flag is false, the parts will initialized empty. +This flag controls two internal paramaters: `read_rendered_cci`, and `write_rendered_cci`. -## New flag: `--fetch-parts =` +When `write_rendered_cci` is active, rustdoc will output the rendered parts to the doc root (`--out-dir`). Rustdoc will generate files like `doc/search-index.js`, `doc/search.desc`, `doc/index.html`, etc if and only if this parameter is true. -If this flag is provided, rustdoc will expect `//` to contain the parts for ``. It will append these parts to the ones it will output. +When `read_rendered_cci` is active, rustdoc will look in the `--out-dir` for rendered cross-crate info files. These files will be used as the base. Any new parts that rustdoc generates with its current invocation and any parts fetched with `include-info-json` will be appended to these base files. When it is disabled, the cross-crate info files start empty and are populated with the current crate's info and any crates fetched with `--include-info-json`. -This flag is the complement to `--parts-out-dir`. - -This flag is similar to `--extern-html-root-url` in that it only applies to externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided is never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--fetch-parts` tells rustdoc where to search for the `doc.parts` directory at documentation-time. It must not be a hyperlink. - -In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc.parts/s`, so rustdoc is called with `--fetch-parts s=s/target/doc.parts`. +* `--merge=auto` (`read_rendered_cci && write_rendered_cci`) is the default, and reflects the current behavior of rustdoc. +* `--merge=none` (`!read_rendered_cci && !write_rendered_cci`) means that rustdoc will ignore the cross-crate files in the doc root. Only generate item docs. +* `--merge=write-only` (`!read_rendered_cci && write_rendered_cci`) only outputs item docs that are based on the current crate and `--include-info-json`'ed crates. +* A (`read_rendered_cci && !write_rendered_cci`) mode would be useless, since the data that is read would be ignored and not written. ## Merge step This step is provided with a list of crates. It merges their documentation. This step involves copying parts (individual item, module documentation) from each of the provided crates. It merges the parts, renders, and writes the CCI to the doc root. -Discussion of the merge step is described in the Unresolved questions. +In short, without any specific merge step, merging the documentation for multiple crates is performed as shown in the Guide-level explanation. + +Discussion of a dedicated merge step is described in the Unresolved questions (Index crate). + +## Compatibility + +This RFC does not alter previous compatibility guarantees made about the output of rustdoc. In particular it does not stabilize the presence of the rendered cross-crate information files, their content, or the HTML generated by rustdoc. + +In the same way that the [rustdoc HTML output is unstable](https://rust-lang.github.io/rfcs/2963-rustdoc-json.html#:~:text=The%20HTML%20output%20of%20rustdoc,into%20a%20different%20format%20impractical), the contents of the `crate-info.json` will be considered unstable. Between versions of rustdoc, breaking changes to the content of `crate-info.json` should be expected. Only the presence of a `crate-info.json` file is promised, under `--write-info-json`. Merging cross-crate information generated by disparate versions of rustdoc is not supported. + +The implementation of the RFC itself is designed to produce only minimal changes to cross-crate info files and the HTML output of rustdoc. Exhaustively, the implementation is allowed to +* Change the sorting order of trait implementations, type implementations, and other cross-crate info in the HTML output of rustdoc +* Add a comment on the last line of generated HTML pages, to store metadata relevant to appending items to them +* Refactor the JavaScript contents of cross-crate information files, in ways that do not change their overall behavior. If the JavaScript fragment declared an array called `ALL_CRATES` with certain contents, it will continue to do so. +Changes this minimal are intended to avoid breaking tools that use the output of rustdoc, like Cargo, docs.rs, and rustdoc's JavaScript frontend, in the near-term. Going forward, rustdoc will not make formal guarantees about the content of cross-crate info files. # Drawbacks @@ -378,23 +366,9 @@ There is an [open issue](https://github.com/bazelbuild/rules_rust/issues/1837) r The Buck2 build system has rules for building and testing rust binaries and libraries. - - - It has a subtarget, `[doc]`, for generating rustdoc for a crate. -You can provide a coarse-grained `extern-html-root-url` for all dependencies. You could document all crates independently, but no cross-crate information would be shared. +You can provide `extern-html-root-url`. You can document all crates independently and manually merge them but no cross-crate information would be shared. buck2 does not natively merge rustdoc from separate targets. The buck2 maintainers have a [proprietary search backend](https://rust-lang.zulipchat.com/#narrow/stream/266220-t-rustdoc/topic/mergable.20rustdoc.20proposal/near/445952204) that merges and parses `search-index.js` files from separately documented crates. Their proprietary tooling does not handle cross-crate trait implementations from upstream crates. By implementing this merging directly in rustdoc, we could avoid fragmentation and bring cross-crate information to more consumers. @@ -413,30 +387,32 @@ Require users to generate documentation bundles via an index crate (current) vs. If one would like to merge the documentation of several crates, we could continue to require users to provide an index crate, like [the fuchsia index](https://fuchsia-docs.firebaseapp.com/rust/rustdoc_index/). This serves as the target of the rustdoc invocation, and the landing page for the collected documentation. Supporting only this style of index would require the fewest changes. This is the mode described in the Guide-level explanation. -The proposition, to allow users of rustdoc the flexibility of not having to produce an index, is to allow rustdoc to be run in a mode where no target crate is provided. The source crates are provided to rustdoc, through a mechanism like `--extern`, rustdoc merges and writes the CCI, and copies the item and module links to the doc root. This would require more extensive changes, as rustdoc assumes that it is invoked with a target crate. This mode is somewhat analogous to the [example scraping mode](https://github.com/rust-lang/rfcs/blob/master/text/3123-rustdoc-scrape-examples.md). Having to create an index crate, that actively uses all of the crates in the environment, might prohibit the use of this feature in settings where users do not intend to produce an index, or where exhaustively listing all dependencies (to `--extern` them) is difficult. +The proposition, to allow users of rustdoc the flexibility of not having to produce an index, is to allow rustdoc to be run in a mode where no target crate is provided. It would generate rendered cross-crate information based only on what is provided through `--include-info-json`. The source crates are provided to rustdoc, through a mechanism like `--extern`, rustdoc merges and writes the CCI, and copies the item and module links to the doc root. This would require more extensive changes, as rustdoc assumes that it is invoked with a target crate. This mode is somewhat analogous to the [example scraping mode](https://github.com/rust-lang/rfcs/blob/master/text/3123-rustdoc-scrape-examples.md). Having to create an index crate, that actively uses all of the crates in the environment, might prohibit the use of this feature in settings where users do not intend to produce an index, or where exhaustively listing all dependencies (to `--extern` them) is difficult. ## Unconditionally generating the `doc.parts` files? Generate no extra files (current) vs. unconditionally creating `doc.parts` to enable more complex future CCI (should consider) The current version of rustdoc performs merging by [collecting JSON](https://github.com/rust-lang/rust/blob/c25ac9d6cc285e57e1176dc2da6848b9d0163810/src/librustdoc/html/render/write_shared.rs#L166) blobs from the contents of the already-rendered CCI. -This proposal proposes to continue reading from the rendered cross-crate information under the default `--read-rendered-cci=true`. It can also read `doc.parts` files, under `--fetch-parts`. However, there are several issues with reading from the rendered CCI that must be stated: +This proposal proposes to continue reading from the rendered cross-crate information under the default `--merge=auto`. It can also read `doc.parts` files, under `--include-info-json`. However, there are several issues with reading from the rendered CCI that must be stated: * Every rustdoc process outputs the CCI to the same doc root by default * It is difficult to extract the items in a diverse set of rendered HTML files. This is anticipating of the CCI to include HTML files that, for example, statically include type+trait implementations directly -* Reading exclusively from `doc.parts` is simpler than the existing `serde_json` dependency for extracting the blobs, as opposed to handwritten CCI-type specific parsing (current) +* Reading exclusively from `crate-info.json` is simpler than the existing `serde_json` dependency for extracting the blobs, as opposed to handwritten CCI-type specific parsing (current) * With this proposal, there will be duplicate logic to read from both `doc.parts` files and rendered CCI. +[@jsha proposes](https://github.com/rust-lang/rfcs/pull/3662#issuecomment-2184077829) unconditionally generating and reading from `crate-info.json`, with no appending to the rendered crate info. + ## Item links? Require users to pass `--extern-html-root-url` on all external dependencies (current), vs. add a new flag to facilitate missing docs links being generated across all user crates (should consider) Cross-crate links are an important consideration when merging documentation. rustdoc provides a flag, `--extern-html-root-url`, which can provide fine-grain control over the generated URL for documentation. We believe this is already sufficient for generating cross-crate links, and we will make no changes to the generation of item links. Example usage of this flag is in the Guide-level explanation. -A proposal is to provide more options to facilitate cross-crate links. For example, we could add a flag that implies `--extern-html-root-url` and `--fetch-parts` on all crates passed through `--extern`. It could be something like `--extern-html-root-url-all-user-crates`. User crates would be taken to mean the transitive dependencies of the current crate, excluding the standard library. This is complicated by things like . +A proposal is to provide more options to facilitate cross-crate links. For example, we could add a flag that implies `--extern-html-root-url` and `--include-info-json` on all crates passed through `--extern`. It could be something like `--extern-html-root-url-all-user-crates`. User crates would be taken to mean the transitive dependencies of the current crate, excluding the standard library. This is complicated by things like . ## Reuse existing option? -Create a new flag, `--write-rendered-cci` (proposed), vs. use existing option `no_emit_shared` +Create a new flag, `--merge` (proposed), vs. use existing option `no_emit_shared` There is a render option, `no_emit_shared`, which is used to conditionally generate the cross-crate information. It also controls the generation of static files, user CSS, etc. @@ -459,4 +435,4 @@ Another possibility is for `doc.parts` to be distributed on `docs.rs` along with A future possibility related to the index crate idea is to have an option for embedding user-specified HTML into the `--enable-index-page`'s HTML. -The changes in this proposal are intended to work with no changes to Cargo and docs.rs. However, there may be benefits to using `--write-rendered-cci=false` with Cargo, as it would remove the need for locking the output directory. More of the documentation process could happen in parallel, which may speed up execution time.. +The changes in this proposal are intended to work with no changes to Cargo and docs.rs. However, there may be benefits to using `--merge=write-only` with Cargo, as it would remove the need for locking the output directory. More of the documentation process could happen in parallel, which may speed up execution time. From 3326b10443e7583ea4452b8652f4b721081b6923 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Mon, 24 Jun 2024 18:34:58 +0000 Subject: [PATCH 11/28] typos --- text/0000-mergeable-rustdoc-cross-crate-info.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index 183c742a395..c7e4baeba3a 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -5,11 +5,11 @@ # Summary -Mergeable cross-crate information in rustdoc. Facilitates the generation of documentation indexes in workspaces with many crates by allowing each crate to write to an independent output directory. Final documentation is rendered with a lightweight merge step. Configurable with command-line flags, this proposal writes a `crate-info.json` directory to hold pre-merge cross-crate information. Currently, rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating rustdoc in build systems that make build actions independent. +Mergeable cross-crate information in rustdoc. Facilitates the generation of documentation indexes in workspaces with many crates by allowing each crate to write to an independent output directory. Final documentation is rendered with a lightweight merge step. Configurable with command-line flags, this proposal writes a `crate-info.json` file to hold pre-merge cross-crate information. Currently, rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating rustdoc in build systems that enforce the independence of build actions. # Motivation -The main goal of this proposal is to facilitate users producing a documentation bundle of every crate in a large environment. When a crate needs to be re-documented, only a relatively lightweight merge step will be needed to produce a complete documentation bundle. This proposal is to facilitate the creation and updating of these bundles. +The main goal of this proposal is to facilitate users producing a documentation bundle of every crate in a large environment. When a crate needs to be re-documented, only a relatively lightweight merge step will be needed to produce an updated documentation bundle. This proposal is to facilitate the creation and updating of these bundles. This proposal also targets documenting individual crates and their dependencies in non-cargo build systems. As will be explained, doc targets in non-cargo build systems often do not support cross-crate information. @@ -300,7 +300,7 @@ If this flag is not provided, no `crate-info.json` will be written. If this flag is provided, rustdoc will expect `//crate-info.json` to contain the parts for ``. It will append these parts to the ones it will render in the doc root (`--out-dir`). This info is not written to `crate-info.json`, as `crate-info.json` only holds the CCI parts for the current crate. -This flag is similar to `--extern-html-root-url` in that it only applies to externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided in `--extern-html-root-url` never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--include-info-json` tells rustdoc where to search for the `crate-info.json` directory at documentation-time. It must not be a hyperlink. +This flag is similar to `--extern-html-root-url` in that it only applies to externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided in `--extern-html-root-url` never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--include-info-json` tells rustdoc where to search for the `crate-info.json` directory at documentation-time. It must not be a URL. In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc.parts/s`, so rustdoc is called with `--include-info-json s=s/target/doc.parts`. @@ -319,11 +319,13 @@ When `read_rendered_cci` is active, rustdoc will look in the `--out-dir` for ren ## Merge step -This step is provided with a list of crates. It merges their documentation. This step involves copying parts (individual item, module documentation) from each of the provided crates. It merges the parts, renders, and writes the CCI to the doc root. +This proposal is capable of addressing two primary use cases. It allows developers to enable CCI in these scenarios: +* Documenting a crate and its transitive dependencies in parallel in build systems that require build actions to be independent +* Producing a documentation index of a large number of crates, in such a way that if one crate is updated, only the updated crates and an index have to be redocumented. This scenario is demonstrated in the Guide-level explanation. -In short, without any specific merge step, merging the documentation for multiple crates is performed as shown in the Guide-level explanation. +CCI is not automatically enabled in either situation. A combination of the `--include-info-json`, `--merge`, and `--write-info-json` flags are needed to produce this behavior. This RFC provides a minimal set of tools that allow developers of build systems, like Bazel and Buck2, to create rules for these scenarios. -Discussion of a dedicated merge step is described in the Unresolved questions (Index crate). +Discussion of whether additional features should be included to facilitate this merge step can be found in Unresolved questions (Index crate). ## Compatibility From cbd62bb6e530d14f89931bd06126485b35bd634b Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Tue, 25 Jun 2024 00:01:46 +0000 Subject: [PATCH 12/28] nits --- text/0000-mergeable-rustdoc-cross-crate-info.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index c7e4baeba3a..a38aac7b2c8 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -256,9 +256,9 @@ $ tree . -a Currently, cross-crate information is written during the invocation of the `write_shared` function in [write_shared.rs](https://github.com/rust-lang/rust/blob/04ab7b2be0db3e6787f5303285c6b2ee6279868d/src/librustdoc/html/render/write_shared.rs#L47). This proposal does not add any new CCI or change their contents (modulo sorting order, whitespace). -The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of a the CCI. +The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of the CCI. -## New subdirectory: `//crate-info.json` +## New file: `//crate-info.json` The `//crate-info.json` file contains the unmerged contents of a single crates' version of their corresponding CCI. Typically, `` is selected as `./target/doc.parts`. This file is written if the flag `--write-info-json=` is provided. @@ -337,6 +337,7 @@ The implementation of the RFC itself is designed to produce only minimal changes * Change the sorting order of trait implementations, type implementations, and other cross-crate info in the HTML output of rustdoc * Add a comment on the last line of generated HTML pages, to store metadata relevant to appending items to them * Refactor the JavaScript contents of cross-crate information files, in ways that do not change their overall behavior. If the JavaScript fragment declared an array called `ALL_CRATES` with certain contents, it will continue to do so. + Changes this minimal are intended to avoid breaking tools that use the output of rustdoc, like Cargo, docs.rs, and rustdoc's JavaScript frontend, in the near-term. Going forward, rustdoc will not make formal guarantees about the content of cross-crate info files. # Drawbacks From eba7fc4753415f1d92aa0065f1994fe39663d2e9 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Fri, 28 Jun 2024 00:16:27 +0000 Subject: [PATCH 13/28] --include-info-json and --write-info-json take path directly to crate-info.json --- ...0000-mergeable-rustdoc-cross-crate-info.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index a38aac7b2c8..498c9b3e4ed 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -48,17 +48,17 @@ rustc --crate-name=s --crate-type=lib --edition=2021 --emit=metadata --out-dir=s rustc --crate-name=i --crate-type=lib --edition=2021 --emit=metadata --out-dir=i/target --extern s=s/target/libs.rmeta --extern t=t/target/libt.rmeta -L t/target i/src/lib.rs ``` -Document `s` and `t` independently, providing `--merge=none` and `--write-info-json=/target/doc.parts` +Document `s` and `t` independently, providing `--merge=none` and `--write-info-json` ```shell -rustdoc -Z unstable-options --crate-name=t --crate-type=lib --edition=2021 --out-dir=t/target/doc --extern-html-root-url t=$(MERGED) --merge=none --write-info-json=t/target/doc.parts t/src/lib.rs -rustdoc -Z unstable-options --crate-name=s --crate-type=lib --edition=2021 --out-dir=s/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --merge=none --write-info-json=t/target/doc.parts --extern t=t/target/libt.rmeta s/src/lib.rs +rustdoc -Z unstable-options --crate-name=t --crate-type=lib --edition=2021 --out-dir=t/target/doc --extern-html-root-url t=$(MERGED) --merge=none --write-info-json=t/target/doc.parts/t/crate-info.json t/src/lib.rs +rustdoc -Z unstable-options --crate-name=s --crate-type=lib --edition=2021 --out-dir=s/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --merge=none --write-info-json=s/target/doc.parts/s/crate-info.json --extern t=t/target/libt.rmeta s/src/lib.rs ``` Link everything with a final invocation of rustdoc on `i`. We will provide `--merge=write-only` and `--include-info-json=`. See the Reference-level explanation about these flags. ```shell -rustdoc -Z unstable-options --crate-name=i --crate-type=lib --edition=2021 --enable-index-page --out-dir=i/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --extern-html-root-url i=$(MERGED) --merge=write-only --include-info-json t=t/target/doc.parts --include-info-json s=s/target/doc.parts --extern t=t/target/libt.rmeta --extern s=s/target/libs.rmeta -L t/target i/src/lib.rs +rustdoc -Z unstable-options --crate-name=i --crate-type=lib --edition=2021 --enable-index-page --out-dir=i/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --extern-html-root-url i=$(MERGED) --merge=write-only --include-info-json=t/target/doc.parts/t/crate-info.json --include-info-json=s/target/doc.parts/s/crate-info.json --extern t=t/target/libt.rmeta --extern s=s/target/libs.rmeta -L t/target i/src/lib.rs ``` Merge the docs with `cp`. This can be avoided if `--out-dir=$(MERGED)` is used for all of the rustdoc calls. We copy here to illustrate that documenting `s` is independent of documenting `t`, and could happen on separate machines. @@ -258,9 +258,9 @@ Currently, cross-crate information is written during the invocation of the `writ The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of the CCI. -## New file: `//crate-info.json` +## New file: `crate-info.json` -The `//crate-info.json` file contains the unmerged contents of a single crates' version of their corresponding CCI. Typically, `` is selected as `./target/doc.parts`. This file is written if the flag `--write-info-json=` is provided. +The `crate-info.json` file contains the unmerged contents of a single crates' version of their corresponding CCI. This file is written if the flag `--write-info-json=path/to/crate-info.json` is provided. `crate-info.json` is a JSON object with several key-value pairs. Every value is a JSON array. Every element of the JSON array is a two-element array: a destination filename relative to the doc root, and the representation of the part. The representation of the part depends on the type of CCI that it describes. @@ -288,21 +288,21 @@ This part is a two element array with the crate name and the JSON representation This part is a two element array with the crate name and the JSON representation of a trait implementation. -## New flag: `--write-info-json=` +## New flag: `--write-info-json=path/to/crate-info.json` -When this flag is provided, the unmerged parts for the current crate will be written to `//crate-info.json`. A typical `` is `./target/doc.parts`. +When this flag is provided, the unmerged parts for the current crate will be written to `path/to/crate-info.json`. A typical `` is `./target/doc.parts//crate-info.json`. Crates `--include-info-json`ed will not appear in `crate-info.json`, as `crate-info.json` only includes the CCI parts for the current crate. If this flag is not provided, no `crate-info.json` will be written. -## New flag: `--include-info-json =` +## New flag: `--include-info-json=path/to/crate-info.json` -If this flag is provided, rustdoc will expect `//crate-info.json` to contain the parts for ``. It will append these parts to the ones it will render in the doc root (`--out-dir`). This info is not written to `crate-info.json`, as `crate-info.json` only holds the CCI parts for the current crate. +If this flag is provided, rustdoc will expect `path/to/crate-info.json` to be the `crate-info.json` file containing the parts for a crate. It will append these parts to the ones it will render in the doc root (`--out-dir`). The info that's included is not written to `crate-info.json`, as `crate-info.json` only holds the CCI parts for the current crate. -This flag is similar to `--extern-html-root-url` in that it only applies to externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided in `--extern-html-root-url` never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--include-info-json` tells rustdoc where to search for the `crate-info.json` directory at documentation-time. It must not be a URL. +This flag is similar to `--extern-html-root-url` in that it only needs to be provided for externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided in `--extern-html-root-url` never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--include-info-json` tells rustdoc where to search for the `crate-info.json` directory at documentation-time. It must not be a URL. -In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc.parts/s`, so rustdoc is called with `--include-info-json s=s/target/doc.parts`. +In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc.parts/s`, so rustdoc is called with `--include-info-json=s/target/doc.parts/s/crate-info.json`. ## New flag: `--merge=auto|none|write-only` @@ -314,7 +314,7 @@ When `read_rendered_cci` is active, rustdoc will look in the `--out-dir` for ren * `--merge=auto` (`read_rendered_cci && write_rendered_cci`) is the default, and reflects the current behavior of rustdoc. * `--merge=none` (`!read_rendered_cci && !write_rendered_cci`) means that rustdoc will ignore the cross-crate files in the doc root. Only generate item docs. -* `--merge=write-only` (`!read_rendered_cci && write_rendered_cci`) only outputs item docs that are based on the current crate and `--include-info-json`'ed crates. +* `--merge=write-only` (`!read_rendered_cci && write_rendered_cci`) outputs crate info based only on the current crate and `--include-info-json`'ed crates. * A (`read_rendered_cci && !write_rendered_cci`) mode would be useless, since the data that is read would be ignored and not written. ## Merge step @@ -397,7 +397,7 @@ The proposition, to allow users of rustdoc the flexibility of not having to prod Generate no extra files (current) vs. unconditionally creating `doc.parts` to enable more complex future CCI (should consider) The current version of rustdoc performs merging by [collecting JSON](https://github.com/rust-lang/rust/blob/c25ac9d6cc285e57e1176dc2da6848b9d0163810/src/librustdoc/html/render/write_shared.rs#L166) blobs from the contents of the already-rendered CCI. -This proposal proposes to continue reading from the rendered cross-crate information under the default `--merge=auto`. It can also read `doc.parts` files, under `--include-info-json`. However, there are several issues with reading from the rendered CCI that must be stated: +This proposal proposes to continue reading from the rendered cross-crate information under the default `--merge=auto`. It can also read `crate-info.json` files, under `--include-info-json`. However, there are several issues with reading from the rendered CCI that must be stated: * Every rustdoc process outputs the CCI to the same doc root by default * It is difficult to extract the items in a diverse set of rendered HTML files. This is anticipating of the CCI to include HTML files that, for example, statically include type+trait implementations directly * Reading exclusively from `crate-info.json` is simpler than the existing `serde_json` dependency for extracting the blobs, as opposed to handwritten CCI-type specific parsing (current) From 7b82ab9f786e71297dc81aef0cab88966ba31ca1 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Tue, 16 Jul 2024 16:18:57 +0000 Subject: [PATCH 14/28] typos, --mode=auto -> --mode=read-write --- .../0000-mergeable-rustdoc-cross-crate-info.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index 498c9b3e4ed..2744e802c6f 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -25,7 +25,7 @@ These considerations motivate adding an option for outputting partial CCI (parts # Guide-level explanation -In this example, there is a crate `t` which defines a trait `T`, and a crate `s` which defines a struct `S` that implements `T`. Our goal in this demo is for `S` to appear as in implementer in `T`'s docs, even if `s` and `t` are documented independently. This guide will be assuming that we want a crate `i` that serves as our documentation index. See the Unresolved questions section for ideas that do not require an index crate. +In this example, there is a crate `t` which defines a trait `T`, and a crate `s` which defines a struct `S` that implements `T`. Our goal in this demo is for `S` to appear as an implementer in `T`'s docs, even if `s` and `t` are documented independently. This guide will be assuming that we want a crate `i` that serves as our documentation index. See the Unresolved questions section for ideas that do not require an index crate. ```shell mkdir -p t/src s/src i/src merged/doc @@ -245,7 +245,7 @@ $ tree . -a │ └── lib.rs.html ├── doc.parts │ └── t -│ │ └── crate-info.json + │ └── crate-info.json └── libs.rmeta @@ -304,7 +304,7 @@ This flag is similar to `--extern-html-root-url` in that it only needs to be pro In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc.parts/s`, so rustdoc is called with `--include-info-json=s/target/doc.parts/s/crate-info.json`. -## New flag: `--merge=auto|none|write-only` +## New flag: `--merge=read-write|none|write-only` This flag controls two internal paramaters: `read_rendered_cci`, and `write_rendered_cci`. @@ -312,7 +312,7 @@ When `write_rendered_cci` is active, rustdoc will output the rendered parts to t When `read_rendered_cci` is active, rustdoc will look in the `--out-dir` for rendered cross-crate info files. These files will be used as the base. Any new parts that rustdoc generates with its current invocation and any parts fetched with `include-info-json` will be appended to these base files. When it is disabled, the cross-crate info files start empty and are populated with the current crate's info and any crates fetched with `--include-info-json`. -* `--merge=auto` (`read_rendered_cci && write_rendered_cci`) is the default, and reflects the current behavior of rustdoc. +* `--merge=read-write` (`read_rendered_cci && write_rendered_cci`) is the default, and reflects the current behavior of rustdoc. * `--merge=none` (`!read_rendered_cci && !write_rendered_cci`) means that rustdoc will ignore the cross-crate files in the doc root. Only generate item docs. * `--merge=write-only` (`!read_rendered_cci && write_rendered_cci`) outputs crate info based only on the current crate and `--include-info-json`'ed crates. * A (`read_rendered_cci && !write_rendered_cci`) mode would be useless, since the data that is read would be ignored and not written. @@ -331,7 +331,7 @@ Discussion of whether additional features should be included to facilitate this This RFC does not alter previous compatibility guarantees made about the output of rustdoc. In particular it does not stabilize the presence of the rendered cross-crate information files, their content, or the HTML generated by rustdoc. -In the same way that the [rustdoc HTML output is unstable](https://rust-lang.github.io/rfcs/2963-rustdoc-json.html#:~:text=The%20HTML%20output%20of%20rustdoc,into%20a%20different%20format%20impractical), the contents of the `crate-info.json` will be considered unstable. Between versions of rustdoc, breaking changes to the content of `crate-info.json` should be expected. Only the presence of a `crate-info.json` file is promised, under `--write-info-json`. Merging cross-crate information generated by disparate versions of rustdoc is not supported. +In the same way that the [rustdoc HTML output is unstable](https://rust-lang.github.io/rfcs/2963-rustdoc-json.html#:~:text=The%20HTML%20output%20of%20rustdoc,into%20a%20different%20format%20impractical), the content of `crate-info.json` will be considered unstable. Between versions of rustdoc, breaking changes to the content of `crate-info.json` should be expected. Only the presence of a `crate-info.json` file is promised, under `--write-info-json`. Merging cross-crate information generated by disparate versions of rustdoc is not supported. The implementation of the RFC itself is designed to produce only minimal changes to cross-crate info files and the HTML output of rustdoc. Exhaustively, the implementation is allowed to * Change the sorting order of trait implementations, type implementations, and other cross-crate info in the HTML output of rustdoc @@ -386,7 +386,7 @@ Currently, the Fuchsia project runs rustdoc on all of their crates to generate a ## Index crate? -Require users to generate documentation bundles via an index crate (current) vs. creating a new mode to allow rustdoc to run without a target crate (proposed) +Require users to generate documentation bundles via an index crate (current) vs. creating a new mode to allow rustdoc to run without a target crate (proposed). If one would like to merge the documentation of several crates, we could continue to require users to provide an index crate, like [the fuchsia index](https://fuchsia-docs.firebaseapp.com/rust/rustdoc_index/). This serves as the target of the rustdoc invocation, and the landing page for the collected documentation. Supporting only this style of index would require the fewest changes. This is the mode described in the Guide-level explanation. @@ -394,10 +394,10 @@ The proposition, to allow users of rustdoc the flexibility of not having to prod ## Unconditionally generating the `doc.parts` files? -Generate no extra files (current) vs. unconditionally creating `doc.parts` to enable more complex future CCI (should consider) +Generate no extra files (current) vs. unconditionally creating `doc.parts` to enable more complex future CCI (should consider). The current version of rustdoc performs merging by [collecting JSON](https://github.com/rust-lang/rust/blob/c25ac9d6cc285e57e1176dc2da6848b9d0163810/src/librustdoc/html/render/write_shared.rs#L166) blobs from the contents of the already-rendered CCI. -This proposal proposes to continue reading from the rendered cross-crate information under the default `--merge=auto`. It can also read `crate-info.json` files, under `--include-info-json`. However, there are several issues with reading from the rendered CCI that must be stated: +This proposal proposes to continue reading from the rendered cross-crate information under the default `--merge=read-write`. It can also read `crate-info.json` files, under `--include-info-json`. However, there are several issues with reading from the rendered CCI that must be stated: * Every rustdoc process outputs the CCI to the same doc root by default * It is difficult to extract the items in a diverse set of rendered HTML files. This is anticipating of the CCI to include HTML files that, for example, statically include type+trait implementations directly * Reading exclusively from `crate-info.json` is simpler than the existing `serde_json` dependency for extracting the blobs, as opposed to handwritten CCI-type specific parsing (current) @@ -415,7 +415,7 @@ A proposal is to provide more options to facilitate cross-crate links. For examp ## Reuse existing option? -Create a new flag, `--merge` (proposed), vs. use existing option `no_emit_shared` +Create a new flag, `--merge` (proposed), vs. use existing option `no_emit_shared`. There is a render option, `no_emit_shared`, which is used to conditionally generate the cross-crate information. It also controls the generation of static files, user CSS, etc. From 63107f90c2a477b340c8084cb0db74a731bcbb12 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Tue, 16 Jul 2024 18:17:30 +0000 Subject: [PATCH 15/28] add versioning, --include-rendered-docs flag --- ...0000-mergeable-rustdoc-cross-crate-info.md | 259 +++++++++--------- 1 file changed, 132 insertions(+), 127 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index 2744e802c6f..d8c5e7f3c85 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -45,26 +45,57 @@ Compile the crates. ```shell rustc --crate-name=t --crate-type=lib --edition=2021 --emit=metadata --out-dir=t/target t/src/lib.rs rustc --crate-name=s --crate-type=lib --edition=2021 --emit=metadata --out-dir=s/target --extern t=t/target/libt.rmeta s/src/lib.rs -rustc --crate-name=i --crate-type=lib --edition=2021 --emit=metadata --out-dir=i/target --extern s=s/target/libs.rmeta --extern t=t/target/libt.rmeta -L t/target i/src/lib.rs ``` -Document `s` and `t` independently, providing `--merge=none` and `--write-info-json` +Document `s` and `t` independently, providing `--merge=none`, `--write-info-json`. ```shell -rustdoc -Z unstable-options --crate-name=t --crate-type=lib --edition=2021 --out-dir=t/target/doc --extern-html-root-url t=$(MERGED) --merge=none --write-info-json=t/target/doc.parts/t/crate-info.json t/src/lib.rs -rustdoc -Z unstable-options --crate-name=s --crate-type=lib --edition=2021 --out-dir=s/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --merge=none --write-info-json=s/target/doc.parts/s/crate-info.json --extern t=t/target/libt.rmeta s/src/lib.rs +rustdoc \ + -Z unstable-options \ + --crate-name=t \ + --crate-type=lib \ + --edition=2021 \ + --out-dir=t/target/doc \ + --extern-html-root-url t=$MERGED \ + --merge=none \ + --write-info-json=t/target/doc.parts/t/crate-info.json \ + t/src/lib.rs +rustdoc \ + -Z unstable-options \ + --crate-name=s \ + --crate-type=lib \ + --edition=2021 \ + --out-dir=s/target/doc \ + --extern-html-root-url s=$MERGED \ + --extern-html-root-url t=$MERGED \ + --merge=none \ + --write-info-json=s/target/doc.parts/s/crate-info.json \ + --extern t=t/target/libt.rmeta \ + s/src/lib.rs ``` -Link everything with a final invocation of rustdoc on `i`. We will provide `--merge=write-only` and `--include-info-json=`. See the Reference-level explanation about these flags. +Link everything with a final invocation of rustdoc on `i`. We will provide `--merge=write-only`, `--include-info-json`, and `--include-rendered-docs`. See the Reference-level explanation about these flags. ```shell -rustdoc -Z unstable-options --crate-name=i --crate-type=lib --edition=2021 --enable-index-page --out-dir=i/target/doc --extern-html-root-url s=$(MERGED) --extern-html-root-url t=$(MERGED) --extern-html-root-url i=$(MERGED) --merge=write-only --include-info-json=t/target/doc.parts/t/crate-info.json --include-info-json=s/target/doc.parts/s/crate-info.json --extern t=t/target/libt.rmeta --extern s=s/target/libs.rmeta -L t/target i/src/lib.rs -``` - -Merge the docs with `cp`. This can be avoided if `--out-dir=$(MERGED)` is used for all of the rustdoc calls. We copy here to illustrate that documenting `s` is independent of documenting `t`, and could happen on separate machines. - -```shell -cp -r s/target/doc/* t/target/doc/* i/target/doc/* merged/doc +rustdoc \ + -Z unstable-options \ + --crate-name=i \ + --crate-type=lib \ + --edition=2021 \ + --enable-index-page \ + --out-dir=i/target/doc \ + --extern-html-root-url s=$MERGED \ + --extern-html-root-url t=$MERGED \ + --extern-html-root-url i=$MERGED \ + --merge=write-only \ + --include-info-json=t/target/doc.parts/t/crate-info.json \ + --include-info-json=s/target/doc.parts/s/crate-info.json \ + --extern t=t/target/libt.rmeta \ + --extern s=s/target/libs.rmeta \ + --include-rendered-docs=t/target/doc/t \ + --include-rendered-docs=s/target/doc/s \ + -L t/target \ + i/src/lib.rs ``` Browse `merged/doc/index.html` with cross-crate information. @@ -81,78 +112,47 @@ $ tree . -a │ ├── src │ │ └── lib.rs │ └── target -│ ├── doc -│ │ ├── crates.js -│ │ ├── help.html -│ │ ├── i -│ │ │ ├── all.html -│ │ │ ├── fn.add.html -│ │ │ ├── index.html -│ │ │ └── sidebar-items.js -│ │ ├── index.html -│ │ ├── .lock -│ │ ├── search.desc -│ │ │ ├── i -│ │ │ │ └── i-desc-0-.js -│ │ │ ├── s -│ │ │ │ └── s-desc-0-.js -│ │ │ └── t -│ │ │ └── t-desc-0-.js -│ │ ├── settings.html -│ │ ├── src -│ │ │ └── i -│ │ │ └── lib.rs.html -│ │ ├── src-files.js -│ │ ├── static.files -│ │ │ ├── COPYRIGHT-23e9bde6c69aea69.txt -│ │ │ ├── favicon-2c020d218678b618.svg -│ │ │ ├── favicon-32x32-422f7d1d52889060.png -│ │ │ ├── FiraSans-LICENSE-db4b642586e02d97.txt -│ │ │ ├── FiraSans-Medium-8f9a781e4970d388.woff2 -│ │ │ ├── FiraSans-Regular-018c141bf0843ffd.woff2 -│ │ │ ├── LICENSE-APACHE-b91fa81cba47b86a.txt -│ │ │ ├── LICENSE-MIT-65090b722b3f6c56.txt -│ │ │ ├── main-20a3ad099b048cf2.js -│ │ │ ├── NanumBarunGothic-0f09457c7a19b7c6.ttf.woff2 -│ │ │ ├── NanumBarunGothic-LICENSE-18c5adf4b52b4041.txt -│ │ │ ├── normalize-76eba96aa4d2e634.css -│ │ │ ├── noscript-df360f571f6edeae.css -│ │ │ ├── rustdoc-dd39b87e5fcfba68.css -│ │ │ ├── rust-logo-151179464ae7ed46.svg -│ │ │ ├── scrape-examples-ef1e698c1d417c0c.js -│ │ │ ├── search-0fe7219eb170c82e.js -│ │ │ ├── settings-4313503d2e1961c2.js -│ │ │ ├── SourceCodePro-It-1cc31594bf4f1f79.ttf.woff2 -│ │ │ ├── SourceCodePro-LICENSE-d180d465a756484a.txt -│ │ │ ├── SourceCodePro-Regular-562dcc5011b6de7d.ttf.woff2 -│ │ │ ├── SourceCodePro-Semibold-d899c5a5c4aeb14a.ttf.woff2 -│ │ │ ├── SourceSerif4-Bold-a2c9cd1067f8b328.ttf.woff2 -│ │ │ ├── SourceSerif4-It-acdfaf1a8af734b1.ttf.woff2 -│ │ │ ├── SourceSerif4-LICENSE-3bb119e13b1258b7.md -│ │ │ ├── SourceSerif4-Regular-46f98efaafac5295.ttf.woff2 -│ │ │ ├── src-script-e66d777a5a92e9b2.js -│ │ │ └── storage-118b08c4c78b968e.js -│ │ └── trait.impl -│ │ ├── core -│ │ │ ├── marker -│ │ │ │ ├── trait.Freeze.js -│ │ │ │ ├── trait.Send.js -│ │ │ │ ├── trait.Sync.js -│ │ │ │ └── trait.Unpin.js -│ │ │ └── panic -│ │ │ └── unwind_safe -│ │ │ ├── trait.RefUnwindSafe.js -│ │ │ └── trait.UnwindSafe.js -│ │ └── t -│ │ └── trait.T.js -│ └── libi.rmeta +│ └── doc +│ ├── crates.js +│ ├── help.html +│ ├── i +│ │ ├── all.html +│ │ ├── index.html +│ │ └── sidebar-items.js +│ ├── index.html +│ ├── .lock +│ ├── search.desc +│ │ └── i +│ │ └── i-desc-0-.js +│ ├── search-index.js +│ ├── settings.html +│ ├── src +│ │ └── i +│ │ └── lib.rs.html +│ ├── src-files.js +│ ├── static.files +│ │ ├── COPYRIGHT-23e9bde6c69aea69.txt +│ │ ├── favicon-2c020d218678b618.svg +│ │ └── +│ └── trait.impl +│ ├── core +│ │ ├── marker +│ │ │ ├── trait.Freeze.js +│ │ │ ├── trait.Send.js +│ │ │ ├── trait.Sync.js +│ │ │ └── trait.Unpin.js +│ │ └── panic +│ │ └── unwind_safe +│ │ ├── trait.RefUnwindSafe.js +│ │ └── trait.UnwindSafe.js +│ └── t +│ └── trait.T.js ├── merged │ └── doc │ ├── crates.js │ ├── help.html │ ├── i │ │ ├── all.html -│ │ ├── fn.add.html │ │ ├── index.html │ │ └── sidebar-items.js │ ├── index.html @@ -162,8 +162,13 @@ $ tree . -a │ │ ├── sidebar-items.js │ │ └── struct.S.html │ ├── search.desc -│ │ └── i -│ │ └── i-desc-0-.js +│ │ ├── i +│ │ │ └── i-desc-0-.js +│ │ ├── s +│ │ │ └── s-desc-0-.js +│ │ └── t +│ │ └── t-desc-0-.js +│ ├── search-index.js │ ├── settings.html │ ├── src │ │ ├── i @@ -174,39 +179,27 @@ $ tree . -a │ │ └── lib.rs.html │ ├── src-files.js │ ├── static.files -│ │ ├── COPYRIGHT-23e9bde6c69aea69.txt -│ │ ├── favicon-2c020d218678b618.svg -│ │ ├── favicon-32x32-422f7d1d52889060.png -│ │ ├── FiraSans-LICENSE-db4b642586e02d97.txt -│ │ ├── FiraSans-Medium-8f9a781e4970d388.woff2 -│ │ ├── FiraSans-Regular-018c141bf0843ffd.woff2 -│ │ ├── LICENSE-APACHE-b91fa81cba47b86a.txt -│ │ ├── LICENSE-MIT-65090b722b3f6c56.txt -│ │ ├── main-20a3ad099b048cf2.js -│ │ ├── NanumBarunGothic-0f09457c7a19b7c6.ttf.woff2 -│ │ ├── NanumBarunGothic-LICENSE-18c5adf4b52b4041.txt -│ │ ├── normalize-76eba96aa4d2e634.css -│ │ ├── noscript-df360f571f6edeae.css -│ │ ├── rustdoc-dd39b87e5fcfba68.css -│ │ ├── rust-logo-151179464ae7ed46.svg -│ │ ├── scrape-examples-ef1e698c1d417c0c.js -│ │ ├── search-0fe7219eb170c82e.js -│ │ ├── settings-4313503d2e1961c2.js -│ │ ├── SourceCodePro-It-1cc31594bf4f1f79.ttf.woff2 -│ │ ├── SourceCodePro-LICENSE-d180d465a756484a.txt -│ │ ├── SourceCodePro-Regular-562dcc5011b6de7d.ttf.woff2 -│ │ ├── SourceCodePro-Semibold-d899c5a5c4aeb14a.ttf.woff2 -│ │ ├── SourceSerif4-Bold-a2c9cd1067f8b328.ttf.woff2 -│ │ ├── SourceSerif4-It-acdfaf1a8af734b1.ttf.woff2 -│ │ ├── SourceSerif4-LICENSE-3bb119e13b1258b7.md -│ │ ├── SourceSerif4-Regular-46f98efaafac5295.ttf.woff2 -│ │ ├── src-script-e66d777a5a92e9b2.js -│ │ └── storage-118b08c4c78b968e.js -│ └── t -│ ├── all.html -│ ├── index.html -│ ├── sidebar-items.js -│ └── trait.T.html +│ │ ├── COPYRIGHT-23e9bde6c69aea69.txt +│ │ ├── favicon-2c020d218678b618.svg +│ │ └── +│ ├── t +│ │ ├── all.html +│ │ ├── index.html +│ │ ├── sidebar-items.js +│ │ └── trait.T.html +│ └── trait.impl +│ ├── core +│ │ ├── marker +│ │ │ ├── trait.Freeze.js +│ │ │ ├── trait.Send.js +│ │ │ ├── trait.Sync.js +│ │ │ └── trait.Unpin.js +│ │ └── panic +│ │ └── unwind_safe +│ │ ├── trait.RefUnwindSafe.js +│ │ └── trait.UnwindSafe.js +│ └── t +│ └── trait.T.js ├── s │ ├── src │ │ └── lib.rs @@ -219,6 +212,9 @@ $ tree . -a │ │ │ ├── index.html │ │ │ ├── sidebar-items.js │ │ │ └── struct.S.html +│ │ ├── search.desc +│ │ │ └── s +│ │ │ └── s-desc-0-.js │ │ ├── settings.html │ │ └── src │ │ └── s @@ -234,19 +230,22 @@ $ tree . -a ├── doc │ ├── help.html │ ├── .lock - │ ├── t - │ │ ├── all.html - │ │ ├── index.html - │ │ ├── sidebar-items.js - │ │ └── trait.T.html + │ ├── search.desc + │ │ └── t + │ │ └── t-desc-0-.js │ ├── settings.html - │ └── src - │ └── t - │ └── lib.rs.html + │ ├── src + │ │ └── t + │ │ └── lib.rs.html + │ └── t + │ ├── all.html + │ ├── index.html + │ ├── sidebar-items.js + │ └── trait.T.html ├── doc.parts │ └── t │ └── crate-info.json - └── libs.rmeta + └── libt.rmeta @@ -254,9 +253,7 @@ $ tree . -a # Reference-level explanation -Currently, cross-crate information is written during the invocation of the `write_shared` function in [write_shared.rs](https://github.com/rust-lang/rust/blob/04ab7b2be0db3e6787f5303285c6b2ee6279868d/src/librustdoc/html/render/write_shared.rs#L47). This proposal does not add any new CCI or change their contents (modulo sorting order, whitespace). - -The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of the CCI. +The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of the CCI. This proposal does not add any new CCI or change their contents (modulo sorting order, whitespace). ## New file: `crate-info.json` @@ -264,6 +261,10 @@ The `crate-info.json` file contains the unmerged contents of a single crates' ve `crate-info.json` is a JSON object with several key-value pairs. Every value is a JSON array. Every element of the JSON array is a two-element array: a destination filename relative to the doc root, and the representation of the part. The representation of the part depends on the type of CCI that it describes. +* key: `version` + +The value will be a string encoding of a version number. If rustdoc is provided a `--include-info-json` flag that points to a `crate-info.json` file with a version number that it cannot support, it will fail. Rustdoc only guarantees that it will accept `crate-info.json` files written by the same version of rustdoc. There is no forward or backward compatibility. A single-digit, incremental version number is chosen for convenience, as `crate-info.json` does not make distinctions between breaking and non-breaking changes. Rustdoc is the only explicitly supported consumer of `crate-info.json`. + * key: `src-files-js`, for `doc/src-files.js` This part is the JSON representation of the source index that is later stored in a `srcIndex` global variable. @@ -296,7 +297,7 @@ Crates `--include-info-json`ed will not appear in `crate-info.json`, as `crate-i If this flag is not provided, no `crate-info.json` will be written. -## New flag: `--include-info-json=path/to/crate-info.json` +## New flag: `--include-info-json=` If this flag is provided, rustdoc will expect `path/to/crate-info.json` to be the `crate-info.json` file containing the parts for a crate. It will append these parts to the ones it will render in the doc root (`--out-dir`). The info that's included is not written to `crate-info.json`, as `crate-info.json` only holds the CCI parts for the current crate. @@ -304,6 +305,10 @@ This flag is similar to `--extern-html-root-url` in that it only needs to be pro In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc.parts/s`, so rustdoc is called with `--include-info-json=s/target/doc.parts/s/crate-info.json`. +## New flag: `--include-rendered-docs=` + +Rustdoc will assume that `` was used as the `--out-dir` for ``. This documentation will be copied into the directory specified by `--out-dir`. Rustdoc will effectively run `cp -r `. + ## New flag: `--merge=read-write|none|write-only` This flag controls two internal paramaters: `read_rendered_cci`, and `write_rendered_cci`. @@ -331,7 +336,7 @@ Discussion of whether additional features should be included to facilitate this This RFC does not alter previous compatibility guarantees made about the output of rustdoc. In particular it does not stabilize the presence of the rendered cross-crate information files, their content, or the HTML generated by rustdoc. -In the same way that the [rustdoc HTML output is unstable](https://rust-lang.github.io/rfcs/2963-rustdoc-json.html#:~:text=The%20HTML%20output%20of%20rustdoc,into%20a%20different%20format%20impractical), the content of `crate-info.json` will be considered unstable. Between versions of rustdoc, breaking changes to the content of `crate-info.json` should be expected. Only the presence of a `crate-info.json` file is promised, under `--write-info-json`. Merging cross-crate information generated by disparate versions of rustdoc is not supported. +In the same way that the [rustdoc HTML output is unstable](https://rust-lang.github.io/rfcs/2963-rustdoc-json.html#:~:text=The%20HTML%20output%20of%20rustdoc,into%20a%20different%20format%20impractical), the content of `crate-info.json` will be considered unstable. Between versions of rustdoc, breaking changes to the content of `crate-info.json` should be expected. Only the presence of a `crate-info.json` file is promised, under `--write-info-json`. Merging cross-crate information generated by disparate versions of rustdoc is not supported. To detect whether `crate-info.json` is compatible, rustdoc includes a version number in these files (see New file: `crate-info.json`). The implementation of the RFC itself is designed to produce only minimal changes to cross-crate info files and the HTML output of rustdoc. Exhaustively, the implementation is allowed to * Change the sorting order of trait implementations, type implementations, and other cross-crate info in the HTML output of rustdoc From 3d0dd8c4d879fbac3ce24b0fb71fee2d48e9adbf Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Wed, 17 Jul 2024 17:37:43 +0000 Subject: [PATCH 16/28] version number is prefixed with V --- text/0000-mergeable-rustdoc-cross-crate-info.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index d8c5e7f3c85..a192b49706f 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -263,7 +263,7 @@ The `crate-info.json` file contains the unmerged contents of a single crates' ve * key: `version` -The value will be a string encoding of a version number. If rustdoc is provided a `--include-info-json` flag that points to a `crate-info.json` file with a version number that it cannot support, it will fail. Rustdoc only guarantees that it will accept `crate-info.json` files written by the same version of rustdoc. There is no forward or backward compatibility. A single-digit, incremental version number is chosen for convenience, as `crate-info.json` does not make distinctions between breaking and non-breaking changes. Rustdoc is the only explicitly supported consumer of `crate-info.json`. +The value will be a string encoding of a version number. If rustdoc is provided a `--include-info-json` flag that points to a `crate-info.json` file with a version number that it cannot support, it will fail. Rustdoc only guarantees that it will accept `crate-info.json` files written by the same version of rustdoc. There is no forward or backward compatibility. A single number prefixed with capital V is chosen for convenience, as `crate-info.json` does not make distinctions between breaking and non-breaking changes. Rustdoc is the only explicitly supported consumer of `crate-info.json`. * key: `src-files-js`, for `doc/src-files.js` From 0ddbcf650887e9235f64504b72431f39f841195d Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Thu, 18 Jul 2024 20:08:59 +0000 Subject: [PATCH 17/28] remove detail of crate-info, suggested workflow --- ...0000-mergeable-rustdoc-cross-crate-info.md | 47 ++++++++----------- 1 file changed, 20 insertions(+), 27 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index a192b49706f..52748b1cbc2 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -250,44 +250,37 @@ $ tree . -a +## Suggested workflows -# Reference-level explanation - -The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of the CCI. This proposal does not add any new CCI or change their contents (modulo sorting order, whitespace). - -## New file: `crate-info.json` - -The `crate-info.json` file contains the unmerged contents of a single crates' version of their corresponding CCI. This file is written if the flag `--write-info-json=path/to/crate-info.json` is provided. - -`crate-info.json` is a JSON object with several key-value pairs. Every value is a JSON array. Every element of the JSON array is a two-element array: a destination filename relative to the doc root, and the representation of the part. The representation of the part depends on the type of CCI that it describes. +### No cross-crate information +Provide `--merge=none` to every invocation of rustdoc. -* key: `version` +### Cross-crate information, mutate shared directory -The value will be a string encoding of a version number. If rustdoc is provided a `--include-info-json` flag that points to a `crate-info.json` file with a version number that it cannot support, it will fail. Rustdoc only guarantees that it will accept `crate-info.json` files written by the same version of rustdoc. There is no forward or backward compatibility. A single number prefixed with capital V is chosen for convenience, as `crate-info.json` does not make distinctions between breaking and non-breaking changes. Rustdoc is the only explicitly supported consumer of `crate-info.json`. +Use `--merge=read-write`, and specify the same `--out-dir` to every invocation of rustdoc. `--merge=read-write` will be the default value if `--merge` is not provided. This is the workflow that Cargo uses. -* key: `src-files-js`, for `doc/src-files.js` +### Cross-crate information, no shared directory -This part is the JSON representation of the source index that is later stored in a `srcIndex` global variable. +Specify a different `--out-dir` to every invocation of rustdoc. Additionally, you should provide `--write-info-json=` and `--merge=none` when documenting the dependencies of your root crate. Then, when you document the root crate, you will provide `--include-info-json=`, `--include-rendered-docs=` for each one of your dependencies, and `--merge=write-only`. You should provide `--extern-html-root-url`, and specify a static, absolute location for the URL. This flag, with the same URL, will be needed for every invocation of rustdoc, for every dependency. -* key: `search-index-js`, for `doc/search-index.js` - -This part is the JSON encoded search index, before it has been installed in `search-index.js`. - -* key: `all-crates`, for `doc/crates.js` - -This part is the crate name. - -* key: `crates-index`, for `doc/index.html` +# Reference-level explanation -This part is also the crate name. It represents a different kind of CCI because it is written to a `doc/index.html`, and rendered as an HTML document instead as JSON. In principal, we could use this part to add more information to the crates index `doc/index.html` (the first line of the top level crate documentation, for example). +The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of the CCI. This proposal does not add any new CCI or change their contents (modulo sorting order, whitespace). -* key: `type-impl`, for `doc/type.impl/**/*.js` +## New file: `crate-info.json` -This part is a two element array with the crate name and the JSON representation of a type implementation. +`crate-info.json` is an artifact that encodes the partial contents and destination of several cross-crate information files. It only encodes information about a single-crate. This file is written if `--write-info-json` is provided. The current crate's information and any `crate-info.json` added through `--include-info-json` are merged and rendered if `--merge=read-write` or `--merge=write-only` are provided. -* key: `trait-impl`, for `doc/trait.impl/**/*.js` +The content of `crate-info.json` is unstable. Rustdoc only guarantees that it will accept `crate-info.json` files written by the same version of rustdoc, and rustdoc is the only explicitly supported consumer of `crate-info.json`. Only the presence of `crate-info.json` is stabilized. Non-normatively, there are several pieces of information that `crate-info.json` may contain: -This part is a two element array with the crate name and the JSON representation of a trait implementation. +* Partial source file index for generating `doc/src-files.js`. +* Partial search index for generating `doc/search-index.js`. +* Crate name for generating `doc/crates.js`. +* Crate name and information for generating `doc/index.html`. +* Trait implementation list for generating `doc/trait.impl/**/*.js`. +* Type implementation list for generating `doc/type.impl/**/*.js`. +* The file may include versioning information intended to assist in generating error messages if an incompatible `crate-info.json` is provided through `--include-info-json`. +* The file may contain other information related to cross-crate information that is added in the future. ## New flag: `--write-info-json=path/to/crate-info.json` From f6393655fdd464106a41a918d22c0b5ee16ade7b Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Wed, 31 Jul 2024 05:21:49 +0000 Subject: [PATCH 18/28] type impl,buck2 link,extern-html-root-url,period fixed broken buck2 link, address notriddle's comment about trait vs type impls, adds period at ends of sentence that are missing periods, and adds clarification about the usefulness of extern-html-root-url --- ...0000-mergeable-rustdoc-cross-crate-info.md | 22 ++++++++++--------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index 52748b1cbc2..30b5266d626 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -261,7 +261,7 @@ Use `--merge=read-write`, and specify the same `--out-dir` to every invocation o ### Cross-crate information, no shared directory -Specify a different `--out-dir` to every invocation of rustdoc. Additionally, you should provide `--write-info-json=` and `--merge=none` when documenting the dependencies of your root crate. Then, when you document the root crate, you will provide `--include-info-json=`, `--include-rendered-docs=` for each one of your dependencies, and `--merge=write-only`. You should provide `--extern-html-root-url`, and specify a static, absolute location for the URL. This flag, with the same URL, will be needed for every invocation of rustdoc, for every dependency. +Specify a different `--out-dir` to every invocation of rustdoc. Additionally, you should provide `--write-info-json=` and `--merge=none` when documenting the dependencies of your root crate. Then, when you document the root crate, you will provide `--include-info-json=`, `--include-rendered-docs=` for each one of your dependencies, and `--merge=write-only`. You should provide `--extern-html-root-url`, and specify a static, absolute location for the URL. This flag, with the same URL, will be needed for every invocation of rustdoc, for every dependency. # Reference-level explanation @@ -332,8 +332,8 @@ This RFC does not alter previous compatibility guarantees made about the output In the same way that the [rustdoc HTML output is unstable](https://rust-lang.github.io/rfcs/2963-rustdoc-json.html#:~:text=The%20HTML%20output%20of%20rustdoc,into%20a%20different%20format%20impractical), the content of `crate-info.json` will be considered unstable. Between versions of rustdoc, breaking changes to the content of `crate-info.json` should be expected. Only the presence of a `crate-info.json` file is promised, under `--write-info-json`. Merging cross-crate information generated by disparate versions of rustdoc is not supported. To detect whether `crate-info.json` is compatible, rustdoc includes a version number in these files (see New file: `crate-info.json`). The implementation of the RFC itself is designed to produce only minimal changes to cross-crate info files and the HTML output of rustdoc. Exhaustively, the implementation is allowed to -* Change the sorting order of trait implementations, type implementations, and other cross-crate info in the HTML output of rustdoc -* Add a comment on the last line of generated HTML pages, to store metadata relevant to appending items to them +* Change the sorting order of trait implementations, type implementations, and other cross-crate info in the HTML output of rustdoc. +* Add a comment on the last line of generated HTML pages, to store metadata relevant to appending items to them. * Refactor the JavaScript contents of cross-crate information files, in ways that do not change their overall behavior. If the JavaScript fragment declared an array called `ALL_CRATES` with certain contents, it will continue to do so. Changes this minimal are intended to avoid breaking tools that use the output of rustdoc, like Cargo, docs.rs, and rustdoc's JavaScript frontend, in the near-term. Going forward, rustdoc will not make formal guarantees about the content of cross-crate info files. @@ -365,7 +365,7 @@ There is an [open issue](https://github.com/bazelbuild/rules_rust/issues/1837) r ## Buck2 -The Buck2 build system has rules for building and testing rust binaries and libraries. +The Buck2 build system has rules for building and testing rust binaries and libraries. It has a subtarget, `[doc]`, for generating rustdoc for a crate. @@ -405,11 +405,13 @@ This proposal proposes to continue reading from the rendered cross-crate informa ## Item links? -Require users to pass `--extern-html-root-url` on all external dependencies (current), vs. add a new flag to facilitate missing docs links being generated across all user crates (should consider) +Require users to pass `--extern-html-root-url` on all external dependencies (current), vs. add a new flag to facilitate missing docs links being generated across all user crates (should consider). -Cross-crate links are an important consideration when merging documentation. rustdoc provides a flag, `--extern-html-root-url`, which can provide fine-grain control over the generated URL for documentation. We believe this is already sufficient for generating cross-crate links, and we will make no changes to the generation of item links. Example usage of this flag is in the Guide-level explanation. +For the purpose of generating cross-crate links, rustdoc classifies the location of crates as external, local, or unknown (relative to the crate in the current invocation of rustdoc). Local crates are the crates that share the same `--out-dir`. External crates have documentation that could not be found in the current `--out-dir`, but otherwise have a known location. Item links are not generated to crates with an unknown location. When the `--extern-html-root-url==` flag is provided, an otherwise unknown crate `` becomes an externally located crate, forcing it to generate item links. -A proposal is to provide more options to facilitate cross-crate links. For example, we could add a flag that implies `--extern-html-root-url` and `--include-info-json` on all crates passed through `--extern`. It could be something like `--extern-html-root-url-all-user-crates`. User crates would be taken to mean the transitive dependencies of the current crate, excluding the standard library. This is complicated by things like . +This is of relevance to this proposal, because users who document crates with separate `--out-dir`s may still expect cross-crate links to work. Currently, `--extern-html-root-url` is the exclusive mechanism for specifying link destinations for crates who would otherwise have an unknown location. We will expect users to provide `--extern-html-root-url` for all direct dependencies of a crate they are documenting. Example usage of this flag is in the Guide-level explanation. + +A proposal is to provide more options to facilitate cross-crate links. For example, we could add a flag that implies `--extern-html-root-url`, with a fixed location, to all crates passed through `--extern`. User crates would be taken to mean the transitive dependencies of the current crate, excluding the standard library. ## Reuse existing option? @@ -427,10 +429,10 @@ We could also leave it as-is: always false unless we're scraping examples, and g # Future possibilities -This change could begin to facilitate type aliases and the trait aliases being +This change could begin to facilitate trait implementations being statically compiled as part of the .html documentation, instead of being loaded -as separate JavaScript files. Each type and trait alias could be stored as an -HTML part, which are then merged into the regular documentation. +as separate JavaScript files. Each trait implementation could be stored as an +HTML part, which are then merged into the regular documentation. Implementations of traits on type aliases should remain separate, as they serve as a [size hack](https://github.com/rust-lang/rust/pull/116471). Another possibility is for `doc.parts` to be distributed on `docs.rs` along with the regular documentation. This would facilitate a mode where documentation of the dependencies could be downloaded externally, instead of being rebuilt locally. From 627a0ba6e1bfb7277743535b79072d18ce89f5d9 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Wed, 31 Jul 2024 17:02:14 +0000 Subject: [PATCH 19/28] rename flags per camelid, clarify workflow * Rename --merge=none|read-write|write-only to --merge=none|shared|finalize * --write-info-json=target/doc.parts/my-crate/crate-info.json -> --parts-out-dir=target/doc.parts/my-crate * --include-info-json=target/doc.parts/my-crate/crate-info.json -> --include-parts-dir=target/doc.parts/my-crate * Clarify the three modes / workflows in the text of the RFC --- ...0000-mergeable-rustdoc-cross-crate-info.md | 89 +++++++++++-------- 1 file changed, 51 insertions(+), 38 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index 30b5266d626..f9104395e0a 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -5,7 +5,7 @@ # Summary -Mergeable cross-crate information in rustdoc. Facilitates the generation of documentation indexes in workspaces with many crates by allowing each crate to write to an independent output directory. Final documentation is rendered with a lightweight merge step. Configurable with command-line flags, this proposal writes a `crate-info.json` file to hold pre-merge cross-crate information. Currently, rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating rustdoc in build systems that enforce the independence of build actions. +Mergeable cross-crate information in rustdoc. Facilitates the generation of documentation indexes in workspaces with many crates by allowing each crate to write to an independent output directory. Final documentation is rendered with a lightweight merge step. Configurable with command-line flags, this proposal writes a `doc.parts` directory to hold pre-merge cross-crate information. Currently, rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating rustdoc in build systems that enforce the independence of build actions. # Motivation @@ -47,7 +47,7 @@ rustc --crate-name=t --crate-type=lib --edition=2021 --emit=metadata --out-dir=t rustc --crate-name=s --crate-type=lib --edition=2021 --emit=metadata --out-dir=s/target --extern t=t/target/libt.rmeta s/src/lib.rs ``` -Document `s` and `t` independently, providing `--merge=none`, `--write-info-json`. +Document `s` and `t` independently, providing `--merge=none`, `--parts-out-dir`. ```shell rustdoc \ @@ -58,7 +58,7 @@ rustdoc \ --out-dir=t/target/doc \ --extern-html-root-url t=$MERGED \ --merge=none \ - --write-info-json=t/target/doc.parts/t/crate-info.json \ + --parts-out-dir=t/target/doc.parts/t \ t/src/lib.rs rustdoc \ -Z unstable-options \ @@ -69,12 +69,12 @@ rustdoc \ --extern-html-root-url s=$MERGED \ --extern-html-root-url t=$MERGED \ --merge=none \ - --write-info-json=s/target/doc.parts/s/crate-info.json \ + --parts-out-dir=s/target/doc.parts/s \ --extern t=t/target/libt.rmeta \ s/src/lib.rs ``` -Link everything with a final invocation of rustdoc on `i`. We will provide `--merge=write-only`, `--include-info-json`, and `--include-rendered-docs`. See the Reference-level explanation about these flags. +Link everything with a final invocation of rustdoc on `i`. We will provide `--merge=finalize`, `--include-parts-dir`, and `--include-rendered-docs`. See the Reference-level explanation about these flags. ```shell rustdoc \ @@ -87,9 +87,9 @@ rustdoc \ --extern-html-root-url s=$MERGED \ --extern-html-root-url t=$MERGED \ --extern-html-root-url i=$MERGED \ - --merge=write-only \ - --include-info-json=t/target/doc.parts/t/crate-info.json \ - --include-info-json=s/target/doc.parts/s/crate-info.json \ + --merge=finalize \ + --include-parts-dir=t/target/doc.parts/t \ + --include-parts-dir=s/target/doc.parts/s \ --extern t=t/target/libt.rmeta \ --extern s=s/target/libs.rmeta \ --include-rendered-docs=t/target/doc/t \ @@ -252,26 +252,39 @@ $ tree . -a ## Suggested workflows -### No cross-crate information -Provide `--merge=none` to every invocation of rustdoc. +With this proposal, there are three modes of invoking rustdoc. These modes are configured through the choice of the `--merge`, `--parts-out-dir`, `--include-parts-dir`, and `--include-rendered-docs` flags. -### Cross-crate information, mutate shared directory +### Default workflow: mutate shared directory -Use `--merge=read-write`, and specify the same `--out-dir` to every invocation of rustdoc. `--merge=read-write` will be the default value if `--merge` is not provided. This is the workflow that Cargo uses. +In this workflow, we document a single crate, or a collection of crates into a shared output directory that is continuously updated. +Files in this output directory are modified by multiple rustdoc invocations. Use `--merge=shared`, and specify the same `--out-dir` to every invocation of rustdoc. `--merge=shared` will be the default value if `--merge` is not provided. This is the workflow that Cargo uses, and only mode of invoking rustdoc before this RFC. -### Cross-crate information, no shared directory +### Document intermediate crates -Specify a different `--out-dir` to every invocation of rustdoc. Additionally, you should provide `--write-info-json=` and `--merge=none` when documenting the dependencies of your root crate. Then, when you document the root crate, you will provide `--include-info-json=`, `--include-rendered-docs=` for each one of your dependencies, and `--merge=write-only`. You should provide `--extern-html-root-url`, and specify a static, absolute location for the URL. This flag, with the same URL, will be needed for every invocation of rustdoc, for every dependency. +Document regular (non-root, non-index) crates using a dedicated HTML output directory and a dedicated "parts" output directory. No cross-crate data nor rendered HTML output is included from other crates. + +This mode only renders the HTML item documentation for the current crate. It does not produce a search index, cross-crate trait implementations, or an index page. It is expected that users follow this mode with 'Document a final crate' if these cross-crate features are desired. + +In this mode, a user may specify a different `--out-dir` to every invocation of rustdoc. Additionally, a user will provide `--parts-out-dir=` and `--merge=none` when documenting every crate. +The user should provide `--extern-html-root-url`, and specify a absolute final location for the URL, if they document crates in separate `--out-dir`s. This flag, with the same URL, will be needed for every invocation of rustdoc, for every dependency. + +### Document a final crate + +In this context, a final crate is a crate that depends directly on every crate that a user intends to appear in the documentation bundle. It may be an index crate that has no meaningful functionality on its own. It may also be a library crate that depends on every crate in a workspace. + +In this mode, rendered HTML and *finalized* cross-crate information are generated into a `target/doc/my-final-crate` folder. No *incremental* parts are generated (i.e., no `target/doc.parts/my-final-crate`). + +When a user documents the final crate, they will provide `--include-parts-dir=`, `--include-rendered-docs=` for each one of the dependencies, and `--merge=finalize`. They will provide `--extern-html-root-url`, in the way described in 'Document an intermediate crate'. # Reference-level explanation The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of the CCI. This proposal does not add any new CCI or change their contents (modulo sorting order, whitespace). -## New file: `crate-info.json` +## New directory: `doc.parts` -`crate-info.json` is an artifact that encodes the partial contents and destination of several cross-crate information files. It only encodes information about a single-crate. This file is written if `--write-info-json` is provided. The current crate's information and any `crate-info.json` added through `--include-info-json` are merged and rendered if `--merge=read-write` or `--merge=write-only` are provided. +`doc.parts` is a directory that holds the partial contents and destination of several cross-crate information files. It only encodes information about a single-crate. This file is written if `--parts-out-dir` is provided. The current crate's information and any `doc.parts` added through `--include-parts-dir` are merged and rendered if `--merge=shared` or `--merge=finalize` are provided. -The content of `crate-info.json` is unstable. Rustdoc only guarantees that it will accept `crate-info.json` files written by the same version of rustdoc, and rustdoc is the only explicitly supported consumer of `crate-info.json`. Only the presence of `crate-info.json` is stabilized. Non-normatively, there are several pieces of information that `crate-info.json` may contain: +The content of `doc.parts` is unstable. Rustdoc only guarantees that it will accept `doc.parts` files written by the same version of rustdoc, and rustdoc is the only explicitly supported consumer of `doc.parts`. Only the presence of `doc.parts` is stabilized. Non-normatively, there are several pieces of information that `doc.parts` may contain: * Partial source file index for generating `doc/src-files.js`. * Partial search index for generating `doc/search-index.js`. @@ -279,40 +292,40 @@ The content of `crate-info.json` is unstable. Rustdoc only guarantees that it wi * Crate name and information for generating `doc/index.html`. * Trait implementation list for generating `doc/trait.impl/**/*.js`. * Type implementation list for generating `doc/type.impl/**/*.js`. -* The file may include versioning information intended to assist in generating error messages if an incompatible `crate-info.json` is provided through `--include-info-json`. +* The file may include versioning information intended to assist in generating error messages if an incompatible `doc.parts` is provided through `--include-parts-dir`. * The file may contain other information related to cross-crate information that is added in the future. -## New flag: `--write-info-json=path/to/crate-info.json` +## New flag: `--parts-out-dir=/doc.parts/` -When this flag is provided, the unmerged parts for the current crate will be written to `path/to/crate-info.json`. A typical `` is `./target/doc.parts//crate-info.json`. +When this flag is provided, the unmerged parts for the current crate will be written to `path/to/doc.parts/`. A typical argument is `./target/doc.parts/rand`. -Crates `--include-info-json`ed will not appear in `crate-info.json`, as `crate-info.json` only includes the CCI parts for the current crate. +Crates `--include-parts-dir`ed will not appear in `doc.parts`, as `doc.parts` only includes the CCI parts for the current crate. -If this flag is not provided, no `crate-info.json` will be written. +If this flag is not provided, no `doc.parts` will be written. -## New flag: `--include-info-json=` +## New flag: `--include-parts-dir=` -If this flag is provided, rustdoc will expect `path/to/crate-info.json` to be the `crate-info.json` file containing the parts for a crate. It will append these parts to the ones it will render in the doc root (`--out-dir`). The info that's included is not written to `crate-info.json`, as `crate-info.json` only holds the CCI parts for the current crate. +If this flag is provided, rustdoc will expect that a previous invocation of rustdoc was made with `--parts-out-dir=`. It will append the parts from the previous invocation to the ones it will render in the doc root (`--out-dir`). The info that's included is not written to its own `doc.parts`, as `doc.parts` only holds the CCI parts for the current crate. -This flag is similar to `--extern-html-root-url` in that it only needs to be provided for externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided in `--extern-html-root-url` never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--include-info-json` tells rustdoc where to search for the `crate-info.json` directory at documentation-time. It must not be a URL. +This flag is similar to `--extern-html-root-url` in that it only needs to be provided for externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided in `--extern-html-root-url` never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--include-parts-dir` tells rustdoc where to search for the `doc.parts` directory at documentation-time. It must not be a URL. -In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc.parts/s`, so rustdoc is called with `--include-info-json=s/target/doc.parts/s/crate-info.json`. +In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc.parts/s`, so rustdoc is called with `--include-parts-dir=s/target/doc.parts/s`. ## New flag: `--include-rendered-docs=` Rustdoc will assume that `` was used as the `--out-dir` for ``. This documentation will be copied into the directory specified by `--out-dir`. Rustdoc will effectively run `cp -r `. -## New flag: `--merge=read-write|none|write-only` +## New flag: `--merge=none|shared|finalize` -This flag controls two internal paramaters: `read_rendered_cci`, and `write_rendered_cci`. +This flag corresponds to the three modes of invoking rustdoc described in 'Suggested workflows'. It controls two internal paramaters: `read_rendered_cci`, and `write_rendered_cci`. When `write_rendered_cci` is active, rustdoc will output the rendered parts to the doc root (`--out-dir`). Rustdoc will generate files like `doc/search-index.js`, `doc/search.desc`, `doc/index.html`, etc if and only if this parameter is true. -When `read_rendered_cci` is active, rustdoc will look in the `--out-dir` for rendered cross-crate info files. These files will be used as the base. Any new parts that rustdoc generates with its current invocation and any parts fetched with `include-info-json` will be appended to these base files. When it is disabled, the cross-crate info files start empty and are populated with the current crate's info and any crates fetched with `--include-info-json`. +When `read_rendered_cci` is active, rustdoc will look in the `--out-dir` for rendered cross-crate info files. These files will be used as the base. Any new parts that rustdoc generates with its current invocation and any parts fetched with `include-parts-dir` will be appended to these base files. When it is disabled, the cross-crate info files start empty and are populated with the current crate's info and any crates fetched with `--include-parts-dir`. -* `--merge=read-write` (`read_rendered_cci && write_rendered_cci`) is the default, and reflects the current behavior of rustdoc. +* `--merge=shared` (`read_rendered_cci && write_rendered_cci`) is the default, and reflects the current behavior of rustdoc. * `--merge=none` (`!read_rendered_cci && !write_rendered_cci`) means that rustdoc will ignore the cross-crate files in the doc root. Only generate item docs. -* `--merge=write-only` (`!read_rendered_cci && write_rendered_cci`) outputs crate info based only on the current crate and `--include-info-json`'ed crates. +* `--merge=finalize` (`!read_rendered_cci && write_rendered_cci`) outputs crate info based only on the current crate and `--include-parts-dir`'ed crates. * A (`read_rendered_cci && !write_rendered_cci`) mode would be useless, since the data that is read would be ignored and not written. ## Merge step @@ -321,7 +334,7 @@ This proposal is capable of addressing two primary use cases. It allows develope * Documenting a crate and its transitive dependencies in parallel in build systems that require build actions to be independent * Producing a documentation index of a large number of crates, in such a way that if one crate is updated, only the updated crates and an index have to be redocumented. This scenario is demonstrated in the Guide-level explanation. -CCI is not automatically enabled in either situation. A combination of the `--include-info-json`, `--merge`, and `--write-info-json` flags are needed to produce this behavior. This RFC provides a minimal set of tools that allow developers of build systems, like Bazel and Buck2, to create rules for these scenarios. +CCI is not automatically enabled in either situation. A combination of the `--include-parts-dir`, `--merge`, and `--parts-out-dir` flags are needed to produce this behavior. This RFC provides a minimal set of tools that allow developers of build systems, like Bazel and Buck2, to create rules for these scenarios. Discussion of whether additional features should be included to facilitate this merge step can be found in Unresolved questions (Index crate). @@ -329,7 +342,7 @@ Discussion of whether additional features should be included to facilitate this This RFC does not alter previous compatibility guarantees made about the output of rustdoc. In particular it does not stabilize the presence of the rendered cross-crate information files, their content, or the HTML generated by rustdoc. -In the same way that the [rustdoc HTML output is unstable](https://rust-lang.github.io/rfcs/2963-rustdoc-json.html#:~:text=The%20HTML%20output%20of%20rustdoc,into%20a%20different%20format%20impractical), the content of `crate-info.json` will be considered unstable. Between versions of rustdoc, breaking changes to the content of `crate-info.json` should be expected. Only the presence of a `crate-info.json` file is promised, under `--write-info-json`. Merging cross-crate information generated by disparate versions of rustdoc is not supported. To detect whether `crate-info.json` is compatible, rustdoc includes a version number in these files (see New file: `crate-info.json`). +In the same way that the [rustdoc HTML output is unstable](https://rust-lang.github.io/rfcs/2963-rustdoc-json.html#:~:text=The%20HTML%20output%20of%20rustdoc,into%20a%20different%20format%20impractical), the content of `doc.parts` will be considered unstable. Between versions of rustdoc, breaking changes to the content of `doc.parts` should be expected. Only the presence of a `doc.parts` directory is promised, under `--parts-out-dir`. Merging cross-crate information generated by disparate versions of rustdoc is not supported. To detect whether `doc.parts` is compatible, rustdoc includes a version number in these files (see New directory: `doc.parts`). The implementation of the RFC itself is designed to produce only minimal changes to cross-crate info files and the HTML output of rustdoc. Exhaustively, the implementation is allowed to * Change the sorting order of trait implementations, type implementations, and other cross-crate info in the HTML output of rustdoc. @@ -388,20 +401,20 @@ Require users to generate documentation bundles via an index crate (current) vs. If one would like to merge the documentation of several crates, we could continue to require users to provide an index crate, like [the fuchsia index](https://fuchsia-docs.firebaseapp.com/rust/rustdoc_index/). This serves as the target of the rustdoc invocation, and the landing page for the collected documentation. Supporting only this style of index would require the fewest changes. This is the mode described in the Guide-level explanation. -The proposition, to allow users of rustdoc the flexibility of not having to produce an index, is to allow rustdoc to be run in a mode where no target crate is provided. It would generate rendered cross-crate information based only on what is provided through `--include-info-json`. The source crates are provided to rustdoc, through a mechanism like `--extern`, rustdoc merges and writes the CCI, and copies the item and module links to the doc root. This would require more extensive changes, as rustdoc assumes that it is invoked with a target crate. This mode is somewhat analogous to the [example scraping mode](https://github.com/rust-lang/rfcs/blob/master/text/3123-rustdoc-scrape-examples.md). Having to create an index crate, that actively uses all of the crates in the environment, might prohibit the use of this feature in settings where users do not intend to produce an index, or where exhaustively listing all dependencies (to `--extern` them) is difficult. +The proposition, to allow users of rustdoc the flexibility of not having to produce an index, is to allow rustdoc to be run in a mode where no target crate is provided. It would generate rendered cross-crate information based only on what is provided through `--include-parts-dir`. The source crates are provided to rustdoc, through a mechanism like `--extern`, rustdoc merges and writes the CCI, and copies the item and module links to the doc root. This would require more extensive changes, as rustdoc assumes that it is invoked with a target crate. This mode is somewhat analogous to the [example scraping mode](https://github.com/rust-lang/rfcs/blob/master/text/3123-rustdoc-scrape-examples.md). Having to create an index crate, that actively uses all of the crates in the environment, might prohibit the use of this feature in settings where users do not intend to produce an index, or where exhaustively listing all dependencies (to `--extern` them) is difficult. ## Unconditionally generating the `doc.parts` files? Generate no extra files (current) vs. unconditionally creating `doc.parts` to enable more complex future CCI (should consider). The current version of rustdoc performs merging by [collecting JSON](https://github.com/rust-lang/rust/blob/c25ac9d6cc285e57e1176dc2da6848b9d0163810/src/librustdoc/html/render/write_shared.rs#L166) blobs from the contents of the already-rendered CCI. -This proposal proposes to continue reading from the rendered cross-crate information under the default `--merge=read-write`. It can also read `crate-info.json` files, under `--include-info-json`. However, there are several issues with reading from the rendered CCI that must be stated: +This proposal proposes to continue reading from the rendered cross-crate information under the default `--merge=shared`. It can also read `doc.parts` directories, under `--include-parts-dir`. However, there are several issues with reading from the rendered CCI that must be stated: * Every rustdoc process outputs the CCI to the same doc root by default * It is difficult to extract the items in a diverse set of rendered HTML files. This is anticipating of the CCI to include HTML files that, for example, statically include type+trait implementations directly -* Reading exclusively from `crate-info.json` is simpler than the existing `serde_json` dependency for extracting the blobs, as opposed to handwritten CCI-type specific parsing (current) +* Reading exclusively from `doc.parts` is simpler than the existing `serde_json` dependency for extracting the blobs, as opposed to handwritten CCI-type specific parsing (current) * With this proposal, there will be duplicate logic to read from both `doc.parts` files and rendered CCI. -[@jsha proposes](https://github.com/rust-lang/rfcs/pull/3662#issuecomment-2184077829) unconditionally generating and reading from `crate-info.json`, with no appending to the rendered crate info. +[@jsha proposes](https://github.com/rust-lang/rfcs/pull/3662#issuecomment-2184077829) unconditionally generating and reading from `doc.parts`, with no appending to the rendered crate info. ## Item links? @@ -438,4 +451,4 @@ Another possibility is for `doc.parts` to be distributed on `docs.rs` along with A future possibility related to the index crate idea is to have an option for embedding user-specified HTML into the `--enable-index-page`'s HTML. -The changes in this proposal are intended to work with no changes to Cargo and docs.rs. However, there may be benefits to using `--merge=write-only` with Cargo, as it would remove the need for locking the output directory. More of the documentation process could happen in parallel, which may speed up execution time. +The changes in this proposal are intended to work with no changes to Cargo and docs.rs. However, there may be benefits to using `--merge=finalize` with Cargo, as it would remove the need for locking the output directory. More of the documentation process could happen in parallel, which may speed up execution time. From b0dc37d4c9e6689856c49990f1a807f3cff15613 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Wed, 31 Jul 2024 18:45:01 +0000 Subject: [PATCH 20/28] move extern-html-root-url, remove no_emit_shared I moved the section discussing --extern-html-root-url from unresolved questions to the Reference-level explanation, as there have been no proposals discussing alternatives. I removed the section about no_emit_shared in unresolved questions, because it has not come up in discussion and no_emit_shared was never a command line option anyway. The names proposed in discussion are better and more declarative suggestions --- ...0000-mergeable-rustdoc-cross-crate-info.md | 32 +++++-------------- 1 file changed, 8 insertions(+), 24 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index f9104395e0a..9f160cf4c9f 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -351,6 +351,14 @@ The implementation of the RFC itself is designed to produce only minimal changes Changes this minimal are intended to avoid breaking tools that use the output of rustdoc, like Cargo, docs.rs, and rustdoc's JavaScript frontend, in the near-term. Going forward, rustdoc will not make formal guarantees about the content of cross-crate info files. +## Note about the existing flag `--extern-html-root-url` + +For the purpose of generating cross-crate links, rustdoc classifies the location of crates as external, local, or unknown (relative to the crate in the current invocation of rustdoc). Local crates are the crates that share the same `--out-dir`. External crates have documentation that could not be found in the current `--out-dir`, but otherwise have a known location. Item links are not generated to crates with an unknown location. When the `--extern-html-root-url==` flag is provided, an otherwise unknown crate `` becomes an externally located crate, forcing it to generate item links. + +This is of relevance to this proposal, because users who document crates with separate `--out-dir`s may still expect cross-crate links to work. Currently, `--extern-html-root-url` is the exclusive command line option for specifying link destinations for crates who would otherwise have an unknown location. We will expect users to provide `--extern-html-root-url` for all direct dependencies of a crate they are documenting, if they use separate `--out-dir`s. Example usage of this flag is in the Guide-level explanation. + +The limitation of `--extern-html-root-url` is that it needs to be provided with an absolute URL for the final docs destination. If your docs are hosted on `https://example.com/docs/`, this URL must be *known at documentation time*, and provided through `--extern-html-root-url==https://example.com/docs/`. *Absolute URLs*, instead of relative URLs, are generated for items in externally located crates. A future proposal may address this limitation by providing a command line option that generates relative URLs (like is done between items in the current crate, or other locally documented crates) for selected external crates, assuming that these crates will end up in the same bundle. The existing `--extern-html-root-url` is sufficient for the use cases envisioned by this RFC, despite the limitation. + # Drawbacks The WIP may change the sorting order of the elements in the CCI. It does not change the content of the documentation, and is intended to work without modifying Cargo and docs.rs. @@ -416,30 +424,6 @@ This proposal proposes to continue reading from the rendered cross-crate informa [@jsha proposes](https://github.com/rust-lang/rfcs/pull/3662#issuecomment-2184077829) unconditionally generating and reading from `doc.parts`, with no appending to the rendered crate info. -## Item links? - -Require users to pass `--extern-html-root-url` on all external dependencies (current), vs. add a new flag to facilitate missing docs links being generated across all user crates (should consider). - -For the purpose of generating cross-crate links, rustdoc classifies the location of crates as external, local, or unknown (relative to the crate in the current invocation of rustdoc). Local crates are the crates that share the same `--out-dir`. External crates have documentation that could not be found in the current `--out-dir`, but otherwise have a known location. Item links are not generated to crates with an unknown location. When the `--extern-html-root-url==` flag is provided, an otherwise unknown crate `` becomes an externally located crate, forcing it to generate item links. - -This is of relevance to this proposal, because users who document crates with separate `--out-dir`s may still expect cross-crate links to work. Currently, `--extern-html-root-url` is the exclusive mechanism for specifying link destinations for crates who would otherwise have an unknown location. We will expect users to provide `--extern-html-root-url` for all direct dependencies of a crate they are documenting. Example usage of this flag is in the Guide-level explanation. - -A proposal is to provide more options to facilitate cross-crate links. For example, we could add a flag that implies `--extern-html-root-url`, with a fixed location, to all crates passed through `--extern`. User crates would be taken to mean the transitive dependencies of the current crate, excluding the standard library. - -## Reuse existing option? - -Create a new flag, `--merge` (proposed), vs. use existing option `no_emit_shared`. - -There is a render option, `no_emit_shared`, which is used to conditionally generate the cross-crate information. It also controls the generation of static files, user CSS, etc. - -This option is not configurable from the command line, and appears to be enabled unless rustdoc is run in its example scraping mode. - -We could make it configurable from the command line, unconditionally generate `doc.parts`, and use it to gate the merging of CCI. - -We could also make it configurable from the command line, and use it to gate the generation of `doc.parts` and the generation of all of the shared files. - -We could also leave it as-is: always false unless we're scraping examples, and gate the generation of `doc.parts` and the generation of all of the shared files. - # Future possibilities This change could begin to facilitate trait implementations being From 4b076f69ed311491fae09dbd71dc08d27bef438c Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Fri, 2 Aug 2024 06:26:03 +0000 Subject: [PATCH 21/28] manishearth nits + stabilize crate-info.json * Explain default behavior of rustdoc in Reference-level explanation * Flag summary in Guide-level explanation * Say that crate-info.json is stable because build systems need to rely on it * t,s,i -> trait-crate, struct-crate, index-crate * T, S -> Trait, Struct --- ...0000-mergeable-rustdoc-cross-crate-info.md | 182 ++++++++++-------- 1 file changed, 106 insertions(+), 76 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index 9f160cf4c9f..9195b02b9ce 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -25,82 +25,106 @@ These considerations motivate adding an option for outputting partial CCI (parts # Guide-level explanation -In this example, there is a crate `t` which defines a trait `T`, and a crate `s` which defines a struct `S` that implements `T`. Our goal in this demo is for `S` to appear as an implementer in `T`'s docs, even if `s` and `t` are documented independently. This guide will be assuming that we want a crate `i` that serves as our documentation index. See the Unresolved questions section for ideas that do not require an index crate. +## New flag summary + +More details are in the Reference-level explanation. + +* `--parts-out-dir=doc.parts/`: Write information from the current crate to a provided directory. +* `--include-parts-dir=doc.parts/`: Include cross-crate information from these previously written `doc.parts` directories into a collection that will be written by the current invocation of rustdoc. +* `--merge=none`: Do not write cross-crate information to the `--out-dir`. +* `--merge=shared`: Read cross-crate information from the `--out-dir`, and write cross-crate information from the current crate and any crates included via `--include-parts-dir` to the `--out-dir`. +* `--merge=finalize`: Write cross-crate information from the current crate and any crates included via `--include-parts-dir`, overwriting conflicting files in the `--out-dir`. + +## Example + +In this example, there is a crate `trait-crate` which defines a trait `Trait`, and a crate `struct-crate` which defines a struct `Struct` that implements `Trait`. Our goal in this demo is for `Struct` to appear as an implementer in `Trait`'s docs, even if `struct-crate` and `trait-crate` are documented independently. This guide will be assuming that we want a crate `index-crate` that serves as our documentation index. See the Unresolved questions section for ideas that do not require an index crate. ```shell -mkdir -p t/src s/src i/src merged/doc -echo "pub trait T {}" > t/src/lib.rs -echo "pub struct S; impl t::T for S {}" > s/src/lib.rs +mkdir -p trait-crate/src struct-crate/src index-crate/src merged/doc +echo "pub trait Trait {}" > trait-crate/src/lib.rs +echo "pub struct Struct; impl trait-crate::Trait for Struct {}" > struct-crate/src/lib.rs MERGED=file://$(realpath merged/doc) ``` -[Actively use](https://doc.rust-lang.org/rustc/command-line-arguments.html#--extern-specify-where-an-external-library-is-located) `t` and `s` in `i`. The `extern crate` declarations are not needed if the crates are otherwise referenced in the index; intra-doc links are enough. +[Actively use](https://doc.rust-lang.org/rustc/command-line-arguments.html#--extern-specify-where-an-external-library-is-located) `trait-crate` and `struct-crate` in `index-crate`. The `extern crate` declarations are not needed if the crates are otherwise referenced in the index; intra-doc links are enough. ```shell -echo "extern crate t; extern crate s;" > i/src/lib.rs +echo "extern crate trait-crate; extern crate struct-crate;" > index-crate/src/lib.rs ``` Compile the crates. ```shell -rustc --crate-name=t --crate-type=lib --edition=2021 --emit=metadata --out-dir=t/target t/src/lib.rs -rustc --crate-name=s --crate-type=lib --edition=2021 --emit=metadata --out-dir=s/target --extern t=t/target/libt.rmeta s/src/lib.rs +rustc \ + --crate-name=trait-crate \ + --crate-type=lib \ + --edition=2021 \ + --emit=metadata \ + --out-dir=trait-crate/target \ + trait-crate/src/lib.rs +rustc \ + --crate-name=struct-crate \ + --crate-type=lib \ + --edition=2021 \ + --emit=metadata \ + --out-dir=struct-crate/target \ + --extern trait-crate=trait-crate/target/libt.rmeta struct-crate/src/lib.rs ``` -Document `s` and `t` independently, providing `--merge=none`, `--parts-out-dir`. +Document `struct-crate` and `trait-crate` independently, providing `--merge=none`, `--parts-out-dir`. ```shell rustdoc \ -Z unstable-options \ - --crate-name=t \ + --crate-name=trait-crate \ --crate-type=lib \ --edition=2021 \ - --out-dir=t/target/doc \ - --extern-html-root-url t=$MERGED \ + --out-dir=trait-crate/target/doc \ + --extern-html-root-url trait-crate=$MERGED \ --merge=none \ - --parts-out-dir=t/target/doc.parts/t \ - t/src/lib.rs + --parts-out-dir=trait-crate/target/doc.parts/trait-crate \ + trait-crate/src/lib.rs rustdoc \ -Z unstable-options \ - --crate-name=s \ + --crate-name=struct-crate \ --crate-type=lib \ --edition=2021 \ - --out-dir=s/target/doc \ - --extern-html-root-url s=$MERGED \ - --extern-html-root-url t=$MERGED \ + --out-dir=struct-crate/target/doc \ + --extern-html-root-url struct-crate=$MERGED \ + --extern-html-root-url trait-crate=$MERGED \ --merge=none \ - --parts-out-dir=s/target/doc.parts/s \ - --extern t=t/target/libt.rmeta \ - s/src/lib.rs + --parts-out-dir=struct-crate/target/doc.parts/struct-crate \ + --extern trait-crate=trait-crate/target/libt.rmeta \ + struct-crate/src/lib.rs ``` -Link everything with a final invocation of rustdoc on `i`. We will provide `--merge=finalize`, `--include-parts-dir`, and `--include-rendered-docs`. See the Reference-level explanation about these flags. +Link everything with a final invocation of rustdoc on `index-crate`. We will provide `--merge=finalize`, `--include-parts-dir`, and `--include-rendered-docs`. See the Reference-level explanation about these flags. ```shell rustdoc \ -Z unstable-options \ - --crate-name=i \ + --crate-name=index-crate \ --crate-type=lib \ --edition=2021 \ --enable-index-page \ - --out-dir=i/target/doc \ - --extern-html-root-url s=$MERGED \ - --extern-html-root-url t=$MERGED \ - --extern-html-root-url i=$MERGED \ + --out-dir=index-crate/target/doc \ + --extern-html-root-url struct-crate=$MERGED \ + --extern-html-root-url trait-crate=$MERGED \ + --extern-html-root-url index-crate=$MERGED \ --merge=finalize \ - --include-parts-dir=t/target/doc.parts/t \ - --include-parts-dir=s/target/doc.parts/s \ - --extern t=t/target/libt.rmeta \ - --extern s=s/target/libs.rmeta \ - --include-rendered-docs=t/target/doc/t \ - --include-rendered-docs=s/target/doc/s \ - -L t/target \ - i/src/lib.rs + --include-parts-dir=trait-crate/target/doc.parts/trait-crate \ + --include-parts-dir=struct-crate/target/doc.parts/struct-crate \ + --extern trait-crate=trait-crate/target/libt.rmeta \ + --extern struct-crate=struct-crate/target/libs.rmeta \ + --include-rendered-docs=trait-crate/target/doc/trait-crate \ + --include-rendered-docs=struct-crate/target/doc/struct-crate \ + -L trait-crate/target \ + index-crate/src/lib.rs ``` Browse `merged/doc/index.html` with cross-crate information. -In general, instead of two crates in the environment (`s` and `t`) you could have thousands. Upon any changes, only the index and the crates that are changed have to be re-documented. +In general, instead of two crates in the environment (`struct-crate` and `trait-crate`) you could have thousands. Upon any changes, only the index and the crates that are changed have to be re-documented.
Click here for a directory listing after running the example above. @@ -108,26 +132,26 @@ In general, instead of two crates in the environment (`s` and `t`) you could hav
 $ tree . -a
 .
-├── i
+├── index-crate
 │   ├── src
 │   │   └── lib.rs
 │   └── target
 │       └── doc
 │           ├── crates.js
 │           ├── help.html
-│           ├── i
+│           ├── index-crate
 │           │   ├── all.html
 │           │   ├── index.html
 │           │   └── sidebar-items.js
 │           ├── index.html
 │           ├── .lock
 │           ├── search.desc
-│           │   └── i
-│           │       └── i-desc-0-.js
+│           │   └── index-crate
+│           │       └── index-crate-desc-0-.js
 │           ├── search-index.js
 │           ├── settings.html
 │           ├── src
-│           │   └── i
+│           │   └── index-crate
 │           │       └── lib.rs.html
 │           ├── src-files.js
 │           ├── static.files
@@ -145,48 +169,48 @@ $ tree . -a
 │               │       └── unwind_safe
 │               │           ├── trait.RefUnwindSafe.js
 │               │           └── trait.UnwindSafe.js
-│               └── t
-│                   └── trait.T.js
+│               └── trait-crate
+│                   └── trait.Trait.js
 ├── merged
 │   └── doc
 │       ├── crates.js
 │       ├── help.html
-│       ├── i
+│       ├── index-crate
 │       │   ├── all.html
 │       │   ├── index.html
 │       │   └── sidebar-items.js
 │       ├── index.html
-│       ├── s
+│       ├── struct-crate
 │       │   ├── all.html
 │       │   ├── index.html
 │       │   ├── sidebar-items.js
-│       │   └── struct.S.html
+│       │   └── struct.Struct.html
 │       ├── search.desc
-│       │   ├── i
-│       │   │   └── i-desc-0-.js
-│       │   ├── s
-│       │   │   └── s-desc-0-.js
-│       │   └── t
-│       │       └── t-desc-0-.js
+│       │   ├── index-crate
+│       │   │   └── index-crate-desc-0-.js
+│       │   ├── struct-crate
+│       │   │   └── struct-crate-desc-0-.js
+│       │   └── trait-crate
+│       │       └── trait-crate-desc-0-.js
 │       ├── search-index.js
 │       ├── settings.html
 │       ├── src
-│       │   ├── i
+│       │   ├── index-crate
 │       │   │   └── lib.rs.html
-│       │   ├── s
+│       │   ├── struct-crate
 │       │   │   └── lib.rs.html
-│       │   └── t
+│       │   └── trait-crate
 │       │       └── lib.rs.html
 │       ├── src-files.js
 │       ├── static.files
 │           │   ├── COPYRIGHT-23e9bde6c69aea69.txt
 │           │   ├── favicon-2c020d218678b618.svg
 │           │   └── 
-│       ├── t
+│       ├── trait-crate
 │       │   ├── all.html
 │       │   ├── index.html
 │       │   ├── sidebar-items.js
-│       │   └── trait.T.html
+│       │   └── trait.Trait.html
 │       └── trait.impl
 │           ├── core
 │           │   ├── marker
@@ -198,32 +222,32 @@ $ tree . -a
 │           │       └── unwind_safe
 │           │           ├── trait.RefUnwindSafe.js
 │           │           └── trait.UnwindSafe.js
-│           └── t
-│               └── trait.T.js
-├── s
+│           └── trait-crate
+│               └── trait.Trait.js
+├── struct-crate
 │   ├── src
 │   │   └── lib.rs
 │   └── target
 │       ├── doc
 │       │   ├── help.html
 │       │   ├── .lock
-│       │   ├── s
+│       │   ├── struct-crate
 │       │   │   ├── all.html
 │       │   │   ├── index.html
 │       │   │   ├── sidebar-items.js
-│       │   │   └── struct.S.html
+│       │   │   └── struct.Struct.html
 │       │   ├── search.desc
-│       │   │   └── s
-│       │   │       └── s-desc-0-.js
+│       │   │   └── struct-crate
+│       │   │       └── struct-crate-desc-0-.js
 │       │   ├── settings.html
 │       │   └── src
-│       │       └── s
+│       │       └── struct-crate
 │       │           └── lib.rs.html
 │       ├── doc.parts
-│       │   └── s
+│       │   └── struct-crate
 │       │       └── crate-info.json
 │       └── libs.rmeta
-└── t
+└── trait-crate
     ├── src
     │   └── lib.rs
     └── target
@@ -231,19 +255,19 @@ $ tree . -a
         │   ├── help.html
         │   ├── .lock
         │   ├── search.desc
-        │   │   └── t
-        │   │       └── t-desc-0-.js
+        │   │   └── trait-crate
+        │   │       └── trait-crate-desc-0-.js
         │   ├── settings.html
         │   ├── src
-        │   │   └── t
+        │   │   └── trait-crate
         │   │       └── lib.rs.html
-        │   └── t
+        │   └── trait-crate
         │       ├── all.html
         │       ├── index.html
         │       ├── sidebar-items.js
-        │       └── trait.T.html
+        │       └── trait.Trait.html
         ├── doc.parts
-        │   └── t
+        │   └── trait-crate
         │       └── crate-info.json
         └── libt.rmeta
 
@@ -280,11 +304,15 @@ When a user documents the final crate, they will provide `--include-parts-dir=< The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of the CCI. This proposal does not add any new CCI or change their contents (modulo sorting order, whitespace). -## New directory: `doc.parts` +## New directory: `doc.parts//crate-info.json` `doc.parts` is a directory that holds the partial contents and destination of several cross-crate information files. It only encodes information about a single-crate. This file is written if `--parts-out-dir` is provided. The current crate's information and any `doc.parts` added through `--include-parts-dir` are merged and rendered if `--merge=shared` or `--merge=finalize` are provided. -The content of `doc.parts` is unstable. Rustdoc only guarantees that it will accept `doc.parts` files written by the same version of rustdoc, and rustdoc is the only explicitly supported consumer of `doc.parts`. Only the presence of `doc.parts` is stabilized. Non-normatively, there are several pieces of information that `doc.parts` may contain: +The directory `doc.parts` contains subdirectories for each crate. The crate names are chosen by the caller that invokes rustdoc, and are `--include-parts-dir` and `--parts-out-dir`. These crate names are arbitrary from the perspective of rustdoc, with the constraint that they must be unique. Rustdoc will write a file called `crate-info.json`, whose contents are unstable, to the subdirectory provided to `--parts-out-dir`. Even though the contents of `crate-info.json` are only intended to be consumed by rustdoc, its existence must be stabilized due to the fact that it may be declared to build systems. + +This directory structure may change with a new edition. + +Rustdoc only guarantees that it will accept `doc.parts` files written by the same version of rustdoc, and rustdoc is the only explicitly supported consumer of `doc.parts`. Only the presence of `doc.parts`, and its opaque child `crate-info.json`, are stabilized. Non-normatively, there are several pieces of information that `doc.parts` may contain: * Partial source file index for generating `doc/src-files.js`. * Partial search index for generating `doc/search-index.js`. @@ -303,6 +331,8 @@ Crates `--include-parts-dir`ed will not appear in `doc.parts`, as `doc.parts` on If this flag is not provided, no `doc.parts` will be written. +The output generated by this flag may be consumed by a future invocation to rustdoc that provides `--include-parts-dir=/doc.parts/`. + ## New flag: `--include-parts-dir=` If this flag is provided, rustdoc will expect that a previous invocation of rustdoc was made with `--parts-out-dir=`. It will append the parts from the previous invocation to the ones it will render in the doc root (`--out-dir`). The info that's included is not written to its own `doc.parts`, as `doc.parts` only holds the CCI parts for the current crate. @@ -323,7 +353,7 @@ When `write_rendered_cci` is active, rustdoc will output the rendered parts to t When `read_rendered_cci` is active, rustdoc will look in the `--out-dir` for rendered cross-crate info files. These files will be used as the base. Any new parts that rustdoc generates with its current invocation and any parts fetched with `include-parts-dir` will be appended to these base files. When it is disabled, the cross-crate info files start empty and are populated with the current crate's info and any crates fetched with `--include-parts-dir`. -* `--merge=shared` (`read_rendered_cci && write_rendered_cci`) is the default, and reflects the current behavior of rustdoc. +* `--merge=shared` (`read_rendered_cci && write_rendered_cci`) is the default, and reflects the current behavior of rustdoc. Rustdoc will look in its `--out-dir` for pre-existing cross-crate information files, and append information to these files from the current crate and any crates referenced through `--include-parts-dir`. * `--merge=none` (`!read_rendered_cci && !write_rendered_cci`) means that rustdoc will ignore the cross-crate files in the doc root. Only generate item docs. * `--merge=finalize` (`!read_rendered_cci && write_rendered_cci`) outputs crate info based only on the current crate and `--include-parts-dir`'ed crates. * A (`read_rendered_cci && !write_rendered_cci`) mode would be useless, since the data that is read would be ignored and not written. From d7ea3b5f1eb86aad3d25c9d1e0dd4b4515ea7395 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Sat, 3 Aug 2024 05:18:14 +0000 Subject: [PATCH 22/28] del --include-rendered-docs, doc.parts suggestions * Remove the --include-rendered-docs flag * --parts-out-dir restricted to --merge=none * --include-parts-dir restricted to --merge=finalize * crate-info.json -> crate-info to emphasize opaqueness * doc.parts contents edition stability promises are replaced with a statement saying that rustdoc will make an reasonable effort to continue just having crate-info files --- ...0000-mergeable-rustdoc-cross-crate-info.md | 96 ++++++++++--------- 1 file changed, 50 insertions(+), 46 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index 9195b02b9ce..a8de8345cbb 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -29,15 +29,15 @@ These considerations motivate adding an option for outputting partial CCI (parts More details are in the Reference-level explanation. -* `--parts-out-dir=doc.parts/`: Write information from the current crate to a provided directory. -* `--include-parts-dir=doc.parts/`: Include cross-crate information from these previously written `doc.parts` directories into a collection that will be written by the current invocation of rustdoc. -* `--merge=none`: Do not write cross-crate information to the `--out-dir`. -* `--merge=shared`: Read cross-crate information from the `--out-dir`, and write cross-crate information from the current crate and any crates included via `--include-parts-dir` to the `--out-dir`. -* `--merge=finalize`: Write cross-crate information from the current crate and any crates included via `--include-parts-dir`, overwriting conflicting files in the `--out-dir`. +* `--parts-out-dir=path/to/doc.parts/`: Write cross-crate linking information to the given directory (only usable with the `--merge=none` mode). This information allows linking the current crate's documentation with other documentation at a later rustdoc invocation. +* `--include-parts-dir=path/to/doc.parts/`: Include cross-crate information from this previously written `doc.parts` directories into a collection that will be written by the current invocation of rustdoc. May only be provided with `--merge=finalize`. May be provided any number of times. +* `--merge=none`: Do not write cross-crate information to the `--out-dir`. The flag `--parts-out-dir` may instead be provided with the destination of the current crate's cross-crate information parts. +* `--merge=shared` (default): Append information from the current crate to any info files found in the `--out-dir`. +* `--merge=finalize`: Write cross-crate information from the current crate and any crates included via `--include-parts-dir` to the `--out-dir`, overwriting conflicting files. ## Example -In this example, there is a crate `trait-crate` which defines a trait `Trait`, and a crate `struct-crate` which defines a struct `Struct` that implements `Trait`. Our goal in this demo is for `Struct` to appear as an implementer in `Trait`'s docs, even if `struct-crate` and `trait-crate` are documented independently. This guide will be assuming that we want a crate `index-crate` that serves as our documentation index. See the Unresolved questions section for ideas that do not require an index crate. +In this example, there is a crate `trait-crate` which defines a trait `Trait`, and a crate `struct-crate` which defines a struct `Struct` that implements `Trait`. Our goal in this demo is for `Struct` to appear as an implementer in `Trait`'s docs, even if `struct-crate` and `trait-crate` are documented independently. This guide will assume that we want a crate `index-crate` that serves as our documentation index. See the Unresolved questions section for ideas that do not require an index crate. ```shell mkdir -p trait-crate/src struct-crate/src index-crate/src merged/doc @@ -98,7 +98,7 @@ rustdoc \ struct-crate/src/lib.rs ``` -Link everything with a final invocation of rustdoc on `index-crate`. We will provide `--merge=finalize`, `--include-parts-dir`, and `--include-rendered-docs`. See the Reference-level explanation about these flags. +Link everything with a final invocation of rustdoc on `index-crate`. We will provide `--merge=finalize`, and `--include-parts-dir`. See the Reference-level explanation about these flags. ```shell rustdoc \ @@ -116,12 +116,16 @@ rustdoc \ --include-parts-dir=struct-crate/target/doc.parts/struct-crate \ --extern trait-crate=trait-crate/target/libt.rmeta \ --extern struct-crate=struct-crate/target/libs.rmeta \ - --include-rendered-docs=trait-crate/target/doc/trait-crate \ - --include-rendered-docs=struct-crate/target/doc/struct-crate \ -L trait-crate/target \ index-crate/src/lib.rs ``` +Copy the docs from external locations to the given `--out-dir`. + +```shell +cp -r struct-crate/target/doc/* trait-crate/target/doc/* index-crate/target/doc/* merged/doc +``` + Browse `merged/doc/index.html` with cross-crate information. In general, instead of two crates in the environment (`struct-crate` and `trait-crate`) you could have thousands. Upon any changes, only the index and the crates that are changed have to be re-documented. @@ -245,7 +249,7 @@ $ tree . -a │ │ └── lib.rs.html │ ├── doc.parts │ │ └── struct-crate -│ │ └── crate-info.json +│ │ └── crate-info │ └── libs.rmeta └── trait-crate ├── src @@ -268,7 +272,7 @@ $ tree . -a │ └── trait.Trait.html ├── doc.parts │ └── trait-crate - │ └── crate-info.json + │ └── crate-info └── libt.rmeta @@ -276,43 +280,54 @@ $ tree . -a ## Suggested workflows -With this proposal, there are three modes of invoking rustdoc. These modes are configured through the choice of the `--merge`, `--parts-out-dir`, `--include-parts-dir`, and `--include-rendered-docs` flags. +With this proposal, there are three modes of invoking rustdoc: `--merge=shared`, `--merge=none`, and `--merge=finalize`. -### Default workflow: mutate shared directory +### Default workflow: mutate shared directory: `--merge=shared` In this workflow, we document a single crate, or a collection of crates into a shared output directory that is continuously updated. Files in this output directory are modified by multiple rustdoc invocations. Use `--merge=shared`, and specify the same `--out-dir` to every invocation of rustdoc. `--merge=shared` will be the default value if `--merge` is not provided. This is the workflow that Cargo uses, and only mode of invoking rustdoc before this RFC. -### Document intermediate crates +### Document intermediate crates: `--merge=none` -Document regular (non-root, non-index) crates using a dedicated HTML output directory and a dedicated "parts" output directory. No cross-crate data nor rendered HTML output is included from other crates. +Document non-root, non-index crates using a dedicated HTML output directory and a dedicated "parts" output directory. No cross-crate data nor rendered HTML output is included from other crates. This mode only renders the HTML item documentation for the current crate. It does not produce a search index, cross-crate trait implementations, or an index page. It is expected that users follow this mode with 'Document a final crate' if these cross-crate features are desired. In this mode, a user may specify a different `--out-dir` to every invocation of rustdoc. Additionally, a user will provide `--parts-out-dir=` and `--merge=none` when documenting every crate. The user should provide `--extern-html-root-url`, and specify a absolute final location for the URL, if they document crates in separate `--out-dir`s. This flag, with the same URL, will be needed for every invocation of rustdoc, for every dependency. -### Document a final crate +### Document a final crate: `---merge=finalized` In this context, a final crate is a crate that depends directly on every crate that a user intends to appear in the documentation bundle. It may be an index crate that has no meaningful functionality on its own. It may also be a library crate that depends on every crate in a workspace. In this mode, rendered HTML and *finalized* cross-crate information are generated into a `target/doc/my-final-crate` folder. No *incremental* parts are generated (i.e., no `target/doc.parts/my-final-crate`). -When a user documents the final crate, they will provide `--include-parts-dir=`, `--include-rendered-docs=` for each one of the dependencies, and `--merge=finalize`. They will provide `--extern-html-root-url`, in the way described in 'Document an intermediate crate'. +When a user documents the final crate, they will provide `--include-parts-dir=` for each one of the dependencies, and `--merge=finalize`. They will provide `--extern-html-root-url`, in the way described in 'Document an intermediate crate'. # Reference-level explanation The existing cross-crate information files, like `search-index.js`, all are lists of elements, rendered in an specified way (e.g. as a JavaScript file with a JSON array or an HTML index page containing an unordered list). The current rustdoc (in `write_shared`) pushes the current crate's version of the CCI into the one that is already found in `doc`, and renders a new version. The rest of the proposal uses the term **part** to refer to the pre-merged, pre-rendered element of the CCI. This proposal does not add any new CCI or change their contents (modulo sorting order, whitespace). -## New directory: `doc.parts//crate-info.json` +## New flag: `--merge=none|shared|finalize` + +This flag corresponds to the three modes of invoking rustdoc described in 'Suggested workflows'. It controls two internal paramaters: `read_rendered_cci`, and `write_rendered_cci`. It also gates whether the user is allowed to provide the `--parts-out-dir` and `--include-parts-dir` flags. -`doc.parts` is a directory that holds the partial contents and destination of several cross-crate information files. It only encodes information about a single-crate. This file is written if `--parts-out-dir` is provided. The current crate's information and any `doc.parts` added through `--include-parts-dir` are merged and rendered if `--merge=shared` or `--merge=finalize` are provided. +When `write_rendered_cci` is active, rustdoc outputs the rendered parts to the doc root (`--out-dir`). Rustdoc will generate files like `doc/search-index.js`, `doc/search.desc`, `doc/index.html`, etc if and only if this parameter is true. -The directory `doc.parts` contains subdirectories for each crate. The crate names are chosen by the caller that invokes rustdoc, and are `--include-parts-dir` and `--parts-out-dir`. These crate names are arbitrary from the perspective of rustdoc, with the constraint that they must be unique. Rustdoc will write a file called `crate-info.json`, whose contents are unstable, to the subdirectory provided to `--parts-out-dir`. Even though the contents of `crate-info.json` are only intended to be consumed by rustdoc, its existence must be stabilized due to the fact that it may be declared to build systems. +When `read_rendered_cci` is active, rustdoc will look in the `--out-dir` for rendered cross-crate info files. These files will be used as the base. Any new parts that rustdoc generates with its current invocation and any parts fetched with `include-parts-dir` will be appended to these base files. When it is disabled, the cross-crate info files start empty and are populated with the current crate's info and any crates fetched with `--include-parts-dir`. + +* `--merge=shared` (`read_rendered_cci && write_rendered_cci`) is the default, and reflects the current behavior of rustdoc. Rustdoc will look in its `--out-dir` for pre-existing cross-crate information files, and append information to these files from the current crate. The user is not allowed to provide `--parts-out-dir` or `--include-parts-dir` in this mode. +* `--merge=none` (`!read_rendered_cci && !write_rendered_cci`) means that rustdoc will ignore the cross-crate files in the doc root. It only generates item docs. The user is optionally allowed to include `--parts-out-dir`, but not `--include-parts-dir`. +* `--merge=finalize` (`!read_rendered_cci && write_rendered_cci`) outputs crate info based only on the current crate and `--include-parts-dir`'ed crates. The user is optionally allowed to include `--include-parts-dir`, but not `--parts-out-dir`. +* A (`read_rendered_cci && !write_rendered_cci`) mode would be useless, since the data that is read would be ignored and not written. -This directory structure may change with a new edition. +The use of `--include-parts-dir` and `--parts-out-dir` is gated by `--merge` in order to prevent meaningless invocations, detect user error, and to provide for future changes to the interface. -Rustdoc only guarantees that it will accept `doc.parts` files written by the same version of rustdoc, and rustdoc is the only explicitly supported consumer of `doc.parts`. Only the presence of `doc.parts`, and its opaque child `crate-info.json`, are stabilized. Non-normatively, there are several pieces of information that `doc.parts` may contain: +## New directory: `doc.parts//crate-info` + +`doc.parts` is the suggested name for the parent of the subdirectory that the user provides to `--parts-out-dir` and `--include-parts-dir`. A unique subdirectory for each crate must be provided to `--parts-out-dir` and `--include-parts-dir`. The user is encouraged to chose a directory outside of the `--out-dir`, as `--parts-out-dir` writes intermediate information that is not intended to be served on a static doc server. + +Rustdoc only guarantees that it will accept `doc.parts` files written by the same version of rustdoc. Rustdoc is the only explicitly supported consumer of `doc.parts`. In the initial implementation, rustdoc will write a file called `crate-info` as a child of the directory provided to `--parts-out-dir`, and an reasonable effort will be made for this to continue to be the structure of the subdirectory. However, the contents of `--parts-out-dir` are considered formally unstable. Non-normatively, there are several pieces of information that `doc.parts` may contain: * Partial source file index for generating `doc/src-files.js`. * Partial search index for generating `doc/search-index.js`. @@ -323,40 +338,27 @@ Rustdoc only guarantees that it will accept `doc.parts` files written by the sam * The file may include versioning information intended to assist in generating error messages if an incompatible `doc.parts` is provided through `--include-parts-dir`. * The file may contain other information related to cross-crate information that is added in the future. -## New flag: `--parts-out-dir=/doc.parts/` +## New flag: `--parts-out-dir=path/to/doc.parts/` + +When this flag is provided, the unmerged parts for the current crate will be written to `path/to/doc.parts/`. A typical argument is `./target/doc.parts/rand`. -When this flag is provided, the unmerged parts for the current crate will be written to `path/to/doc.parts/`. A typical argument is `./target/doc.parts/rand`. +This flag may only be used in the `--merge=none` mode. Crates `--include-parts-dir`ed will not appear in `doc.parts`, as `doc.parts` only includes the CCI parts for the current crate. If this flag is not provided, no `doc.parts` will be written. -The output generated by this flag may be consumed by a future invocation to rustdoc that provides `--include-parts-dir=/doc.parts/`. +The output generated by this flag may be consumed by a future invocation to rustdoc that provides `--include-parts-dir=path/to/doc.parts/`. -## New flag: `--include-parts-dir=` +## New flag: `--include-parts-dir=path/to/doc.parts/` -If this flag is provided, rustdoc will expect that a previous invocation of rustdoc was made with `--parts-out-dir=`. It will append the parts from the previous invocation to the ones it will render in the doc root (`--out-dir`). The info that's included is not written to its own `doc.parts`, as `doc.parts` only holds the CCI parts for the current crate. +If this flag is provided, rustdoc will expect that a previous invocation of rustdoc was made with `--parts-out-dir=path/to/doc.parts/`. It will append the parts from the previous invocation to the ones it will render in the doc root (`--out-dir`). The info that's included is not written to its own `doc.parts`, as `doc.parts` only holds the CCI parts for the current crate. -This flag is similar to `--extern-html-root-url` in that it only needs to be provided for externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided in `--extern-html-root-url` never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--include-parts-dir` tells rustdoc where to search for the `doc.parts` directory at documentation-time. It must not be a URL. - -In the Guide-level explanation, for example, crate `i` needs to identify the location of `s`'s parts. Since they could be located in an arbitrary directory, `i` must be instructed on where to fetch them. In this example, `s`'s parts happen to be in `./s/target/doc.parts/s`, so rustdoc is called with `--include-parts-dir=s/target/doc.parts/s`. - -## New flag: `--include-rendered-docs=` - -Rustdoc will assume that `` was used as the `--out-dir` for ``. This documentation will be copied into the directory specified by `--out-dir`. Rustdoc will effectively run `cp -r `. - -## New flag: `--merge=none|shared|finalize` +This flag may only be used in the `--merge=finalize` mode. -This flag corresponds to the three modes of invoking rustdoc described in 'Suggested workflows'. It controls two internal paramaters: `read_rendered_cci`, and `write_rendered_cci`. +In the Guide-level explanation, for example, the `index-crate` needs to identify the location of the `struct-crate`'s parts. Since they could be located in an arbitrary directory, the `index-crate` must be instructed on where to fetch them. In this example, the `struct-crate`'s parts happen to be in `./struct-crate/target/doc.parts/struct-crate`, so rustdoc is called with `--include-parts-dir=s/target/doc.parts/s`. -When `write_rendered_cci` is active, rustdoc will output the rendered parts to the doc root (`--out-dir`). Rustdoc will generate files like `doc/search-index.js`, `doc/search.desc`, `doc/index.html`, etc if and only if this parameter is true. - -When `read_rendered_cci` is active, rustdoc will look in the `--out-dir` for rendered cross-crate info files. These files will be used as the base. Any new parts that rustdoc generates with its current invocation and any parts fetched with `include-parts-dir` will be appended to these base files. When it is disabled, the cross-crate info files start empty and are populated with the current crate's info and any crates fetched with `--include-parts-dir`. - -* `--merge=shared` (`read_rendered_cci && write_rendered_cci`) is the default, and reflects the current behavior of rustdoc. Rustdoc will look in its `--out-dir` for pre-existing cross-crate information files, and append information to these files from the current crate and any crates referenced through `--include-parts-dir`. -* `--merge=none` (`!read_rendered_cci && !write_rendered_cci`) means that rustdoc will ignore the cross-crate files in the doc root. Only generate item docs. -* `--merge=finalize` (`!read_rendered_cci && write_rendered_cci`) outputs crate info based only on the current crate and `--include-parts-dir`'ed crates. -* A (`read_rendered_cci && !write_rendered_cci`) mode would be useless, since the data that is read would be ignored and not written. +This flag is similar to `--extern-html-root-url` in that it only needs to be provided for externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided in `--extern-html-root-url` never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--include-parts-dir` tells rustdoc where to search for the `doc.parts` directory at documentation-time. It must not be a URL. ## Merge step @@ -364,7 +366,9 @@ This proposal is capable of addressing two primary use cases. It allows develope * Documenting a crate and its transitive dependencies in parallel in build systems that require build actions to be independent * Producing a documentation index of a large number of crates, in such a way that if one crate is updated, only the updated crates and an index have to be redocumented. This scenario is demonstrated in the Guide-level explanation. -CCI is not automatically enabled in either situation. A combination of the `--include-parts-dir`, `--merge`, and `--parts-out-dir` flags are needed to produce this behavior. This RFC provides a minimal set of tools that allow developers of build systems, like Bazel and Buck2, to create rules for these scenarios. +CCI is not automatically enabled in either situation. A combination of the `--include-parts-dir`, `--merge`, and `--parts-out-dir` flags are needed to produce this behavior. This RFC provides a minimal set of tools that allow developers of build systems, like Bazel and Buck2, to create rules for these scenarios. + +With separate `--out-dir`s, copying item docs to an output destination is needed. Rustdoc will never support the entire breadth of workflows needed to merge arbitrary directories, and will rely on users to run external commands like `mv`, `cp`, `rsync`, `scp`, etc. for these purposes. Discussion of whether additional features should be included to facilitate this merge step can be found in Unresolved questions (Index crate). From 628c7a9edd873001308575fc4d40978718436a4b Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Sat, 3 Aug 2024 05:37:35 +0000 Subject: [PATCH 23/28] fix typo in merge sentence --- text/0000-mergeable-rustdoc-cross-crate-info.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index a8de8345cbb..f147c6b623c 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -120,7 +120,7 @@ rustdoc \ index-crate/src/lib.rs ``` -Copy the docs from external locations to the given `--out-dir`. +Copy the docs from the given `--out-dir`s to a central location. ```shell cp -r struct-crate/target/doc/* trait-crate/target/doc/* index-crate/target/doc/* merged/doc From c765097118c19a00ced3b97b384023808fdc7fc6 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Mon, 12 Aug 2024 22:18:54 -0700 Subject: [PATCH 24/28] editorial changes in response to @jsha * New directory: `doc.parts//crate-info` -> New directory: `doc.parts`. * Number of times `--parts-out-dir`, `--merge`, and `--include-parts-dir` can be provided are mentioned. * Describe why `doc.parts` is a directory. * Description of multiple parallel invocations of rustdoc. * Describe why to run rustc before rustdoc. * Fix typos. --- ...0000-mergeable-rustdoc-cross-crate-info.md | 24 ++++++++++--------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index f147c6b623c..c310b8c1a85 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -52,7 +52,7 @@ MERGED=file://$(realpath merged/doc) echo "extern crate trait-crate; extern crate struct-crate;" > index-crate/src/lib.rs ``` -Compile the crates. +Compile the crates. This will produce the `.rmeta` files that rustdoc requires to depend on a crate. ```shell rustc \ @@ -68,7 +68,8 @@ rustc \ --edition=2021 \ --emit=metadata \ --out-dir=struct-crate/target \ - --extern trait-crate=trait-crate/target/libt.rmeta struct-crate/src/lib.rs + --extern trait-crate=trait-crate/target/libt.rmeta \ + struct-crate/src/lib.rs ``` Document `struct-crate` and `trait-crate` independently, providing `--merge=none`, `--parts-out-dir`. @@ -293,12 +294,13 @@ Document non-root, non-index crates using a dedicated HTML output directory and This mode only renders the HTML item documentation for the current crate. It does not produce a search index, cross-crate trait implementations, or an index page. It is expected that users follow this mode with 'Document a final crate' if these cross-crate features are desired. -In this mode, a user may specify a different `--out-dir` to every invocation of rustdoc. Additionally, a user will provide `--parts-out-dir=` and `--merge=none` when documenting every crate. -The user should provide `--extern-html-root-url`, and specify a absolute final location for the URL, if they document crates in separate `--out-dir`s. This flag, with the same URL, will be needed for every invocation of rustdoc, for every dependency. +In this mode, a user will provide `--parts-out-dir=` and `--merge=none` to each crate's rustdoc invocation. The user should provide `--extern-html-root-url`, and specify a absolute final destination for the docs, as a URL. The `--extern-html-root-url` flag should be provided for each crate's rustdoc invocation, for every dependency. + +The same `--out-dir` may be used for multiple parallel rustdoc invocations, as rustdoc will continue to acquire an flock to prevent conflicts. A user may select a different `--out-dir` for each crate's rustdoc invocation. If so, the user must merge the docs to a central location (e.g. `cp -r crate1/doc crate2/doc crate3/doc destination`) after 'Document a final crate'. ### Document a final crate: `---merge=finalized` -In this context, a final crate is a crate that depends directly on every crate that a user intends to appear in the documentation bundle. It may be an index crate that has no meaningful functionality on its own. It may also be a library crate that depends on every crate in a workspace. +In this context, a final crate is a crate that depends directly on every crate that a user intends to appear in the documentation bundle. It may be an index crate that has no meaningful functionality on its own. It may also be a library crate that depends on several crates. In this mode, rendered HTML and *finalized* cross-crate information are generated into a `target/doc/my-final-crate` folder. No *incremental* parts are generated (i.e., no `target/doc.parts/my-final-crate`). @@ -310,7 +312,7 @@ The existing cross-crate information files, like `search-index.js`, all are list ## New flag: `--merge=none|shared|finalize` -This flag corresponds to the three modes of invoking rustdoc described in 'Suggested workflows'. It controls two internal paramaters: `read_rendered_cci`, and `write_rendered_cci`. It also gates whether the user is allowed to provide the `--parts-out-dir` and `--include-parts-dir` flags. +This flag corresponds to the three modes of invoking rustdoc described in 'Suggested workflows'. It controls two internal paramaters: `read_rendered_cci`, and `write_rendered_cci`. It also gates whether the user is allowed to provide the `--parts-out-dir` and `--include-parts-dir` flags. It can be provided at most once. When `write_rendered_cci` is active, rustdoc outputs the rendered parts to the doc root (`--out-dir`). Rustdoc will generate files like `doc/search-index.js`, `doc/search.desc`, `doc/index.html`, etc if and only if this parameter is true. @@ -323,11 +325,11 @@ When `read_rendered_cci` is active, rustdoc will look in the `--out-dir` for ren The use of `--include-parts-dir` and `--parts-out-dir` is gated by `--merge` in order to prevent meaningless invocations, detect user error, and to provide for future changes to the interface. -## New directory: `doc.parts//crate-info` +## New directory: `doc.parts/` `doc.parts` is the suggested name for the parent of the subdirectory that the user provides to `--parts-out-dir` and `--include-parts-dir`. A unique subdirectory for each crate must be provided to `--parts-out-dir` and `--include-parts-dir`. The user is encouraged to chose a directory outside of the `--out-dir`, as `--parts-out-dir` writes intermediate information that is not intended to be served on a static doc server. -Rustdoc only guarantees that it will accept `doc.parts` files written by the same version of rustdoc. Rustdoc is the only explicitly supported consumer of `doc.parts`. In the initial implementation, rustdoc will write a file called `crate-info` as a child of the directory provided to `--parts-out-dir`, and an reasonable effort will be made for this to continue to be the structure of the subdirectory. However, the contents of `--parts-out-dir` are considered formally unstable. Non-normatively, there are several pieces of information that `doc.parts` may contain: +Rustdoc only guarantees that it will accept `doc.parts` files written by the same version of rustdoc. Rustdoc is the only explicitly supported consumer of `doc.parts`. In the initial implementation, rustdoc will write a file called `crate-info` as a child of the directory provided to `--parts-out-dir`, and an reasonable effort will be made for this to continue to be the structure of the subdirectory. However, the contents of `--parts-out-dir` are considered formally unstable, leaving open the possible future addition of other related files. Non-normatively, there are several pieces of information that `doc.parts` may contain: * Partial source file index for generating `doc/src-files.js`. * Partial search index for generating `doc/search-index.js`. @@ -342,7 +344,7 @@ Rustdoc only guarantees that it will accept `doc.parts` files written by the sam When this flag is provided, the unmerged parts for the current crate will be written to `path/to/doc.parts/`. A typical argument is `./target/doc.parts/rand`. -This flag may only be used in the `--merge=none` mode. +This flag may only be used in the `--merge=none` mode. It is optional, and may be provided at most one time. Crates `--include-parts-dir`ed will not appear in `doc.parts`, as `doc.parts` only includes the CCI parts for the current crate. @@ -354,9 +356,9 @@ The output generated by this flag may be consumed by a future invocation to rust If this flag is provided, rustdoc will expect that a previous invocation of rustdoc was made with `--parts-out-dir=path/to/doc.parts/`. It will append the parts from the previous invocation to the ones it will render in the doc root (`--out-dir`). The info that's included is not written to its own `doc.parts`, as `doc.parts` only holds the CCI parts for the current crate. -This flag may only be used in the `--merge=finalize` mode. +This flag may only be used in the `--merge=finalize` mode. It is optional, and can be provided any number of times (once per crate whose documentation is merged). -In the Guide-level explanation, for example, the `index-crate` needs to identify the location of the `struct-crate`'s parts. Since they could be located in an arbitrary directory, the `index-crate` must be instructed on where to fetch them. In this example, the `struct-crate`'s parts happen to be in `./struct-crate/target/doc.parts/struct-crate`, so rustdoc is called with `--include-parts-dir=s/target/doc.parts/s`. +In the Guide-level explanation, for example, the `index-crate` needs to identify the location of the `struct-crate`'s parts. Since they could be located in an arbitrary directory, the `index-crate` must be instructed on where to fetch them. In this example, the `struct-crate`'s parts happen to be in `./struct-crate/target/doc.parts/struct-crate`, so rustdoc is called with `--include-parts-dir=struct-crate/target/doc.parts/struct-crate`. This flag is similar to `--extern-html-root-url` in that it only needs to be provided for externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided in `--extern-html-root-url` never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--include-parts-dir` tells rustdoc where to search for the `doc.parts` directory at documentation-time. It must not be a URL. From ac9208c9bb713bc6172839968b6bb8c7cb54a7e6 Mon Sep 17 00:00:00 2001 From: EtomicBomb Date: Tue, 13 Aug 2024 21:15:15 +0000 Subject: [PATCH 25/28] resolve in favor of no index crate * remove 'Unresolved: index crate?' from the unresolved questions section * remove the requirement on the index crate from various sections --- ...0000-mergeable-rustdoc-cross-crate-info.md | 136 ++++-------------- 1 file changed, 28 insertions(+), 108 deletions(-) diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md index c310b8c1a85..9d48971138c 100644 --- a/text/0000-mergeable-rustdoc-cross-crate-info.md +++ b/text/0000-mergeable-rustdoc-cross-crate-info.md @@ -23,6 +23,8 @@ Rustdoc needing global mutable access to the files that encode this cross-crate These considerations motivate adding an option for outputting partial CCI (parts), which are merged (linked) with a later step. +This RFC has the goal of enabling the future deprecation of the default (called `--merge=shared` here) practice of appending to cross-crate information files in the doc root. + # Guide-level explanation ## New flag summary @@ -33,26 +35,20 @@ More details are in the Reference-level explanation. * `--include-parts-dir=path/to/doc.parts/`: Include cross-crate information from this previously written `doc.parts` directories into a collection that will be written by the current invocation of rustdoc. May only be provided with `--merge=finalize`. May be provided any number of times. * `--merge=none`: Do not write cross-crate information to the `--out-dir`. The flag `--parts-out-dir` may instead be provided with the destination of the current crate's cross-crate information parts. * `--merge=shared` (default): Append information from the current crate to any info files found in the `--out-dir`. -* `--merge=finalize`: Write cross-crate information from the current crate and any crates included via `--include-parts-dir` to the `--out-dir`, overwriting conflicting files. +* `--merge=finalize`: Write cross-crate information from the current crate and any crates included via `--include-parts-dir` to the `--out-dir`, overwriting conflicting files. This flag may be used with or without an input crate root, as it can links existing docs. ## Example -In this example, there is a crate `trait-crate` which defines a trait `Trait`, and a crate `struct-crate` which defines a struct `Struct` that implements `Trait`. Our goal in this demo is for `Struct` to appear as an implementer in `Trait`'s docs, even if `struct-crate` and `trait-crate` are documented independently. This guide will assume that we want a crate `index-crate` that serves as our documentation index. See the Unresolved questions section for ideas that do not require an index crate. +In this example, there is a crate `trait-crate` which defines a trait `Trait`, and a crate `struct-crate` which defines a struct `Struct` that implements `Trait`. Our goal in this demo is for `Struct` to appear as an implementer in `Trait`'s docs, even if `struct-crate` and `trait-crate` are documented independently. ```shell -mkdir -p trait-crate/src struct-crate/src index-crate/src merged/doc +mkdir -p trait-crate/src struct-crate/src merged/doc echo "pub trait Trait {}" > trait-crate/src/lib.rs echo "pub struct Struct; impl trait-crate::Trait for Struct {}" > struct-crate/src/lib.rs MERGED=file://$(realpath merged/doc) ``` -[Actively use](https://doc.rust-lang.org/rustc/command-line-arguments.html#--extern-specify-where-an-external-library-is-located) `trait-crate` and `struct-crate` in `index-crate`. The `extern crate` declarations are not needed if the crates are otherwise referenced in the index; intra-doc links are enough. - -```shell -echo "extern crate trait-crate; extern crate struct-crate;" > index-crate/src/lib.rs -``` - -Compile the crates. This will produce the `.rmeta` files that rustdoc requires to depend on a crate. +Compile `trait-crate`, so that `struct-crate` can depend on its `.rmeta` file. ```shell rustc \ @@ -62,21 +58,12 @@ rustc \ --emit=metadata \ --out-dir=trait-crate/target \ trait-crate/src/lib.rs -rustc \ - --crate-name=struct-crate \ - --crate-type=lib \ - --edition=2021 \ - --emit=metadata \ - --out-dir=struct-crate/target \ - --extern trait-crate=trait-crate/target/libt.rmeta \ - struct-crate/src/lib.rs ``` -Document `struct-crate` and `trait-crate` independently, providing `--merge=none`, `--parts-out-dir`. +Document `struct-crate` and `trait-crate` independently, providing `--merge=none`, and `--parts-out-dir`. ```shell rustdoc \ - -Z unstable-options \ --crate-name=trait-crate \ --crate-type=lib \ --edition=2021 \ @@ -86,7 +73,6 @@ rustdoc \ --parts-out-dir=trait-crate/target/doc.parts/trait-crate \ trait-crate/src/lib.rs rustdoc \ - -Z unstable-options \ --crate-name=struct-crate \ --crate-type=lib \ --edition=2021 \ @@ -99,37 +85,26 @@ rustdoc \ struct-crate/src/lib.rs ``` -Link everything with a final invocation of rustdoc on `index-crate`. We will provide `--merge=finalize`, and `--include-parts-dir`. See the Reference-level explanation about these flags. +Link everything with a final invocation of rustdoc. We will provide `--merge=finalize`, and `--include-parts-dir`. See the Reference-level explanation about these flags. Notice that this invocation is given no source input file. ```shell rustdoc \ - -Z unstable-options \ - --crate-name=index-crate \ - --crate-type=lib \ - --edition=2021 \ --enable-index-page \ - --out-dir=index-crate/target/doc \ - --extern-html-root-url struct-crate=$MERGED \ - --extern-html-root-url trait-crate=$MERGED \ - --extern-html-root-url index-crate=$MERGED \ - --merge=finalize \ --include-parts-dir=trait-crate/target/doc.parts/trait-crate \ --include-parts-dir=struct-crate/target/doc.parts/struct-crate \ - --extern trait-crate=trait-crate/target/libt.rmeta \ - --extern struct-crate=struct-crate/target/libs.rmeta \ - -L trait-crate/target \ - index-crate/src/lib.rs + --out-dir=merged/doc \ + --merge=finalize ``` Copy the docs from the given `--out-dir`s to a central location. ```shell -cp -r struct-crate/target/doc/* trait-crate/target/doc/* index-crate/target/doc/* merged/doc +cp -r struct-crate/target/doc/* trait-crate/target/doc/* merged/doc ``` Browse `merged/doc/index.html` with cross-crate information. -In general, instead of two crates in the environment (`struct-crate` and `trait-crate`) you could have thousands. Upon any changes, only the index and the crates that are changed have to be re-documented. +In general, instead of two crates in the environment (`struct-crate` and `trait-crate`) a user could have thousands. Upon any changes, only the crates that change have to be re-documented.
Click here for a directory listing after running the example above. @@ -137,53 +112,10 @@ In general, instead of two crates in the environment (`struct-crate` and `trait-
 $ tree . -a
 .
-├── index-crate
-│   ├── src
-│   │   └── lib.rs
-│   └── target
-│       └── doc
-│           ├── crates.js
-│           ├── help.html
-│           ├── index-crate
-│           │   ├── all.html
-│           │   ├── index.html
-│           │   └── sidebar-items.js
-│           ├── index.html
-│           ├── .lock
-│           ├── search.desc
-│           │   └── index-crate
-│           │       └── index-crate-desc-0-.js
-│           ├── search-index.js
-│           ├── settings.html
-│           ├── src
-│           │   └── index-crate
-│           │       └── lib.rs.html
-│           ├── src-files.js
-│           ├── static.files
-│           │   ├── COPYRIGHT-23e9bde6c69aea69.txt
-│           │   ├── favicon-2c020d218678b618.svg
-│           │   └── 
-│           └── trait.impl
-│               ├── core
-│               │   ├── marker
-│               │   │   ├── trait.Freeze.js
-│               │   │   ├── trait.Send.js
-│               │   │   ├── trait.Sync.js
-│               │   │   └── trait.Unpin.js
-│               │   └── panic
-│               │       └── unwind_safe
-│               │           ├── trait.RefUnwindSafe.js
-│               │           └── trait.UnwindSafe.js
-│               └── trait-crate
-│                   └── trait.Trait.js
 ├── merged
 │   └── doc
 │       ├── crates.js
 │       ├── help.html
-│       ├── index-crate
-│       │   ├── all.html
-│       │   ├── index.html
-│       │   └── sidebar-items.js
 │       ├── index.html
 │       ├── struct-crate
 │       │   ├── all.html
@@ -191,8 +123,6 @@ $ tree . -a
 │       │   ├── sidebar-items.js
 │       │   └── struct.Struct.html
 │       ├── search.desc
-│       │   ├── index-crate
-│       │   │   └── index-crate-desc-0-.js
 │       │   ├── struct-crate
 │       │   │   └── struct-crate-desc-0-.js
 │       │   └── trait-crate
@@ -200,8 +130,6 @@ $ tree . -a
 │       ├── search-index.js
 │       ├── settings.html
 │       ├── src
-│       │   ├── index-crate
-│       │   │   └── lib.rs.html
 │       │   ├── struct-crate
 │       │   │   └── lib.rs.html
 │       │   └── trait-crate
@@ -286,25 +214,27 @@ With this proposal, there are three modes of invoking rustdoc: `--merge=shared`,
 ### Default workflow: mutate shared directory: `--merge=shared`
 
 In this workflow, we document a single crate, or a collection of crates into a shared output directory that is continuously updated.
-Files in this output directory are modified by multiple rustdoc invocations. Use `--merge=shared`, and specify the same `--out-dir` to every invocation of rustdoc. `--merge=shared` will be the default value if `--merge` is not provided. This is the workflow that Cargo uses, and only mode of invoking rustdoc before this RFC.
+Files in this output directory are modified by multiple rustdoc invocations. Use `--merge=shared`, and specify the same `--out-dir` to every invocation of rustdoc. `--merge=shared` will be the default value if `--merge` is not provided. This is the workflow that Cargo uses, and only mode of invoking rustdoc before this RFC. This RFC is intended to enable the future deprecation of this mode.
 
-### Document intermediate crates: `--merge=none`
+### Document crates, delaying generation of cross-crate information: `--merge=none`
 
-Document non-root, non-index crates using a dedicated HTML output directory and a dedicated "parts" output directory. No cross-crate data nor rendered HTML output is included from other crates.
+Document crates using a dedicated HTML output directory and a dedicated "parts" output directory. No cross-crate data nor rendered HTML output is included from other crates.
 
-This mode only renders the HTML item documentation for the current crate. It does not produce a search index, cross-crate trait implementations, or an index page. It is expected that users follow this mode with 'Document a final crate' if these cross-crate features are desired.
+This mode only renders the HTML item documentation for the current crate. It does not produce a search index, cross-crate trait implementations, or an index page. It is expected that users follow this mode with 'Link documentation' if these cross-crate features are desired.
 
 In this mode, a user will provide `--parts-out-dir=` and `--merge=none` to each crate's rustdoc invocation. The user should provide `--extern-html-root-url`, and specify a absolute final destination for the docs, as a URL. The `--extern-html-root-url` flag should be provided for each crate's rustdoc invocation, for every dependency.
 
-The same `--out-dir` may be used for multiple parallel rustdoc invocations, as rustdoc will continue to acquire an flock to prevent conflicts. A user may select a different `--out-dir` for each crate's rustdoc invocation. If so, the user must merge the docs to a central location (e.g. `cp -r crate1/doc crate2/doc crate3/doc destination`) after 'Document a final crate'.
+The same `--out-dir` may be used for multiple parallel rustdoc invocations, as rustdoc will continue to acquire an flock on the `--out-dir` to address conflicts. A user may select a different `--out-dir` for each crate's rustdoc invocation. 
 
-### Document a final crate: `---merge=finalized`
+### Link documentation: `--merge=finalize`
 
-In this context, a final crate is a crate that depends directly on every crate that a user intends to appear in the documentation bundle. It may be an index crate that has no meaningful functionality on its own. It may also be a library crate that depends on several crates.
+In this mode, rendered HTML and *finalized* cross-crate information are generated into a `doc` folder. No *incremental* parts are generated (i.e., no `target/doc.parts/my-final-crate`).
 
-In this mode, rendered HTML and *finalized* cross-crate information are generated into a `target/doc/my-final-crate` folder. No *incremental* parts are generated (i.e., no `target/doc.parts/my-final-crate`).
+This flag can be used with or without an target crate root. When used with a target crate, the parts for the target crate are included in the final docs. Otherwise, this mode functions merely to merge the input docs.
 
-When a user documents the final crate, they will provide  `--include-parts-dir=` for each one of the dependencies, and `--merge=finalize`. They will provide `--extern-html-root-url`, in the way described in 'Document an intermediate crate'.
+When a user documents the final crate, they will provide  `--include-parts-dir=` for each crate whose documentation is being combined, and `--merge=finalize`.
+
+The user must merge every distinct `--out-dir` selected during the `--merge=none`, (e.g. `cp -r crate1/doc crate2/doc crate3/doc destination`). Most workspaces are expected to use a single `--out-dir`, so no manual merging is needed.
 
 # Reference-level explanation
 
@@ -358,7 +288,7 @@ If this flag is provided, rustdoc will expect that a previous invocation of rust
 
 This flag may only be used in the `--merge=finalize` mode. It is optional, and can be provided any number of times (once per crate whose documentation is merged).
 
-In the Guide-level explanation, for example, the `index-crate` needs to identify the location of the `struct-crate`'s parts. Since they could be located in an arbitrary directory, the `index-crate` must be instructed on where to fetch them. In this example, the `struct-crate`'s parts happen to be in `./struct-crate/target/doc.parts/struct-crate`, so rustdoc is called with `--include-parts-dir=struct-crate/target/doc.parts/struct-crate`.
+In the Guide-level explanation, for example, the final invocation of rustdoc needs to identify the location of the `struct-crate`'s parts. Since they could be located in an arbitrary directory, the final invocation must be instructed on where to fetch them. In this example, the `struct-crate`'s parts happen to be in `./struct-crate/target/doc.parts/struct-crate`, so rustdoc is called with `--include-parts-dir=struct-crate/target/doc.parts/struct-crate`.
 
 This flag is similar to `--extern-html-root-url` in that it only needs to be provided for externally documented crates. The flag `--extern-html-root-url` controls hyperlink generation. The hyperlink provided in `--extern-html-root-url` never accessed by rustdoc, and represents the final destination of the documentation. The new flag `--include-parts-dir` tells rustdoc where to search for the `doc.parts` directory at documentation-time. It must not be a URL.
 
@@ -366,13 +296,11 @@ This flag is similar to `--extern-html-root-url` in that it only needs to be pro
 
 This proposal is capable of addressing two primary use cases. It allows developers to enable CCI in these scenarios:
 * Documenting a crate and its transitive dependencies in parallel in build systems that require build actions to be independent
-* Producing a documentation index of a large number of crates, in such a way that if one crate is updated, only the updated crates and an index have to be redocumented. This scenario is demonstrated in the Guide-level explanation.
+* Producing a documentation index of every crate in a workspace, in such a way that if one crate is updated, only the updated crates and an index have to be redocumented. This scenario is demonstrated in the Guide-level explanation.
 
 CCI is not automatically enabled in either situation. A combination of the `--include-parts-dir`, `--merge`, and `--parts-out-dir` flags are needed to produce this behavior. This RFC provides a minimal set of tools that allow developers of build systems, like Bazel and Buck2, to create rules for these scenarios. 
 
-With separate `--out-dir`s, copying item docs to an output destination is needed. Rustdoc will never support the entire breadth of workflows needed to merge arbitrary directories, and will rely on users to run external commands like `mv`, `cp`, `rsync`, `scp`, etc. for these purposes.
-
-Discussion of whether additional features should be included to facilitate this merge step can be found in Unresolved questions (Index crate).
+With separate `--out-dir`s, copying item docs to an output destination is needed. Rustdoc will never support the entire breadth of workflows needed to merge arbitrary directories, and will rely on users to run external commands like `mv`, `cp`, `rsync`, `scp`, etc. for these purposes. Most users are expected to use a single `--out-dir` for all crates, in which case these external tools are not needed.
 
 ## Compatibility
 
@@ -439,14 +367,6 @@ Currently, the Fuchsia project runs rustdoc on all of their crates to generate a
 
 # Unresolved questions
 
-## Index crate?
-
-Require users to generate documentation bundles via an index crate (current) vs. creating a new mode to allow rustdoc to run without a target crate (proposed).
-
-If one would like to merge the documentation of several crates, we could continue to require users to provide an index crate, like [the fuchsia index](https://fuchsia-docs.firebaseapp.com/rust/rustdoc_index/). This serves as the target of the rustdoc invocation, and the landing page for the collected documentation. Supporting only this style of index would require the fewest changes. This is the mode described in the Guide-level explanation.
-
-The proposition, to allow users of rustdoc the flexibility of not having to produce an index, is to allow rustdoc to be run in a mode where no target crate is provided. It would generate rendered cross-crate information based only on what is provided through `--include-parts-dir`. The source crates are provided to rustdoc, through a mechanism like `--extern`, rustdoc merges and writes the CCI, and copies the item and module links to the doc root. This would require more extensive changes, as rustdoc assumes that it is invoked with a target crate. This mode is somewhat analogous to the [example scraping mode](https://github.com/rust-lang/rfcs/blob/master/text/3123-rustdoc-scrape-examples.md). Having to create an index crate, that actively uses all of the crates in the environment, might prohibit the use of this feature in settings where users do not intend to produce an index, or where exhaustively listing all dependencies (to `--extern` them) is difficult.
-
 ## Unconditionally generating the `doc.parts` files?
 
 Generate no extra files (current) vs. unconditionally creating `doc.parts` to enable more complex future CCI (should consider).
@@ -462,6 +382,8 @@ This proposal proposes to continue reading from the rendered cross-crate informa
 
 # Future possibilities
 
+This RFC is primarily intended be followed by the deprecation of the now-default `--merge=shared` mode. This will reduce complexity in the long term. Changes to Cargo, docs.rs and other tools that directly invoke rustdoc will be required. To verify that the `--merge=none` -> `--merge=finalize` workflow is sufficient for real use cases, the deprecation of `--merge=shared` will be delayed to a future RFC.
+
 This change could begin to facilitate trait implementations being
 statically compiled as part of the .html documentation, instead of being loaded
 as separate JavaScript files. Each trait implementation could be stored as an
@@ -469,6 +391,4 @@ HTML part, which are then merged into the regular documentation. Implementations
 
 Another possibility is for `doc.parts` to be distributed on `docs.rs` along with the regular documentation. This would facilitate a mode where documentation of the dependencies could be downloaded externally, instead of being rebuilt locally.
 
-A future possibility related to the index crate idea is to have an option for embedding user-specified HTML into the `--enable-index-page`'s HTML.
-
 The changes in this proposal are intended to work with no changes to Cargo and docs.rs. However, there may be benefits to using `--merge=finalize` with Cargo, as it would remove the need for locking the output directory. More of the documentation process could happen in parallel, which may speed up execution time.

From e66bb63588b18386b74e4042c1706d42f3bdce5c Mon Sep 17 00:00:00 2001
From: EtomicBomb 
Date: Tue, 20 Aug 2024 15:56:39 +0000
Subject: [PATCH 26/28] fix typo identified by jsha

typo in --merge=finalize description: can links
---
 text/0000-mergeable-rustdoc-cross-crate-info.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md
index 9d48971138c..422a5ce7224 100644
--- a/text/0000-mergeable-rustdoc-cross-crate-info.md
+++ b/text/0000-mergeable-rustdoc-cross-crate-info.md
@@ -35,7 +35,7 @@ More details are in the Reference-level explanation.
 * `--include-parts-dir=path/to/doc.parts/`: Include cross-crate information from this previously written `doc.parts` directories into a collection that will be written by the current invocation of rustdoc. May only be provided with `--merge=finalize`. May be provided any number of times.
 * `--merge=none`: Do not write cross-crate information to the `--out-dir`. The flag `--parts-out-dir` may instead be provided with the destination of the current crate's cross-crate information parts.
 * `--merge=shared` (default): Append information from the current crate to any info files found in the `--out-dir`.
-* `--merge=finalize`: Write cross-crate information from the current crate and any crates included via `--include-parts-dir` to the `--out-dir`, overwriting conflicting files. This flag may be used with or without an input crate root, as it can links existing docs.
+* `--merge=finalize`: Write cross-crate information from the current crate and any crates included via `--include-parts-dir` to the `--out-dir`, overwriting conflicting files. This flag may be used with or without an input crate root, in which case it only links crates included via `--include-parts-dir`.
 
 ## Example
 

From 665a58f7392572fe715a53bbc8bf048fca4d40a3 Mon Sep 17 00:00:00 2001
From: EtomicBomb 
Date: Wed, 21 Aug 2024 00:56:22 +0000
Subject: [PATCH 27/28] editorial fixes from @aDotInTheVoid's suggestions

* full sentences in intro
* Clarify Cargo's expected use of `--merge=none|shared|finalize`
* Explain why you would use multiple rustdoc invocations with the same --out-dir
  and merge=none
* Remove justification for why rustdoc output can be unstable in general
* Permalink specific fuchsia commit
---
 .../0000-mergeable-rustdoc-cross-crate-info.md | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md
index 422a5ce7224..f51a39aec78 100644
--- a/text/0000-mergeable-rustdoc-cross-crate-info.md
+++ b/text/0000-mergeable-rustdoc-cross-crate-info.md
@@ -5,7 +5,7 @@
 
 # Summary
 
-Mergeable cross-crate information in rustdoc. Facilitates the generation of documentation indexes in workspaces with many crates by allowing each crate to write to an independent output directory. Final documentation is rendered with a lightweight merge step. Configurable with command-line flags, this proposal writes a `doc.parts` directory to hold pre-merge cross-crate information. Currently, rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating rustdoc in build systems that enforce the independence of build actions.
+This RFC discusses mergeable cross-crate information in rustdoc. It facilitates the generation of documentation indexes in workspaces with many crates by allowing each crate to write to an independent output directory. The final documentation is rendered by combining these independent directories with a lightweight merge step. When provided with `--parts-out-dir`, this proposal writes a `doc.parts` directory to hold pre-merge cross-crate information. Currently, rustdoc requires global mutable access to a single output directory to generate cross-crate information, which is an obstacle to integrating rustdoc in build systems that enforce the independence of build actions.
 
 # Motivation
 
@@ -17,9 +17,7 @@ There are some files in the rustdoc output directory that are read and overwritt
 
 Build systems may run build actions in a distributed environment across separate logical filesystems. It might also be desirable to run rustdoc in a lock-free parallel mode, where every rustdoc process writes to a disjoint set of files.
 
-Cargo fully supports cross-crate information, at the cost of requiring global read-write access to the doc root (`target/doc`). There are significant scalability issues with this approach.
-
-Rustdoc needing global mutable access to the files that encode this cross-crate information has implications for caching, reproducible builds, and content hashing. By adding an option to avoid this mutation, rustdoc will serve as a first-class citizen in non-cargo build systems.
+Cross-crate information is supported in Cargo. It calls rustdoc with a single `--out-dir`, which requires global read-write access to the doc root (e.g. `target/doc`). There are significant scalability issues with this approach. Global mutable access to the files that encode this cross-crate information has implications for caching, reproducible builds, and content hashing. By adding an option to avoid this mutation, rustdoc will serve as a first-class citizen in non-cargo build systems.
 
 These considerations motivate adding an option for outputting partial CCI (parts), which are merged (linked) with a later step.
 
@@ -31,9 +29,9 @@ This RFC has the goal of enabling the future deprecation of the default (called
 
 More details are in the Reference-level explanation.
 
+* `--merge=none`: Do not write cross-crate information to the `--out-dir`. The flag `--parts-out-dir` may instead be provided with the destination of the current crate's cross-crate information parts.
 * `--parts-out-dir=path/to/doc.parts/`: Write cross-crate linking information to the given directory (only usable with the `--merge=none` mode). This information allows linking the current crate's documentation with other documentation at a later rustdoc invocation.
 * `--include-parts-dir=path/to/doc.parts/`: Include cross-crate information from this previously written `doc.parts` directories into a collection that will be written by the current invocation of rustdoc. May only be provided with `--merge=finalize`. May be provided any number of times.
-* `--merge=none`: Do not write cross-crate information to the `--out-dir`. The flag `--parts-out-dir` may instead be provided with the destination of the current crate's cross-crate information parts.
 * `--merge=shared` (default): Append information from the current crate to any info files found in the `--out-dir`.
 * `--merge=finalize`: Write cross-crate information from the current crate and any crates included via `--include-parts-dir` to the `--out-dir`, overwriting conflicting files. This flag may be used with or without an input crate root, in which case it only links crates included via `--include-parts-dir`.
 
@@ -224,7 +222,9 @@ This mode only renders the HTML item documentation for the current crate. It doe
 
 In this mode, a user will provide `--parts-out-dir=` and `--merge=none` to each crate's rustdoc invocation. The user should provide `--extern-html-root-url`, and specify a absolute final destination for the docs, as a URL. The `--extern-html-root-url` flag should be provided for each crate's rustdoc invocation, for every dependency.
 
-The same `--out-dir` may be used for multiple parallel rustdoc invocations, as rustdoc will continue to acquire an flock on the `--out-dir` to address conflicts. A user may select a different `--out-dir` for each crate's rustdoc invocation. 
+A user may select a different `--out-dir` for each crate's rustdoc invocation.
+
+The same `--out-dir` may also be used for multiple parallel rustdoc invocations, as rustdoc will continue to acquire an flock on the `--out-dir` to address conflicts. This is in anticipation of the possibility of deprecating `--merge=shared`, and Cargo adopting a `--merge=none` + `--merge=finalize` workflow. Cargo is expected continue using the same `--out-dir` for all crates in a workspace, as this eliminates the operations needed to merge multiple `--out-dirs`.
 
 ### Link documentation: `--merge=finalize`
 
@@ -300,13 +300,13 @@ This proposal is capable of addressing two primary use cases. It allows develope
 
 CCI is not automatically enabled in either situation. A combination of the `--include-parts-dir`, `--merge`, and `--parts-out-dir` flags are needed to produce this behavior. This RFC provides a minimal set of tools that allow developers of build systems, like Bazel and Buck2, to create rules for these scenarios. 
 
-With separate `--out-dir`s, copying item docs to an output destination is needed. Rustdoc will never support the entire breadth of workflows needed to merge arbitrary directories, and will rely on users to run external commands like `mv`, `cp`, `rsync`, `scp`, etc. for these purposes. Most users are expected to use a single `--out-dir` for all crates, in which case these external tools are not needed.
+With separate `--out-dir`s, copying item docs to an output destination is needed. Rustdoc will never support the entire breadth of workflows needed to merge arbitrary directories, and will rely on users to run external commands like `mv`, `cp`, `rsync`, `scp`, etc. for these purposes. Most users are expected to continue to use a single `--out-dir` for all crates, in which case these external tools are not needed. It is expected that build systems with the need to be hermetic will use separate `--out-dir`s for `--merge=none`, while Cargo will continue to use the same `--out-dir` for every rustdoc invocation.
 
 ## Compatibility
 
 This RFC does not alter previous compatibility guarantees made about the output of rustdoc. In particular it does not stabilize the presence of the rendered cross-crate information files, their content, or the HTML generated by rustdoc.
 
-In the same way that the [rustdoc HTML output is unstable](https://rust-lang.github.io/rfcs/2963-rustdoc-json.html#:~:text=The%20HTML%20output%20of%20rustdoc,into%20a%20different%20format%20impractical), the content of `doc.parts` will be considered unstable. Between versions of rustdoc, breaking changes to the content of `doc.parts` should be expected. Only the presence of a `doc.parts` directory is promised, under `--parts-out-dir`. Merging cross-crate information generated by disparate versions of rustdoc is not supported. To detect whether `doc.parts` is compatible, rustdoc includes a version number in these files (see New directory: `doc.parts`).
+The content of `doc.parts` will be considered unstable. Between versions of rustdoc, breaking changes to the content of `doc.parts` should be expected. Only the presence of a `doc.parts` directory is promised, under `--parts-out-dir`. Merging cross-crate information generated by disparate versions of rustdoc is not supported. To detect whether `doc.parts` is compatible, rustdoc includes a version number in these files (see New directory: `doc.parts`).
 
 The implementation of the RFC itself is designed to produce only minimal changes to cross-crate info files and the HTML output of rustdoc. Exhaustively, the implementation is allowed to 
 * Change the sorting order of trait implementations, type implementations, and other cross-crate info in the HTML output of rustdoc.
@@ -363,7 +363,7 @@ buck2 does not natively merge rustdoc from separate targets. The buck2 maintaine
 
 ## Ninja [(GN)](https://fuchsia.dev/fuchsia-src/development/build/build_system/intro) + Fuchsia
 
-Currently, the Fuchsia project runs rustdoc on all of their crates to generate a [documentation index](https://fuchsia-docs.firebaseapp.com/rust/rustdoc_index/). This index is effectively generated as an [atomic step](https://cs.opensource.google/fuchsia/fuchsia/+/main:tools/devshell/contrib/lib/rust/rustdoc.py) in the build system. It takes [3 hours](https://ci.chromium.org/ui/p/fuchsia/builders/global.ci/firebase-docs/b8744777376580022225/overview) to document the ~2700 crates in the environment. With this proposal, building each crate's documentation could be done as separate build actions, which would have a number of benefits. These include parallelism, caching (avoid rebuilding docs unnecessarily), and robustness (automatically reject pull requests that break documentation).
+Currently, the Fuchsia project runs rustdoc on all of their crates to generate a [documentation index](https://fuchsia-docs.firebaseapp.com/rust/rustdoc_index/). This index is effectively generated as an [atomic step](https://cs.opensource.google/fuchsia/fuchsia/+/4eefc272d36835959f2e44be6e06a6fbb504e418:tools/devshell/contrib/lib/rust/rustdoc.py) in the build system. It takes [3 hours](https://ci.chromium.org/ui/p/fuchsia/builders/global.ci/firebase-docs/b8744777376580022225/overview) to document the ~2700 crates in the environment. With this proposal, building each crate's documentation could be done as separate build actions, which would have a number of benefits. These include parallelism, caching (avoid rebuilding docs unnecessarily), and robustness (automatically reject pull requests that break documentation).
 
 # Unresolved questions
 

From e1b5e5b05c5ab6285dbabcd460966492a331de4a Mon Sep 17 00:00:00 2001
From: Ethan Williams 
Date: Sun, 25 Aug 2024 16:27:03 -0700
Subject: [PATCH 28/28] replace WIP -> implementation, per camelid

Change sentence reading "The WIP may change the sorting order..." to "The implementation may change the sorting order..."

Co-authored-by: Noah Lev 
---
 text/0000-mergeable-rustdoc-cross-crate-info.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/text/0000-mergeable-rustdoc-cross-crate-info.md b/text/0000-mergeable-rustdoc-cross-crate-info.md
index f51a39aec78..64e2806360a 100644
--- a/text/0000-mergeable-rustdoc-cross-crate-info.md
+++ b/text/0000-mergeable-rustdoc-cross-crate-info.md
@@ -325,7 +325,7 @@ The limitation of `--extern-html-root-url` is that it needs to be provided with
 
 # Drawbacks
 
-The WIP may change the sorting order of the elements in the CCI. It does not change the content of the documentation, and is intended to work without modifying Cargo and docs.rs.
+The implementation may change the sorting order of the elements in the CCI. It does not change the content of the documentation, and is intended to work without modifying Cargo and docs.rs.
 
 # Rationale and alternatives