diff --git a/ci/README.md b/ci/README.md new file mode 100644 index 00000000000..28809a58a18 --- /dev/null +++ b/ci/README.md @@ -0,0 +1,66 @@ +# ci + +## API docs + +The site's `/api` page links to API documentation for many RAPIDS projects. + +Those API docs are built in CI of the individual projects, published to object storage, +then pulled into builds here and integrated into the rest of the docs site. + +The steps are roughly as follows. + +### Step 1: determine libraries and versions to build + +`get-projects-to-versions.sh` is responsible for holding all the logic relevant to the questions "What projects' API docs should be hosted? What versions?". + +Run it from the root of the repo to see for yourself. + +```shell +./ci/get-projects-to-versions.sh +``` + +That script is reused by other automation to determine which projects and versions to build. + +### Step 2: download API docs + +All of the API docs for RAPIDS projects are uploaded to an S3 bucket from those projects' CI. + +To integrate them into the docs site, they need to be downloaded and placed in `_site/api/` locally. + +```shell +./ci/download_from_s3.sh +``` + +### Step 3: post-processing + +After downloading the API docs, they'll all be arranged locally in directories named with version numbers, like this: + +```text +_site/api/ +├── cudf +│ ├── 25.06 +│ ├── 25.08 +│ ├── 25.10 +├── cudf-java +│ ├── 25.06 +│ ├── 25.08 +├── cuml +│ ├── 25.06 +│ ├── 25.08 +│ ├── 25.10 +``` + +Further processing is then needed to integrate them into the docs site, including: + +* mapping version numbers like `25.10` to version identifies like `"stable"` (to populate links like https://docs.rapids.ai/api/cudf/stable/) +* adding version selectors on all the docs pages +* other formatting and visual consistency changes + +That is handled by the code in the `ci/customization` directory. +You can invoke it all like this: + +```shell +./ci/post-process.sh +``` + +See the docs in `ci/customization` for more details ([customization/README.md](./customization/README.md)). diff --git a/ci/customization/README.md b/ci/customization/README.md index 9e4b44d4b8b..5c858dece07 100644 --- a/ci/customization/README.md +++ b/ci/customization/README.md @@ -1,10 +1,15 @@ # Documentation Customization -The documentation below describes how the scripts work to customize the generated HTML files. The scripts should be run in the order they appear in the table of contents. +The documentation below describes how the scripts work to customize the generated HTML files. +The scripts should be run in the order they appear in the table of contents. + +Unless otherwise stated, all scripts should be run from the root of this repository. ## Table of Contents - [Dependencies](#dependencies) +- [Quickstart](#quckstart) +- [Project Versions](#project-versions) - [Update Symlinks](#symlinks) - [Generate Library Map](#generate-library-map) - [Customize HTML File](#customize-html) @@ -12,35 +17,94 @@ The documentation below describes how the scripts work to customize the generate ### Dependencies -- [jq](https://github.com/stedolan/jq) -- [BeautifulSoup4](https://github.com/waylan/beautifulsoup) +- `aws` CLI ([docs](https://aws.amazon.com/cli/)) +- `jq` ([docs](https://jqlang.org/manual/)) +- Python / `pip` +- `yq` ([docs](https://mikefarah.gitbook.io/yq)) + +### Quickstart + +If you just want to try running all of the customization locally, download all of the API docs ([as described in `ci/README.md`](../README.md)), then invoke the following script: + +```shell +./ci/post-process.sh +``` + +To learn more about each step, keep reading. + +### Project Versions + +Building the API docs requires answering these questions: + +* which RAPIDS projects? +* what version types? (stable? legacy? nightly?) +* what version numbers correspond to those version types? + +Logic for all of that is centralized in a script. + +Invoke it to see what will be built. + +```shell +./ci/get-projects-to-versions.sh > ./ci/customization/projects-to-versions.json +``` ### Symlinks -For the doc customization to work correctly, all of the symlinks need to be up to date. The symlinks ultimately enable us to generate paths like `/cudf/stable/` that point to the latest RAPIDS release version folder (i.e. `/cudf/0.13/`). +Initially, the downloaded API docs are in directories named with version numbers, like this: -[update_symlinks.sh](/update_symlinks.sh) is a shell script that updates each project's symlinks according to the versions in [releases.json](/_data/releases.json). It looks for the legacy, stable, and nightly version folders (i.e. `0.11`, `0.12`, etc.) and creates the corresponding symlinks if those folders exist. +```console +$ tree -L 2 ./_site/api +_site/api/ +├── cudf +│ ├── 25.06 +│ ├── 25.08 +│ ├── 25.10 +``` -**Usage:** +Those directory paths become links on the docs site, like https://docs.rapids.ai/api/cudf/25.10/. -> **Note:** This script is intended to be run from the project's root. +To be more bookmark-friendly, the docs site also has links with version _types_ like `/legacy`, `/nightly`, and `/stable`. -```sh -update_symlinks.sh +To avoid fully copying files, that's done via creating symlinks, similar to this: + +```shell +pushd ./_site/api/cudf +ln -s 25.10 stable ``` -### Generate Library Map +To avoid hard-coding all those versions and paths, that logic is automated in a script. +Try running it to generate the symlinks. + +```shell +./ci/update_symlinks.sh ./ci/customization/projects-to-versions.json +``` + +After that is run, you'll see links like the following have been created. + +```console +$ tree -L 2 ./_site/api +./_site/api/ +├── cudf +│ ├── 25.06 +│ ├── 25.08 +│ ├── 25.10 +│ ├── latest -> 25.08 +│ ├── legacy -> 25.06 +│ ├── nightly -> 25.10 +│ └── stable -> 25.08 +``` -One of the ways we customize the documentation files is by adding dropdown selectors to each doc page that allows visitors to select and navigate between RAPIDS libraries and their versions. In order to know what page a user will be directed to upon selection, a map of available libraries and versions needs to be generated. [lib_map.sh](lib_map.sh) generates this map. +### Generate Library Map -`lib_map.sh` creates a JSON file that is used by `customize_docs.py` to populate the available options for the _libray_ and _version_ selectors. +One of the ways we customize the documentation files is by adding dropdown selectors to each doc page that allows visitors to select and navigate between RAPIDS libraries and their versions. -**Usage:** +In order to know what page a user will be directed to upon selection, a map of available libraries and versions needs to be generated. +[lib_map.sh](lib_map.sh) generates this map. -> **Note:** This script is intended to be run from the project's root. +`lib_map.sh` creates a JSON file that is used by `customize_docs.py` to populate the available options for the _library_ and _version_ selectors. ```sh -customization/lib_map.sh +./ci/customization/lib_map.sh ``` An excerpt of the generated JSON file is shown below: @@ -48,26 +112,28 @@ An excerpt of the generated JSON file is shown below: ```json { "cuml": { - "nightly": "/cuml/en/nightly/api.html", - "stable": "/cuml/en/stable/api.html", - "legacy": "/cuml/en/legacy/api.html" + "nightly": "/cuml/en/nightly", + "stable": "/cuml/en/stable", + "legacy": "/cuml/en/legacy" + }, + "libcudf": { + "nightly": "/libcudf/nightly/namespacecudf/", + "stable": "/libcudf/stable/namespacecudf/", + "legacy": "/libcudf/legacy/namespacecudf/" }, - "cuxfilter": { - "nightly": "/cuxfilter/en/nightly", + "rapidsmpf": { + "nightly": "/api/rapidsmpf/nightly", "stable": null, "legacy": null }, - "libcudf": { - "nightly": "/libcudf/nightly/namespacecudf.html", - "stable": "/libcudf/stable/namespacecudf.html", - "legacy": "/libcudf/legacy/namespacecudf.html" - } } ``` ### Customize HTML -[customize_doc.py](customize_doc.py) is a Python script that uses [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) to theme Doxygen and Sphinx HTML documentation files. It adds the following elements to each documentation page: +[customize_doc.py](customize_doc.py) is a Python script that uses [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) to theme Doxygen and Sphinx HTML documentation files. + +It adds the following elements to each documentation page: - A _Home_ button that links to https://docs.rapids.ai/api - A _library_ selector that enables navigation to different RAPIDS libraries @@ -78,39 +144,18 @@ An excerpt of the generated JSON file is shown below: `customize_doc.py` can be safely run multiple times on a single file without adding duplicate elements to the page. -**Usage:** - -> Before running the script, it is important that [update_symlinks.sh](/update_symlinks.sh) and [lib_map.sh](lib_map.sh) have been run. +That's invoked by a wrapper script that determines HTML files to pass in. ```sh -python customization/customize_docs.py ${ABS_PATH_TO_HTML_FILE} ${CURRENT_RAPIDS_VERSION} -``` - -#### Helper Script - -[customize_docs_in_folder.sh](customize_docs_in_folder.sh) is a helper script that makes it easy to recursively customize all of the files in a specified folder. - -**Usage:** - -> **Note:** This script is intended to be run from the project's root. - -```sh -customization/customize_docs_in_folder.sh api/ - -# or - -customization/customize_docs_in_folder.sh api/rmm -``` - -### TL;DR - -To customize the docs, run: - -```sh -update_symlinks.sh # ensures symlink accuracy - -customization/lib_map.sh # generates a JSON file needed by customize_docs.py +PROJECTS_TO_JSON_PATH="./ci/customization/projects-to-versions.json" -customization/customize_docs_in_folder.sh api/rmm +# all API docs +./ci/customization/customize_docs_in_folder.sh \ + ./_site/api \ + "${PROJECTS_TO_JSON_PATH}" +# just cudf +./ci/customization/customize_docs_in_folder.sh \ + ./site/api/cudf \ + ./ci/customization/projects-to-versions.json ```