Understanding how the Ansible docs build

Understanding the Ansible Docs Build
Jenkins builds - where the magic happens
- Staging a PR with the Jenkins builds
Much Ado about Redirects
Preparing for a release
- Updating intersphinx links

Table of contents generated with markdown-toc

Understanding the Ansible Docs Build

There are many components to creating the Ansible documentation. This article attempts to start documenting some of these parts and pieces. It's a work in progress for sure, so don't take this all as proven fact just yet :-)

Sphinx - The workhorse of it all

Ansible uses Sphinx to convert our rst files to html. Fairly straightforward, but the main file it uses to configure it all is conf.py. We won't repeat the Sphinx conf.py documentation, but we do make use of a few items that we describe later in order to create two docsites from the ansible/ansible Github repository.

make webdocs - What does it do?

Whether you are building locally, or triggering the internal to Red Hat build script that publishes to docs.ansible.com, the most common command you use is make webdocs.

Prior to Ansible 2.10, make webdocs was the main command to make all the documentation. In 2.10 and later, there are two commands:

make webdocs ANSIBLE_VERSION=[2.10,3,4] - this chooses between building all the docs (for 2.10) or building the Ansible package docs for 3 or later. The main difference is 3 or later does not include ansible-core roadmap or porting guides. If you don't select a version, it chooses the latest available from ansible-build-data.
make coredocs - Excludes network and scenario guides, and any Ansible package roadmap/porting guides.

make webdocs - How does it do it?

This level of detail is mostly for the curious or anyone who has to change what and how these commands work. Feel free to ignore.

Not ignoring? Okay, let's take a walk through Makefiles and some python code.

Makefiles

In docs/docsite/ you'll see two makefiles:

Makefile - sets up what make webdocs and make coredocs do at the highest level.
Makefile.sphinx - sets up what Sphinx (the tool that converts our rst files to html) does.

There is also a section in the main Ansible Makefile make webdocs etc that allows you to run the make commands from the root directory instead of from docs/docsite. If you add some new variant on making the docs, consider if you should also add that variant to the main Ansible Makefile as well. Otherwise your new variant only works from `docs/docsite/

Docs Makefile

There's quite a lot in this Makefile, but let's start with the basics. Starting in Ansible 2.10, we need to create two separate docsites effectively from the same set of .rst files. To do this, we use a few Sphinx and Makefile tricks:

Use the sphinx -c <path> option for a separate conf.py file to control the docsites and some other parameters.
Use exclude_pattern[] to exclude network, galaxy, and scenario guides and the individual porting guides and roadmap files from the appropriate sites.
Use symlinks in the Makefile to link index.rst from either ansible_index file for ansible or core_index.rst for core.

docs_build.py and the fun it has!

running notes from a session today... will need to make these more coherent later

docs-build from the Makefile gets saved into a variable PLUGIN_FORMATTER=../../hacking/build-ansible.py docs-build

The build-ansible.py script is a bit non-standard. The structure is that it's a lite wrapper script which sets up the location that the script can find its python files : https://github.com/ansible/ansible/blob/devel/hacking/build-ansible.py#L40-L41 : hacking/build-ansible.py:40-41 sys.path.insert(0, ansible_lib_path()) sys.path.insert(0, build_lib_path())

And then it loads the parts of the script that do actual work as plugins. So instead having a literal import docs_build somewhere, the loading of the code is done programmatically in a loop: https://github.com/ansible/ansible/blob/devel/hacking/build-ansible.py#L64
hacking/build-ansible.py:64 subcommands = load('build_ansible.command_plugins', subclasses=commands.Command)

Then the plugin we want (docs_build) is selected in this for loop: https://github.com/ansible/ansible/blob/devel/hacking/build-ansible.py#L82 hacking/build-ansible.py:82 for subcommand in subcommands:

And we execute the plugin's main() function here: https://github.com/ansible/ansible/blob/devel/hacking/build-ansible.py#L92 (edited) hacking/build-ansible.py:92 retval = command.main(args)

Explaining load('build_ansible.command_plugins', subclasses=commands.Command)

Okay, so load() comes from the plugin library we're using (straight.plugin). It takes a python package as its first argument and looks inside of that location for any python files . The second argument, subclasses=commands.Command, tells load () that valid plugins are commands.Command classes within the files its examining. So in our case, we added the hacking/build_library directory to sys.path (the directories that python checks for python libraries) on Line 41. So load finds the python package we specified in https://github.com/ansible/ansible/tree/devel/hacking/build_library/build_ansible/command_plugins

Each of those files should have a class inside of them that inherits from build_ansible.commands.Command (example: https://github.com/ansible/ansible/blob/devel/hacking/build_library/build_ansible/command_plugins/docs_build.py#L123 ) (edited) hacking/build_library/build_ansible/command_plugins/docs_build.py:123 class CollectionPluginDocs(Command):

load() finds that class in each of the files, and returns a list of all of them.

So make coredocs builds ansible-core docs. make webdocs ANSIBLE_VERSION=[2.10,3,4] builds ansible package docs. The latter even works in devel because the package builder is selecting what the version of the collection docs are.

We will Autodetect the latest collection *.deps file and use that.

(future ) Implement building the docs for the latest version of collections on galaxy?

But where's the collection info from?

Good question! So the plugin documentation is pulled in from collections using the antsibull tool. It checks for the collection build data for the version of Ansible we're building and pulls in the plugin docs for the version of each collection in that data file.

Hacking more collections into the build or - how to stress test build memory

Since we've had some problems with the Jenkins builds failing due to memory problems (see Jenkins general info in the next section) - we now have a way to dummy up an Ansible package build with additional collections to stress test things before a release.

To do this:

Checkout ansible-build-data
- git clone https://github.com/ansible-community/ansible-build-data
For testing the devel tree, modify the latest ansible.in file (currently ansible-build-data/5/ansible.in)
- Example, this adds the community.dns collection to the collection docs that will be built:
- echo 'community.dns ' >> ansible-build-data/5/ansible.in
For testing the stable tree, you will have to modify the latest ansible-X.Y.Z.deps file:
- echo 'community.dns: 1.0.1' >> ansible-build-data/4/ansible-4.2.0.deps
Checkout the ansible-core code:
- git clone https://github.com/ansible/ansible
cd ansible
Tell the docs build to use your local copy of ansible-build-data when building the ansible package devel docs:
- Note: the ansible-build-data path can be absolute. If relative as in the example, it is relative to the ansible/docs/docsite directory
- make webdocs EXTRA_PLUGIN_FORMATTER_ARGS='--ansible-build-data=../../../ansible-build-data'

Turning off breadcrumbs

The collection-level breadcrumbs can be a memory hog when building docs. The Jenkins builds (below) have a toggle to turn them off as needed. To turn of breadcrumb generation on your local docs build:

echo "breadcrumbs = false" > ~/.antsibull.cfg
Build the docs (e.g. make webdocs).

Jenkins builds - where the magic happens

The Jenkins builds internal to Red Hat are what finally publishes all these documents. There are:

Nightly builds -for ansible-core and Ansible, based on devel.
Manual builds - mostly for tests, but also to republish docs for a given minor release of ansible-core or Ansible.

When a new release happens, We need to update the Ansible jenkins build to recognize the new release number, and update redirects (see below).

Staging a PR with the Jenkins builds

While this is a task that only a Red Hat person can do, here's some notes for said Red Hat person to remember ;-)

Fill in build parameters for the github fork (ansible or the PR owner), and branch (devel, stable-2.11, PR branch name, etc).

Much Ado about Redirects

Starting with Ansible 2.10, we have server side redirects in place to allow the version switcher to switch between a module in 2.9 (say https://docs.ansible.com/ansible/2.9/modules/docker_image_module.html) to the same module now in 2.10 (https://docs.ansible.com/ansible/2.10/collections/community/general/docker_image_module.html). A significant (aka thousands) batch of redirects were generated for 2.9 and 2.10 to allow for the version switcher (and user bookmarks) to go to the correct new location for modules migrated to collections.

For future releases, we need to create a release based folder at https://github.com/ansible/docsite/tree/master/ansible (private repo) and copy the 2.10 redirects into that new folder (for example 3/).

Preparing for a release

Since we now have two releases - Ansible and ansible-core, we also have two checklists for how to prepare and publish a new docset for each. The most recent examples are:

These checklists change over time so search for a similar issue for releases that come after these two.

Updating intersphinx links

We use the intersphinx links extension to provide easy cross-references. After each release, we need to update those links as follows from within a local copy of the devel branch:

add intersphinx_mapping for the new release to docs/docsite/sphinx_conf/all_conf.py (e.g. 'ansible_3': ('https://docs.ansible.com/ansible/3/', (None, '../ansible_3.inv')),)
source hacking/env_setup
hacking/build-ansible.py update-intersphinx-cache -o docs/docsite -c docs/docsite/sphinx_conf/all_conf.py
merge and backport to the appropriate stable branch associated with the release.

This Wiki is used for quick notes, not for support or documentation.