How can we extend CI testing on docs to include rst files that are part of the Ansible package but not core?

[samccann]

Summary

We have two docsites now (docs.ansible.com/ansible for Ansible, and docs.ansible.com/ansible-core for ansible-core). Since this split, we have a number of RST files that are excluded from the ansible-core docs. This means the current CI tests in ansible/ansible repository where the docs get built from are not actually running against a growing number of files. (30+ rst files and all the Ansible package porting guides and roadmaps).

See https://github.com/ansible/ansible/blob/devel/docs/docsite/sphinx_conf/core_conf.py#L93 for the list of excluded files.

The specific CI test is ansible-test sanity --test docs-build and you can see here where it builds only core docs:
https://github.com/ansible/ansible/blob/devel/test/sanity/code-smell/docs-build.py#L31

If we want to bring CI testing back to these Ansible package-only files, we'll need to update the Makefile to have a special make for all the Ansible guides but not the collection /module files, and then change the CI test to use this instead of the current core_singlehtmldocs.

My guess is we would then have to maintain that new Makefile option to ensure it stays up todate with any new files added that are only for Ansible package docs.

Maybe there's another way to do this?

[samccann]

To clarify - these RST files live in ansible/ansible repository, but because they were deemed not applicable to someone using ansible-core we don't publish them to core.

[samccann]

Summarizing some of the discussion that happened in docs matrix channel:
Difficulties:

• some of these rst files reference collections which are not in ansible/ansible so we would introduce a different set of CI failures. For example, An RST file that is excluded from core might have a :ref: to a collection or module that is also not in core.

Options discussed:

• add a temporary .rst file for only the check build that contains a list of link targets that are needed for the build to succeed. Drawback - it would make the CI testing longer. TBD how long and if it would be too long for PR CI tests.

• have some other 'nightly build' test to detect problems. Drawback - would not detect problem prior to PR merge and thus require fixing in a new PR etc.

• Can get some mitigation of the problem with using the docs-build actions to prevent bad references from even landing in their default branch: https://github.com/ansible-community/github-docs-build. Drawback- It doesn't prevent cross-repo breaks, such as a collection renames a label but files in ansible/ansible still use the old label.

Other difficulties - Different ansible/ansible branches can reference the same collection version while other branches in ansible/ansible reference a different collection version (where the label for xref might have changed).

[briantist]

@felixfontein has a PR for the docs-build actions that will make it feasible for more collections to use them: ansible-community/github-docs-build#43

[samccann]

Closing as we shouldn't have additional CI in ansible/ansible for collection stuff.