Linting Clang documentation

[cor3ntin]

It would be nice to be able to catch and report during Pre Commit checks syntax errors in the RsT as these are frequents and easy to miss.

Some Clang RsT files are created by TableGen, so that leaves us with 2 options:

Only checking the non-generated documentation files.

This is fairly easy, we can run an existing action to run sphinx-build in clang/docs, and there are pre-existing actions that could
be adapted to make this work, at least once #65664 is fixed.

Unfortunately, for that we would have to ignore all references / missing files errors as some of the generated files would not exist.

Build the docs as part of the existing CI checks

The idea would be to run both check-clang and docs-clang-html in the same job.
This would be enough to fail CI on invalid docs, which is not bad.
To report errors, we can:

• Force sphinx to log the errors in a file in addition of the output using the env variable SPHINXOPTS
• Add an action after the build that parse that logs and turn it into annotations. I started to looking into the viability of that
  https://github.com/cor3ntin/llvm-sphinx-log-collector/blob/main/src/index.ts - it seems doable
  Linting would appear as a check in github's PR interface
• This would require installing sphinx and requirements on the build agent.

The issue with that approach is that it's more work, and each project would have to adapt slightly.
But talking to @AaronBallman we are eventually going to make sure we don't ignore generated files, so we might as well start now.
LLDB also has different requirements.

Not sure when I'll have time to do more thinkering there (if you are interesting on taking on that work let me know!), but i wanted to write up what I found.

Reporting formatting, compilation and clang-tidy errors in the same way would be neat too.

[llvmbot]

@llvm/issue-subscribers-infrastructure

[EugeneZelenko]

@mydeveloperday tried to implement similar functionality: https://reviews.llvm.org/D55523

[boomanaiden154]

It might also be feasible to do this solely in Github actions? There are only a couple hundred build steps to build the clang docs and one to build the LLVM docs (not sure about other subprojects). That would keep build times at a decent level. If we pulled in an already built tablegen, we might also be able to reduce that number further (not totally sure about this though).

While it's not bad if the docs failing to build causes the whole pipeline to fail, separating out the docs build from the actual build would be ideal, I think.

[boomanaiden154]

I think we can do this with even less build steps. If we use LLVM_NATIVE_TOOLS_DIR and pass in an installation directory with clang-tblgen and llvm-tblgen we can omit most of the build steps and there should just be a couple left. Need to do some actual testing with this though.

I think we can probably implement this as follows:

1. Set up a job somewhere (probably nightly), maybe in llvm/actions, that builds a container image with the necessary tools (llvm-tblgen, clang-tblgen, myst/other dependencies). This mainly prevents us from having to rebuild tblgen/reinstall python dependencies each invocation.
2. Implement actions that build subproject docs when changed using said container image.
3. Maybe add annotations back into the diff (probably somewhere in a later step).

This means the docs will build fast, everything should be native to Github, and people will easily get feedback if their docs changes will build. There would be some latency with the container setup if a tblgen change is necessary for docs to generate, but I don't expect that would happen very often.

Doing everything in Github would also make the buildbots that build the docs a little bit more redundant, but given that they don't seem to get a lot of maintenance, that seems like a good thing.

I'd be interested in taking this on over the next week or two, just want to make sure this approach sounds reasonable to others/no one else is currently already working on implementing this?

[AaronBallman]

This means the docs will build fast, everything should be native to Github, and people will easily get feedback if their docs changes will build. There would be some latency with the container setup if a tblgen change is necessary for docs to generate, but I don't expect that would happen very often.

I would say it's uncommon but not rare. Can we have some sort of mechanism to trigger a rebuild of that container as-needed?

Doing everything in Github would also make the buildbots that build the docs a little bit more redundant, but given that they don't seem to get a lot of maintenance, that seems like a good thing.

Doing it in GitHub means people get informed that there's a problem before it gets committed to the repo, so I think it's a significant win over the current post-commit bots.

I'd be interested in taking this on over the next week or two, just want to make sure this approach sounds reasonable to others/no one else is currently already working on implementing this?

I think the approach sounds reasonable, but I also don't have much insight into how much compute power we have access to for the precommit CI work in GitHub. Maybe @tstellar @tru or @asl has more helpful feedback here.

[boomanaiden154]

I would say it's uncommon but not rare. Can we have some sort of mechanism to trigger a rebuild of that container as-needed?

Definitely! That would be very easy to setup. Not sure what types of permissions are needed to trigger workflows, but it's definitely doable.

I think the approach sounds reasonable, but I also don't have much insight into how much compute power we have access to for the precommit CI work in GitHub. Maybe @tstellar @tru or @asl has more helpful feedback here.

I think this could easily work with the Github free runners. They're only two threads, but building the docs after the initial tblgen build seems to be mostly serial from what I can tell and they build relatively quickly. I think there might be limits to how many runners we can have running concurrently (but I don't think PRs count as I believe the job counts against the fork t that point), but given the short job duration and limiting the docs build to changes that actually touch the documentation, I think that shouldn't be an issue.

[tstellar]

I would recommend not trying to use pre-built tools or at least implement that later if the doc builds ends up being too slow. There are other ways we could speed up the build, like with ccache.

[boomanaiden154]

Good point. CCache does work particularly well within Github actions. It presents the problem of cache invalidation, but we should be able to figure something out there.

I'll try and get a prototype of this up tonight just building everything within Github actions and then we can iterate from there.

[tstellar]

@boomanaiden154 If you haven't seen this already, we've been using the hendrikmuhs/ccache-action@v1 action in the release branch. It seamlessly integrates with the github actions cache.

[boomanaiden154]

I did not release that was already being used on the release branch. I've seen the action used before in other places though. I'll get an initial implementation running and then make sure to experiment with using that for ccache.

We'd probably have to do something with the cache key though because we can only have 10GB of cache data. I don't expect these caches to be large, but if we have a lot of docs builds and not many other builds also using the caches, we'll end up evicting a lot of those less frequently used caches with the default key strategy of that action where it just tags it by time.