API reference using doc-builder

[LysandreJik]

This issue documents the will to have a separate API reference using the doc-builder tool that was created initially for transformers, later adapted for datasets, optimum, and accelerate.

The resulting documentations can be found here:

• transformers
• datasets

The optimum and accelerate docs are currently being addressed, see huggingface/accelerate#271 and huggingface/optimum#86.

Porting the huggingface_hub documentation to the huggingface.co website using this utility completes the work of having all documentation using the same backend/frontend, in an effort to have coherent documentation across all of our tools.

The choice to not re-use Sphinx/reStructuredText can be (and has been) heavily debated, it eventually boils down to:

• having all of our documentation have the best integration we can with the rest of the website, making them as dynamic, enticing, and enjoyable as possible (integration with widgets, gradio demos, specific logged-in user features, etc), which is arguably something more difficult using a tool we do not have full control over
• having MarkDown as the only way to write documentation as we have had friction both internally and externally with the RST format, and we would like for documentation writing to be very accessible for newcomers and veterans alike.

---

Here's a breakdown of what needs to be done:

• Create a docs/hub/index.mdx as a landing page for the hub documentation (should be similar to the resulting README file)
• Create a docs/hub/_toctree.yml containing the sections and their title.
• Update the inline docstrings to follow the format defined here
• Create .mdx files under docs/hub referencing the objects to be put in the documentation.
• Clean-up the README.md file so that it is not a shallow API reference but a good introduction for the library, how to install it, contribute to it, etc.
• Either setup the Github Actions workflow for automatic upload as well as the ability to check the compiled documentation on each PR, or wait until the workflows are upstreamed in doc-builder for re-use.

And, finally:

• Add learnings to the doc-builder documentation so that it is simpler to write documentation (Documentation about usage and expected documentation doc-builder#132)

[adrinjalali]

There are a few things I'm missing here:

• is doc-builder open sourced already? It doesn't seem to have much documentation. Contributors should be able to build the docs locally w/o having access to HF infrastructure.
• Where do we have the mdx specs that people can read and learn how to write docs?
• I think it would make sense to wait until we can have GH actions to build and deploy the docs. Otherwise people will be writing docs kinda in the dark.

[LysandreJik]

• doc-builder is open sourced already: doc-builder. The documentation will be completed as part of this refactor, and is part of Documentation about usage and expected documentation doc-builder#132
• The .mdx format is defined here, and as mentioned in Documentation about usage and expected documentation doc-builder#132, the goal is for it to be visible right from doc-builder's documentation rather than from transformers'.
• The GH actions already exist for all repos mentioned above. The question is whether we implement it in the repository here or if we wait until these actions are upstreamed in doc-builder, but at no point will contributors writing documentation will lack visibility on the docs that they're writing. Furthermore, since guides are written in the .mdx format, they will be rendered directly in GitHub (example), so the question is only for inline docstrings.

[adrinjalali]

Cool, that sounds nice.

Another question i have is that by reading the "how to write documentation" you provide, it's clear how to write docstrings, but it's not clear how to write user guides and examples:

• I'm thinking something along the lines of having something resembling sphinx-gallery examples, and then referencing those examples from user guides.
• How does one reference other objects, classes, and methods from the current library and external libraries that are rendered with a hyperlink to the corresponding API doc?
• how to we reference other sections in the user guide from the API doc and the user guide itself?
• what is the workflow to add directives such as versionadded, versionchanged, etc?
• I see this in the docs:

It helps to keep the old links working when renaming section header and/or moving sections from one document to another.

In rst you reference a tag, so renaming headers doesn't actually do anything to break existing links. How does mdx handle this?

I'm not familiar with mdx, so I'm not sure how to approach these issues.

[osanseviero]

These are very good questions @adrinjalali 👍

Maybe it makes sense to move this discussion to an issue in https://github.com/huggingface/doc-builder since this will be relevant for other repos as well. (also that way we keep the discussion here relevant to the migration)

[Wauplin]

Closing this (old) issue as we are using doc-builder since quite some time now :)

[julien-c]

yay, thanks for the cleanup @Wauplin