Serve llms.txt in proxito

[ericholscher]

Summary

• add proxito llms.txt view and URL mirroring robots.txt handling
• add default, delisted, and spam llms.txt templates with user documentation
• cover llms.txt serving and caching with proxito tests and middleware overrides

Testing

• DJANGO_SETTINGS_MODULE=readthedocs.settings.test uv run pytest readthedocs/proxito/tests/test_full.py::TestAdditionalDocViews::test_default_llms_txt readthedocs/proxito/tests/test_full.py::TestAdditionalDocViews::test_custom_llms_txt readthedocs/proxito/tests/test_full.py::TestAdditionalDocViews::test_custom_llms_txt_private_version readthedocs/proxito/tests/test_headers.py::ProxitoHeaderTests::test_cache_headers_llms_txt_with_private_projects_not_allowed readthedocs/proxito/tests/test_headers.py::ProxitoHeaderTests::test_cache_headers_llms_txt_with_private_projects_allowed -q

---

Codex Task

[read-the-docs-community]

Documentation build overview

📚 docs | 🛠️ Build #30471475 | 📁 Comparing 9b90e7a against latest (f47553a)

🔍 Preview build

Show files changed (17 files in total): 📝 14 modified | ➕ 1 added | ➖ 2 deleted

File	Status
link-previews.html	📝 modified
visual-diff.html	📝 modified
config-file/v2.html	📝 modified
intro/antora.html	📝 modified
intro/doctools.html	📝 modified
intro/docusaurus.html	📝 modified
intro/markdoc.html	📝 modified
intro/mdbook.html	📝 modified
intro/mkdocs.html	📝 modified
intro/mystmd.html	➖ deleted
intro/sphinx.html	📝 modified
intro/vitepress.html	📝 modified
reference/features.html	📝 modified
reference/llms.html	➕ added
reference/main-content-detection.html	➖ deleted
server-side-search/index.html	📝 modified
tutorials/index.html	📝 modified