Building a maintainers dashboard with Quetz

[jaimergp]

📌 Summary

Prepare the conda ecosystem for OCI-based storage compatibility.

📝 Background

There is no straightforward way to monitor the operational status of conda-forge's infrastructure.

conda-forge.org/status offers a "maintainers dashboard" with information about:

• Operational status of many of the services that conda-forge relies on, including CI, bots, CDN cloning, webservices, documentation...
• Ongoing migrations (collection of PRs automatically issued by the bots)
• Communication about known incidents

Unfortunately, this is far from being comprehensive view of ongoing maintenance tasks, bottlenecks, or the overall health of the many bots and infrastructure pieces.

Having a detailed picture of the infrastructure and automation tools will significantly improve the maintainers' workflow and aid with identifying critical risks— which is essential to keeping up with the increasing growth and demand from the community.

Quetz is chosen as an open-source server for hosting conda packages, thus allowing for increased transparency and extensibility.
This would have the added benefit of centralizing the currently scattered-across-repositories packaging metadata in a canonical, API-first, performant-at-scale database, laying the foundation for further infrastructure automation and improvements to the building processes.

🚀 Tasks / Deliverables

See issues labeled as mission: dashboard 🎛

ℹ️ References

• conda-forge.org/status: Status dashboard on conda-forge.org
• OCI: Open Container Initiative
• ORAS: OCI Registry As Storage
• oras-py: a Python library to interact with an OCI registry
• channel-mirrors organization: OCI-based mirror of conda-forge and bioconda

[atrawog]

My suggestion would be to use the same approach as https://mybinder.readthedocs.io/en/latest/about/status.html and have a dedicated status page, but use Prometheus/Grafana as a data backend https://grafana.mybinder.org/d/fLoQvRHmk/status?orgId=1

Quetz already has the Middleware in place to do basic metric reporting via prometheus
https://github.com/mamba-org/quetz/blob/main/quetz/metrics/middleware.py and I would improve and extend these reporting statistics to give better insight into the performance of a Quetz instance.

But most of the performance issues of Quetz at the moment are actual problems of things like the S3 storage backend and to catch and diagnose these issue we will need a full blown monitoring system with good Quetz integration that's capably to monitor not just Quetz, but the whole cloud infrastructure conda-forge depends on.