docs: how to maintain Fedora Copr hypervisors

Closes: fedora-copr#2883
Relates: fedora-copr#2869

doc/maintenance/hypervisors.rst

@@ -0,0 +1,65 @@
+.. _hypervisors:
+
+Fedora Copr hypervisors
+=======================
+
+Fedora Copr has several several hypervisors hosted within Fedora Infrastructure
+lab.  These are monitored using `Nagios probes`_.
+
+Running playbooks
+-----------------
+
+.. warning::
+   * ssh access to `batcave01`_ is required
+
+Running playbooks is as easy as::
+
+    $ ssh batcave01.iad2.fedoraproject.org
+    [yourname@batcave01 ~][PROD-IAD2]$ sudo rbac-playbook groups/copr-hypervisor.yml
+    ...
+
+Reboot
+------
+
+If the hypervisor ends up in an inconsistent state, you might want to
+reboot it (the Resalloc server is able to recover from the reboot, just note
+that the running builds - if any - will be rescheduled by copr-backend).
+
+Normally it is enough to execute the same hypervisors' playbook above but with
+the ``-t trigger_reboot`` option.  But since you typically do not want to reboot
+all the hypervisors, limit the playbook for hosts using ``-l`` pattern, e.g.::
+
+    one-box $ sudo rbac-playbook groups//copr-hypervisor.yml -t trigger_reboot -l 'vmhost-x86-copr02.rdu-cc.fedoraproject.org'
+    all-x86 $ sudo rbac-playbook groups//copr-hypervisor.yml -t trigger_reboot -l '*x86*'
+    ...
+
+Access to KVM, cold rebooting
+-----------------------------
+
+.. warning::
+   * ssh access to ``cloud-noc-os01.rdu-cc.fedoraproject.org`` is needed
+
+You need to use ``cloud-noc-os01.rdu-cc.fedoraproject.org`` hop-box to access
+the management consoles of our hypervisors (consoles are only available within
+the local management network).  Either use ``elinks`` there, or use SOCKS
+proxy::
+
+    $ ssh -ND 9999 cloud-noc-os01.rdu-cc.fedoraproject.org
+
+Then go to Firefox settings, search for "socks", open "Network Settings".  Pick
+"Manual proxy configuration", specify "SOCKS" ``Host: localhost`` and
+``Port: 9999``.  Then you can visit the management console IP within the
+management network range like ``http://172.X.Y.Z``.
+
+The info about IPs and passwords for particular hosts can be found in the team's
+bitwaarden account.  See the Secret Note ``Fedora Copr Hypervisors``.
+
+Add a new hypervisor
+--------------------
+
+There's a separate `how to install hypervisor`_ section in the Fedora Infra
+ansible repo.
+
+.. _`Nagios probes`: https://nagios.fedoraproject.org/nagios/cgi-bin//status.cgi?hostgroup=copr_hypervisor&style=detail
+.. _`how to install hypervisor`: https://pagure.io/fedora-infra/ansible/blob/main/f/roles/copr/hypervisor/README
+.. _`Batcave01`: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/infra-git-repo/

doc/maintenance_documentation.rst

@@ -14,6 +14,7 @@ This section contains information about maintenance topics. You may also be inte
    how_to_upgrade_persistent_instances
    How to manage active chroots <how_to_manage_chroots>
    How to rename chroots <how_to_rename_chroot>
+   Fedora Copr hypervisors <maintenance/hypervisors>
 
 
 .. toctree::