Is your feature request related to a problem? Please describe.

Managing and scaling monitoring configurations for Kubernetes workloads can become cumbersome, especially when dealing with a large number of services and pods. Manually defining PodMonitor and ServiceMonitor resources in Prometheus Operator for each application introduces overhead and is prone to errors. Automating this process based on annotations simplifies the configuration process and ensures that monitoring can dynamically adapt as pods and services are created or updated.

Describe the solution you'd like.

Enable Scribe to automatically create PodMonitor and ServiceMonitor resources based on specific annotations present on pods and services.

Proposed Workflow:

• Annotations:
  • Use default annotations prometheus.io/scrape: "true" with optional parameters like prometheus.io/port and prometheus.io/path.
• Controller Logic:
  • Watch for Pods and Services with these annotations.
  • Dynamically generate and reconcile corresponding PodMonitor and ServiceMonitor resources.
  • Ensure proper scoping for namespaces and labels.
• Garbage Collection:
  • Automatically delete PodMonitor or ServiceMonitor resources if the annotations are removed or the associated resource is deleted.

import (
	"k8s.io/client-go/discovery"
)

type AutoDetect struct {
	dcl      discovery.DiscoveryInterface
}

func (a *AutoDetect) PrometheusCRsAvailability() (bool, error) {
	apiList, err := a.dcl.ServerGroups()
	if err != nil {
		return false, err
	}

	foundServiceMonitor := false
	foundPodMonitor := false
	apiGroups := apiList.Groups
	for i := 0; i < len(apiGroups); i++ {
		if apiGroups[i].Name == "monitoring.coreos.com" {
			for _, version := range apiGroups[i].Versions {
				resources, err := a.dcl.ServerResourcesForGroupVersion(version.GroupVersion)
				if err != nil {
					return false, err
				}

				for _, resource := range resources.APIResources {
					if resource.Kind == "ServiceMonitor" {
						foundServiceMonitor = true
					} else if resource.Kind == "PodMonitor" {
						foundPodMonitor = true
					}
				}
			}
		}
	}

	if foundServiceMonitor && foundPodMonitor {
		return true, nil
	}

	return false, nil
}

Describe alternatives you've considered.

• Manual Definition: Creating PodMonitor and ServiceMonitor resources manually for each workload. This approach is time-intensive and does not scale well for dynamically changing workloads.
• Direct Prometheus Scraping: Scraping directly from pod and service annotations without using PodMonitor and ServiceMonitor. While this works, it bypasses the benefits of Prometheus Operator's CRDs, such as fine-grained configuration and better management.

Additional Information

No response