Adding 1 second delay between pod health checks requeue

[pjuarezd]

Right now is running too often causing a lot of logs in Operator too quick, usually MinIO will take some seconds to restart.

Operator showing the following error tens of time per second

I0613 16:45:56.643423       1 monitoring.go:123] 'tenant-1/myminio' Failed to get cluster health: Get "https://minio.tenant-1.svc.cluster.local/minio/health/cluster": dial tcp 10.96.194.110:443: i/o timeout

[shtripat]

lgtm

[ramondeklein]

Alternative options (not sure if that would be better):

• Use exponential back-off
• Reduce logging after a while

[pjuarezd]

Alternative options (not sure if that would be better):

• Use exponential back-off
• Reduce logging after a while

this queue already have a rate limiter

operator/pkg/controller/main-controller.go

Line 280 in 54e4091

healthCheckQueue: queue.NewRateLimitingQueueWithConfig(MinIOControllerRateLimiter(), queue.RateLimitingQueueConfig{Name: "TenantsHealth"}),

operator/pkg/controller/main-controller.go

Lines 1446 to 1452 in 54e4091

	func MinIOControllerRateLimiter() queue.RateLimiter {
	return queue.NewMaxOfRateLimiter(
	queue.NewItemExponentialFailureRateLimiter(5*time.Second, 60*time.Second),
	// 10 qps, 100 bucket size. This is only for retry speed and its only the overall factor (not per item)
	&queue.BucketRateLimiter{Limiter: rate.NewLimiter(rate.Limit(10), 100)},
	)
	}