Make autoscaler leader aware

[pierDipi]

Fixes #6732

The autoscaler runs in every controller replica [1], it tries
to scale down on every replica after the given refresh period,
and sometimes the 2 replicas don't agree on which value to use
for the new replicas since the state is lister/cache based,
leading to a too fast scale up or down behavior or sometime
not converging.
(also because of #6733)

Implementations should be using knative/pkg#2675
for enabling leader-aware autoscaler. (PR knative/pkg#2688)

[1]

eventing/pkg/scheduler/statefulset/autoscaler.go

Lines 85 to 103 in 1092472

	func (a *autoscaler) Start(ctx context.Context) {
	attemptScaleDown := false
	pending := int32(0)
	for {
	select {
	case <-ctx.Done():
	return
	case <-time.After(a.refreshPeriod):
	attemptScaleDown = true
	case pending = <-a.trigger:
	attemptScaleDown = false
	}
	// Retry a few times, just so that we don't have to wait for the next beat when
	// a transient error occurs
	a.syncAutoscale(ctx, attemptScaleDown, pending)
	pending = int32(0)
	}
	}

[codecov]

Codecov Report

Patch coverage: 84.61% and no project coverage change.

Comparison is base (eabc874) 80.45% compared to head (0e207a0) 80.46%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6814   +/-   ##
=======================================
  Coverage   80.45%   80.46%           
=======================================
  Files         236      236           
  Lines       12213    12237   +24     
=======================================
+ Hits         9826     9846   +20     
- Misses       1896     1899    +3     
- Partials      491      492    +1     

Impacted Files	Coverage Δ
pkg/scheduler/statefulset/autoscaler.go	77.65% <81.25%> (+0.21%)	⬆️
pkg/scheduler/statefulset/scheduler.go	77.27% <90.00%> (+0.21%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[pierDipi]

/cc @matzew

[matzew]

/lgtm
/approve

[knative-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: matzew, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/scheduler/OWNERS [matzew,pierDipi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment