[scheduler, mtsource] Initial HA support for MT Kafka source

[aavarghese]

Fixes #463

Depends on knative/eventing#5360

Proposed Changes

• Controller has new env var to control "scheduler policy type",
• Stateful set adapter has new anti-affinity rules for pods to be placed on new nodes in new zone topology,
• "scheduler policy type" is an enum with two values currently MaxFillup and EvenSpread. For EvenSpread, scheduler places vreplicas for HA by spreading replicas equally across zones. Within a zone, vreplicas are filled up in each pod upto capacity. If MaxFillup strategy is chosen, scheduler places vreplicas in existing pods and fills them up before creating new pods (zones aren't considered)
• Placement type in KafkaSource status has a new type for ZoneName when EvenSpread scheduling
• HA controls are also shared with autoscaler, and state accessor
• State accessor computes the number of zones in cluster and stores a map of node name to zone info

Release Note

[codecov]

Codecov Report

Merging #587 (781f66c) into main (5350b3b) will increase coverage by 0.51%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##             main     #587      +/-   ##
==========================================
+ Coverage   72.29%   72.81%   +0.51%     
==========================================
  Files         140      140              
  Lines        5675     5811     +136     
==========================================
+ Hits         4103     4231     +128     
- Misses       1357     1358       +1     
- Partials      215      222       +7     

Impacted Files	Coverage Δ
pkg/common/scheduler/statefulset/scheduler.go	81.86% <84.80%> (+15.19%)	⬆️
pkg/common/scheduler/statefulset/state.go	88.00% <90.00%> (+2.70%)	⬆️
pkg/common/scheduler/statefulset/autoscaler.go	73.58% <100.00%> (+0.50%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5350b3b...781f66c. Read the comment docs.

[knative-metrics-robot]

The following is the coverage report on the affected files.
Say /test pull-knative-sandbox-eventing-kafka-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/common/scheduler/statefulset/autoscaler.go	76.9%	77.5%	0.6
pkg/common/scheduler/statefulset/scheduler.go	70.9%	86.9%	16.0
pkg/common/scheduler/statefulset/state.go	91.4%	93.9%	2.4

[lionelvillard]

Let's get this in so we can move forward with the issues raised during the review.

Thanks @aavarghese this is amazing work!

[knative-prow-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lionelvillard

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [lionelvillard]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment