Moving Placeable type and new Scheduler to eventing core

[aavarghese]

Signed-off-by: aavarghese avarghese@us.ibm.com

Fixes knative-extensions/eventing-kafka#925

Proposed Changes

• 🎁 Controller reads and validates the policy profile (containing predicates and priorities) for both scheduling and descheduling from ConfigMaps.
• 🎁 Scheduler imports all the available plugins which invokes their init() and registers all the core and source specific (kafka) plugins.
• 🎁 Scheduler runs all the predicates and priorities specified in the policy (configmap) file to eliminate all the pods that do not meet the predicate, and then prioritizes the feasible pods to find the best placement given the current state of the cluster.
• 🎁 Same for descheduler (has separate predicates and priorities).
• 🎁 Controls are also shared with autoscaler, and state accessor.
• 🎁 Autoscaler runs autoscaling synchronously using a lock so that autoscaling does not run together with the scheduler, and also scales up number of pods using a scale factor based on number of resources in the HA domain
• 🎁 State accessor computes the spread of existing and reserved vreplicas across pods, nodes and zones and stores a map of vpod key to a map of podname/nodename/zonename to vreplica count. It also stores the number of pods, nodes, zones and a list of "schedulable" pods (unschedulable pods are pods that are marked for eviction for compacting, and pods that are on unschedulable nodes (cordoned or unreachable nodes))
• 🎁 Pending vreplicas that cannot be scheduled triggers the autoscaler to add new pods. This leads to rebalancing of all the already placed vreplicas together with the pending vreplicas.
• 🎁 When its time to autoscale down (after refresh period), compacting the pods is attempted if space is available on lower pods. Higher pods are evicted and removed from the "schedulable" pods list. This leads to another rebalancing of all the vreplicas.
• 🎁 Recovery situations are addressed by making sure pods on failed nodes/zones are restarted on new nodes (statefulset adapter settings) and state collector is aware of nodes/zones being down.

Pre-review Checklist

• At least 80% unit test coverage
• E2E tests for any new behavior
• Docs PR for any user-facing impact
• Spec PR for any new API feature
• Conformance test for any change to the spec

Release Note

Next generation Multi-Tenant Scheduler and Descheduler: uses a plugin interface to specify a Scheduler profile with predicates and priorities that run filtering and scoring of pods, respectively to compute the best vreplica placements. When the autoscaler adds new pods, scheduler performs a rebalancing of the already placed vreplicas along with the new vreplicas. A Descheduler profile must be installed when vreplicas need to be scaled down and placements removed. 

Docs

[knative-prow-robot]

@aavarghese: GitHub didn't allow me to request PR reviews from the following users: starting, the, in, PR, for, eventing, move, this.

Note that only knative members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @lionelvillard @pierDipi starting the move in this PR for eventing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aavarghese]

/cc @lionelvillard @pierDipi
starting the move in this PR for eventing

[codecov]

Codecov Report

Merging #5818 (2e1efc1) into main (c804fd2) will decrease coverage by 0.34%.
The diff coverage is 79.44%.

❗ Current head 2e1efc1 differs from pull request most recent head 31a3c56. Consider uploading reports for the commit 31a3c56 to get more accurate results

@@            Coverage Diff             @@
##             main    #5818      +/-   ##
==========================================
- Coverage   82.31%   81.96%   -0.35%     
==========================================
  Files         204      220      +16     
  Lines        6558     7458     +900     
==========================================
+ Hits         5398     6113     +715     
- Misses        794      918     +124     
- Partials      366      427      +61     

Impacted Files	Coverage Δ
...e/lowestordinalpriority/lowest_ordinal_priority.go	66.66% <66.66%> (ø)
...alpriority/remove_with_highest_ordinal_priority.go	66.66% <66.66%> (ø)
...uler/plugins/core/evenpodspread/even_pod_spread.go	70.90% <70.90%> (ø)
...labilitynodepriority/availability_node_priority.go	72.72% <72.72%> (ø)
...labilityzonepriority/availability_zone_priority.go	74.28% <74.28%> (ø)
pkg/scheduler/state/interface.go	75.75% <75.75%> (ø)
pkg/scheduler/statefulset/autoscaler.go	78.07% <78.07%> (ø)
pkg/scheduler/statefulset/scheduler.go	78.59% <78.59%> (ø)
pkg/scheduler/state/state.go	82.31% <82.31%> (ø)
.../kafka/nomaxresourcecount/no_max_resource_count.go	82.35% <82.35%> (ø)
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c804fd2...31a3c56. Read the comment docs.

[matzew]

/assign

[aavarghese]

/retest

[pierDipi]

@aavarghese can you please take a look at https://github.com/knative/eventing/pull/5818/checks?check_run_id=3910420138?

LGTM

[aavarghese]

@aavarghese can you please take a look at https://github.com/knative/eventing/pull/5818/checks?check_run_id=3910420138?

LGTM

@pierDipi that link looks good with the latest build now. (I'm not seeing old checks/failures)

[pierDipi]

/lgtm
/approve
/hold for @matzew review

Linter errors are unrelated.

[knative-prow-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aavarghese, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [pierDipi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[matzew]

I know this is copy over from eventing-kafka. So... LGTM... since it's there already 😉

But... should we do this move after the 1.0 🎉 release (aka 0.27) ?

[lionelvillard]

But... should we do this move after the 1.0 🎉 release (aka 0.27) ?

Any particular concerns?

[matzew]

No, just generally asking what folks are thinking.

Feel free to unhold the PR if there is no issue 😄

[lionelvillard]

@aavarghese can you rebase?

[aavarghese]

@lionelvillard rebase done.
Re-ran goimports . and gofmt -s - no changes created locally. So nothing new to commit.

[lionelvillard]

Are you using go 1.17?

[aavarghese]

Are you using go 1.17?

Thanks. I was at 1.16. Upgraded and committed the changes done manually. Checks ok now.

[aavarghese]

/cc @lionelvillard

[lionelvillard]

@aavarghese there are still errors reported by the linter: Error: G404: Use of weak random number generator (math/rand instead of crypto/rand) (gosec)

[knative-metrics-robot]

The following is the coverage report on the affected files.
Say /test pull-knative-eventing-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/scheduler/placement.go	Do not exist	100.0%
pkg/scheduler/plugins/core/availabilitynodepriority/availability_node_priority.go	Do not exist	81.8%
pkg/scheduler/plugins/core/availabilityzonepriority/availability_zone_priority.go	Do not exist	83.3%
pkg/scheduler/plugins/core/evenpodspread/even_pod_spread.go	Do not exist	81.8%
pkg/scheduler/plugins/core/lowestordinalpriority/lowest_ordinal_priority.go	Do not exist	66.7%
pkg/scheduler/plugins/core/podfitsresources/pod_fits_resources.go	Do not exist	100.0%
pkg/scheduler/plugins/core/removewithavailabilitynodepriority/remove_with_availability_node_priority.go	Do not exist	87.9%
pkg/scheduler/plugins/core/removewithavailabilityzonepriority/remove_with_availability_zone_priority.go	Do not exist	88.9%
pkg/scheduler/plugins/core/removewithevenpodspreadpriority/remove_with_even_pod_spread_priority.go	Do not exist	86.7%
pkg/scheduler/plugins/core/removewithhighestordinalpriority/remove_with_highest_ordinal_priority.go	Do not exist	66.7%
pkg/scheduler/plugins/kafka/nomaxresourcecount/no_max_resource_count.go	Do not exist	88.2%
pkg/scheduler/state/helpers.go	Do not exist	92.6%
pkg/scheduler/state/interface.go	Do not exist	86.2%
pkg/scheduler/state/state.go	Do not exist	88.3%
pkg/scheduler/statefulset/autoscaler.go	Do not exist	85.3%
pkg/scheduler/statefulset/scheduler.go	Do not exist	85.1%

[lionelvillard]

/lgtm
/unhold