Add support for scale subresource to Connect, S2I, MM1, MM2, Bridge and Connectors

[scholzj]

Type of change

• Enhancement / new feature

Description

This PR adds support for scale subresource. Rght now it is added to following resources:

• KafkaConnect, KafkaConnectS2I and KafkaMirrorMaker2
• KafkaMirrorMaker
• KafkaBridge
• KafkaConnector

The scale subresource is useful because it makes it easier to scale the resources and for example use different tools for autoscaling on them.

To allow support for the scale subresource, some new status fields have been added. One is a field which mirrors the corresponding number of replicas. Other one is label selector which is important for use with Horizontal Pod Autoscalers.

For all Deployment based resources, this is done by a new field .status.replicas which mirrors .spec.replicas and .status.podSelector which can be used to select the pods. KafkaConnector is an example of scaling a resource not represented by pods. In this case, the label selector is not present and the .spec.tasksMax and .status.tasksMax represent the replicas field.

The tests include a test for the CRD generator with CRD including status and scale subresources, tests of the api module that test that the scale subresource works and Cluster Operator tests testing the new status fields. To make the api tests pass I had to update kubectl to 1.16 (1.15 has a bug kubernetes/kubernetes#81342 due to which it doesn't work).

This PR also fixes the flaky test KafkaCrdIT.testKafkaWithTemplate which was causing problems while testing this.

Checklist

• Write tests
• Make sure all tests pass
• Update documentation
• Check RBAC rights for Kubernetes / OpenShift roles
• Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
• Update CHANGELOG.md

[scholzj]

@tombentley Before adding this to more resources and writting tests, I wondered what do you think abotu the code and whether you have any early comments.

[tombentley]

Out of interest, what metrics do you think an autoscaler for connectors would use to decide how to scale, since it's Kube has no way to attribute CPU etc to a particular connector within a multi-connector cluster?

[ppatierno]

How much do you think that the scale subresource is useful for the bridge? We know that actually it's not so simple scaling the bridge and we always suggest to use different deployments.

[samuel-hawker]

Aside from the other maintainers concerns this looks good to me!

[scholzj]

How much do you think that the scale subresource is useful for the bridge? We know that actually it's not so simple scaling the bridge and we always suggest to use different deployments.

It could be actually quite useful when using the bridge for producing messages for example, or? That might be even something where you can use the usual autoscaling mechanisms such as CPU utilization etc.

This PR doesn't really do anything else than making it possible for the users to use the scale subresource and nothing more. That is something what the user can already do by editing the resource anyway, this is just an easier way. As with any autoscaling, the user needs to plug it in manually and think about the autoscaling rules. So I think only future will show how useful or useless this is depending on whether users find some good way to use it, but I think there is only a little harm.

[ppatierno]

Yes, good point. The bridge scaling problem is mostly related to consumers; using scaling for producer could make sense.
Other than CPU, if we provide HTTP metrics out of the bridge, they could be used for scaling as well (of course considering the PUT ones for sending messages on specific endpoints). I was wondering how could be interesting integration with KEDA for this.
I will take a look, it's interesting stuff for me :-)

[scholzj]

Other than CPU, if we provide HTTP metrics out of the bridge, they could be used for scaling as well (of course considering the PUT ones for sending messages on specific endpoints). I was wondering how could be interesting integration with KEDA for this.

Yeah ... we should have some metrics in the bridge as well. But that is missing right now. As for KEDA, I need to check with them, but from the docs my udnerstanding is that today, KEDA does not support scaling custom resources and supports only scaling of deployments. So ideally once we have this ready, we should convince them to add support for scaling any resource and not just Deployments.

[ppatierno]

@scholzj I had a chat with KEDA guys. They have the feature planned for KEDA v2 (in one month or so) kedacore/keda#703.
I will plan to work on adding metrics on the bridge for this.

[scholzj]

@ppatierno Right, this is exactly what would make these changes a bit more useful.

[scholzj]

@ppatierno @samuel-hawker @tombentley I added tests etc. and this should now be ready for final review.

[scholzj]

/azp run system-tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[scholzj]

@strimzi-ci run tests

[strimzi-ci]

❌ Test Summary ❌

TEST_PROFILE: acceptance
EXCLUDED_GROUPS: networkpolicies
TEST_CASE: *ST
TOTAL: 25
PASS: 9
FAIL: 16
SKIP: 0
BUILD_NUMBER: 1073
BUILD_ENV: oc cluster up

❗ Test Failures ❗

• io.strimzi.systemtest.bridge.HttpBridgeTlsST in io.strimzi.systemtest.bridge.HttpBridgeTlsST
• io.strimzi.systemtest.oauth.OauthTlsST in io.strimzi.systemtest.oauth.OauthTlsST
• testKafkaAndZookeeperScaleUpScaleDown in io.strimzi.systemtest.rollingupdate.RollingUpdateST
• testAutoRenewAllCaCertsTriggeredByAnno in io.strimzi.systemtest.security.SecurityST
• testCustomSoloCertificatesForRoute in io.strimzi.systemtest.kafka.ListenersST
• testNodePortTls in io.strimzi.systemtest.KafkaST
• testSendMessagesPlainScramSha in io.strimzi.systemtest.KafkaST
• testLoadBalancerTls in io.strimzi.systemtest.KafkaST
• io.strimzi.systemtest.RecoveryST in io.strimzi.systemtest.RecoveryST
• testMirrorMaker2TlsAndTlsClientAuth in io.strimzi.systemtest.MirrorMaker2ST
• testKafkaConnectorWithConnectS2IAndConnectWithSameName in io.strimzi.systemtest.ConnectS2IST
• testProducerConsumerStreamsService in io.strimzi.systemtest.tracing.TracingST
• testMirrorMakerTlsAuthenticated in io.strimzi.systemtest.MirrorMakerST
• testUpdateUser in io.strimzi.systemtest.UserST
• testMultiNodeKafkaConnectWithConnectorCreation in io.strimzi.systemtest.ConnectST
• testKafkaConnectAndConnectorFileSinkPlugin in io.strimzi.systemtest.ConnectST

Re-run command:
@strimzi-ci run tests false profile=acceptance testcase=io.strimzi.systemtest.bridge.HttpBridgeTlsST,io.strimzi.systemtest.oauth.OauthTlsST,io.strimzi.systemtest.rollingupdate.RollingUpdateST#testKafkaAndZookeeperScaleUpScaleDown,io.strimzi.systemtest.security.SecurityST#testAutoRenewAllCaCertsTriggeredByAnno,io.strimzi.systemtest.kafka.ListenersST#testCustomSoloCertificatesForRoute,io.strimzi.systemtest.KafkaST#testNodePortTls,io.strimzi.systemtest.KafkaST#testSendMessagesPlainScramSha,io.strimzi.systemtest.KafkaST#testLoadBalancerTls,io.strimzi.systemtest.RecoveryST,io.strimzi.systemtest.MirrorMaker2ST#testMirrorMaker2TlsAndTlsClientAuth,io.strimzi.systemtest.ConnectS2IST#testKafkaConnectorWithConnectS2IAndConnectWithSameName,io.strimzi.systemtest.tracing.TracingST#testProducerConsumerStreamsService,io.strimzi.systemtest.MirrorMakerST#testMirrorMakerTlsAuthenticated,io.strimzi.systemtest.UserST#testUpdateUser,io.strimzi.systemtest.ConnectST#testMultiNodeKafkaConnectWithConnectorCreation,io.strimzi.systemtest.ConnectST#testKafkaConnectAndConnectorFileSinkPlugin

[scholzj]

/azp run system-tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[exherb]

{
"kind": "Scale",
"apiVersion": "autoscaling/v1",
"metadata": {
"name": "kafka-connect-xapi",
"namespace": "xapi",
"selfLink": "/apis/kafka.strimzi.io/v1beta1/namespaces/xapi/kafkaconnects/kafka-connect-xapi/scale",
"uid": "f9055528-79af-4012-8026-2ac80d3b81ff",
"resourceVersion": "204389134",
"creationTimestamp": "2020-07-31T10:27:37Z"
},
"spec": {
"replicas": 3
},
"status": {
"replicas": 3
}
}

status.selector has no value?

[scholzj]

@exherb Hmm, looks like in the actual resource it is named .status.podSelector ... but in the CRD .status.selector 🤦‍♂️ ... I opened #3432 to fix it.

As a workaround, you should be able to to replace the labelSelectorPath: .status.selector for labelSelectorPath: .status.podSelector and repply the CRDs.