Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add metrics to monitor the ConsumerGroup replicas changes #3386

Merged
merged 1 commit into from
Oct 23, 2023
Merged

Add metrics to monitor the ConsumerGroup replicas changes #3386

merged 1 commit into from
Oct 23, 2023

Conversation

pierDipi
Copy link
Member

Understanding how KEDA scales up and down Knative Kafka
resources is critical to debugging any performance problem
that the autoscaler might introduce, so we need to expose new
metrics that record mutations to Knative resources done by KEDA.

This is the table representation of the new metrics

Metric Description Type Tags Unit
kafka_controller_consumer_group_expected_replicas Number of expected replicas for a given Kafka consumer group resource Gauge consumer_name (resource name) namespace_name (resource namespace) consumer_kind (resource Kind, enum: KafkaSource, Trigger, Subscription) Dimensionless
kafka_controller_consumer_group_ready_replicas Number of ready replicas for a given Kafka consumer group resource Gauge consumer_name (resource name) namespace_name (resource namespace) consumer_kind (resource Kind, enum: KafkaSource, Trigger, Subscription) Dimensionless

note: resource in this context means user-facing resources: KafkaSource, Trigger, Subscription; so no internal or generated names should be used.

Part of #1537

Proposed Changes

  • Add metrics to monitor the ConsumerGroup replicas changes

Release Note

Add new metrics to monitor the ConsumerGroup replicas change

Metric | Description | Type | Tags | Unit
-- | -- | -- | -- | --
kafka_controller_consumer_group_expected_replicas | Number of expected replicas for a given Kafka consumer group resource | Gauge | consumer_name (resource name) namespace_name (resource namespace) consumer_kind (resource Kind, enum: KafkaSource, Trigger, Subscription) | Dimensionless
kafka_controller_consumer_group_ready_replicas | Number of ready replicas for a given Kafka consumer group resource | Gauge | consumer_name (resource name) namespace_name (resource namespace) consumer_kind (resource Kind, enum: KafkaSource, Trigger, Subscription) | Dimensionless

Docs

@pierDipi
Record expected and ready replicas for consumer group resource
9d1c673 
Understanding how KEDA scales up and down Knative Kafka
resources is critical to debugging any performance problem
that the autoscaler might introduce, so we need to expose new
metrics that record mutations to Knative resources done by KEDA.

`kafka_controller_consumer_group_expected_replicas` records number
of expected replicas for a given Kafka consumer group resource.

`kafka_controller_consumer_group_ready_replicas` records number
of ready replicas for a given Kafka consumer group resource.

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
@knative-prow knative-prow bot requested review from matzew and odacremolbap October 10, 2023 15:38
@knative-prow knative-prow bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Oct 10, 2023
@pierDipi pierDipi requested review from Cali0707 and creydr October 10, 2023 15:39
@codecov
Copy link

codecov bot commented Oct 10, 2023 •
edited
Loading

Codecov Report

Merging #3386 (9d1c673) into main (13ce94b) will increase coverage by 12.37%.
Report is 15 commits behind head on main.
The diff coverage is n/a.

@@              Coverage Diff              @@
##               main    #3386       +/-   ##
=============================================
+ Coverage     58.47%   70.84%   +12.37%     
- Complexity        0      761      +761     
=============================================
  Files            91       90        -1     
  Lines          9245     3077     -6168     
  Branches          0      265      +265     
=============================================
- Hits           5406     2180     -3226     
+ Misses         3410      730     -2680     
+ Partials        429      167      -262
Flag Coverage Δ
java-unittests 70.84% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 181 files with indirect coverage changes

@pierDipi
Copy link
Member Author

/retest-required

4 similar comments
@pierDipi
Copy link
Member Author

/retest-required

@pierDipi
Copy link
Member Author

/retest-required

@pierDipi
Copy link
Member Author

/retest-required

@pierDipi
Copy link
Member Author

/retest-required

@pierDipi
Copy link
Member Author

/test reconciler-tests-namespaced-broker

Cali0707
Cali0707 approved these changes Oct 11, 2023
View reviewed changes
Copy link
Member

@Cali0707 Cali0707 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@knative-prow knative-prow bot added the lgtm Indicates that a PR is ready to be merged. label Oct 11, 2023
@knative-prow
Copy link

knative-prow bot commented Oct 11, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Cali0707, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@pierDipi
Copy link
Member Author

/retest-required

5 similar comments
@pierDipi
Copy link
Member Author

/retest-required

@pierDipi
Copy link
Member Author

/retest-required

@Cali0707
Copy link
Member

/retest-required

@Cali0707
Copy link
Member

/retest-required

@Cali0707
Copy link
Member

/retest-required

@Cali0707
Copy link
Member

/test channel-integration-tests-sasl-ssl

@pierDipi
Copy link
Member Author

/retest-required

@pierDipi
Copy link
Member Author

/test channel-integration-tests-sasl-plain

@pierDipi
Copy link
Member Author

/test reconciler-tests-namespaced-broker

@pierDipi
Copy link
Member Author

/test upgrade-tests

@pierDipi
Copy link
Member Author

/test reconciler-tests

@knative-prow knative-prow bot merged commit 9ef55f2 into knative-extensions:main Oct 23, 2023
6 checks passed
@Cali0707
Copy link
Member

It merged 🎉

pierDipi added a commit to pierDipi/eventing-kafka-broker that referenced this pull request Oct 24, 2023
@pierDipi
Record expected and ready replicas for consumer group resource (knati…
4924239 
…ve-extensions#3386)

Understanding how KEDA scales up and down Knative Kafka
resources is critical to debugging any performance problem
that the autoscaler might introduce, so we need to expose new
metrics that record mutations to Knative resources done by KEDA.

`kafka_controller_consumer_group_expected_replicas` records number
of expected replicas for a given Kafka consumer group resource.

`kafka_controller_consumer_group_ready_replicas` records number
of ready replicas for a given Kafka consumer group resource.

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
@pierDipi pierDipi deleted the record-expected-ready-replicas-for-consumer-group branch October 24, 2023 08:45
openshift-ci bot pushed a commit to openshift-knative/eventing-kafka-broker that referenced this pull request Oct 24, 2023
@pierDipi
Record expected and ready replicas for consumer group resource (knati…
603adb9 
…ve-extensions#3386) (#879)

Understanding how KEDA scales up and down Knative Kafka
resources is critical to debugging any performance problem
that the autoscaler might introduce, so we need to expose new
metrics that record mutations to Knative resources done by KEDA.

`kafka_controller_consumer_group_expected_replicas` records number
of expected replicas for a given Kafka consumer group resource.

`kafka_controller_consumer_group_ready_replicas` records number
of ready replicas for a given Kafka consumer group resource.

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
@pierDipi pierDipi restored the record-expected-ready-replicas-for-consumer-group branch August 5, 2024 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Cali0707 Cali0707 Cali0707 approved these changes

@matzew matzew Awaiting requested review from matzew

@odacremolbap odacremolbap Awaiting requested review from odacremolbap

@creydr creydr Awaiting requested review from creydr

Assignees

@Cali0707 Cali0707

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/control-plane lgtm Indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
Eventing Kafka Broker Scheduling and Scaling
Status: ✅ Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pierDipi @Cali0707