Added Public API for Preferred/Unclean Leader Election on Topic Partitions #4845

PratRanj07 · 2024-09-11T06:13:55Z

This PR introduces the public API rd_kafka_ElectLeaders, enabling the execution of Preferred or Unclean leader election for specific topic partitions. Additionally, unit tests are also included.

pranavrth · 2024-09-16T20:33:45Z

Please use plural form of elect leader i.e. elect leaders instead of elect leader. Its all over the changes. Fix those. For example RD_KAFKA_EVENT_ELECTLEADER_RESULT instead of RD_KAFKA_EVENT_ELECTLEADERS_RESULT

pranavrth

First pass of implementation excluding tests and examples.

src/rdkafka.h

src/rdkafka_aux.h

src/rdkafka.h

src/rdkafka_admin.c

src/rdkafka_admin.h

src/rdkafka_aux.c

pranavrth · 2024-09-17T00:06:34Z

tests/0080-admin_ut.c

+                                 rd_kafka_t *rk,
+                                 rd_kafka_queue_t *useq,
+                                 int with_options,
+                                 int etype) {


Take rd_kafka_ElectionType_t input directly.

confluent-cla-assistant · 2024-09-19T09:35:24Z

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

pranavrth

Another round of review. Example and Test remaining.

src/rdkafka.h

src/rdkafka_aux.c

src/rdkafka_aux.h

src/rdkafka_op.h

src/rdkafka_request.c

pranavrth · 2024-09-30T08:45:17Z

Please add following documentations.

Add CHANGELOG
Add this KIP to the supported list of KIPs in INTRODUCTION.md

pranavrth · 2024-09-30T08:45:57Z

Please also ensure there are no memory leaks.

pranavrth

Another round of review.

src/rdkafka.h

pranavrth · 2024-10-01T14:58:31Z

src/rdkafka_admin.c

+        result->err = err;
+        rd_list_init_copy(&result->partitions, partitions);
+        rd_list_copy_to(&result->partitions, partitions, NULL, NULL);
+        return result;


Why are we not using rd_list_copy directly here?

src/rdkafka_admin.c

src/rdkafka_aux.c

pranavrth

Almost there with the code and documentation part.

examples/elect_leaders.c

src/rdkafka.h

src/rdkafka_aux.h

src/rdkafka_request.c

src/rdkafka_admin.c

src/rdkafka.h

src/rdkafka_aux.c

src/rdkafka_aux.h

src/rdkafka_admin.c

src/rdkafka.h

src/rdkafka_admin.c

src/rdkafka_aux.h

src/rdkafka.h

pranavrth

Comments for example.

examples/elect_leaders.c

pranavrth

For the test

tests/0080-admin_ut.c

pranavrth

Few minor changes.

src/rdkafka_admin.c

examples/elect_leaders.c

src/rdkafka_admin.c

pranavrth

LGTM!
Great Work Pratyush!.

src/rdkafka.h

pranavrth · 2024-10-08T08:10:23Z

src/rdkafka_admin.c

+        /* Non empty topic_partition_list should be present */
+        if (elect_leaders->partitions->cnt == 0) {
+                rd_kafka_admin_result_fail(rko, RD_KAFKA_RESP_ERR__INVALID_ARG,
+                                           "No partitions specified");
+                rd_kafka_admin_common_worker_destroy(rk, rko,
+                                                     rd_true /*destroy*/);
+                return;
+        }


I was going through the Java implementation and figured out that null list is also allowed which will trigger leader election for all the topic partitions. We need to change the code to support this. There maybe multiple places we need to do this change.

https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/Admin.java#L1040-L1041

Topic Partition is nullable - https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/ElectLeadersRequest.json#L29

emasab

Thanks Pratyush, left these comments

INTRODUCTION.md

src/rdkafka_admin.c

src/rdkafka_aux.c

src/rdkafka_request.c

src/rdkafka.h

tests/0080-admin_ut.c

examples/elect_leaders.c

INTRODUCTION.md

src/rdkafka.h

examples/elect_leaders.c

Dev elect leaders

requested changes

emasab

LGTM, thanks Pratyush!

pranavrth

Great Work Pratyush!
LGTM!.

PratRanj07 requested a review from a team as a code owner September 11, 2024 06:13

pranavrth requested changes Sep 17, 2024

View reviewed changes

PratRanj07 requested a review from pranavrth September 19, 2024 10:08

airlock-confluentinc bot closed this Sep 25, 2024

airlock-confluentinc bot force-pushed the dev_electLeaders branch from ea378a6 to f55f3ec Compare September 25, 2024 08:25

PratRanj07 reopened this Sep 25, 2024

pranavrth requested changes Sep 30, 2024

View reviewed changes

pranavrth requested changes Oct 1, 2024

View reviewed changes

pranavrth requested changes Oct 2, 2024

View reviewed changes

pranavrth requested changes Oct 3, 2024

View reviewed changes

src/rdkafka_aux.h Outdated Show resolved Hide resolved

src/rdkafka.h Outdated Show resolved Hide resolved

pranavrth requested changes Oct 3, 2024

View reviewed changes

tests/0080-admin_ut.c Outdated Show resolved Hide resolved

airlock-confluentinc bot force-pushed the dev_electLeaders branch from 783aaa9 to 71884f5 Compare October 3, 2024 18:45

pranavrth requested changes Oct 3, 2024

View reviewed changes

src/rdkafka_admin.c Outdated Show resolved Hide resolved

examples/elect_leaders.c Outdated Show resolved Hide resolved

examples/elect_leaders.c Outdated Show resolved Hide resolved

src/rdkafka_admin.c Outdated Show resolved Hide resolved

airlock-confluentinc bot force-pushed the dev_electLeaders branch from 71884f5 to 004ff33 Compare October 3, 2024 20:13

pranavrth approved these changes Oct 3, 2024

View reviewed changes

emasab requested changes Oct 4, 2024

View reviewed changes

src/rdkafka.h Outdated Show resolved Hide resolved

src/rdkafka.h Outdated Show resolved Hide resolved

src/rdkafka.h Outdated Show resolved Hide resolved

PratRanj07 added 7 commits October 7, 2024 17:33

electLeaders Api 1st draft

020a855

Minor change

d5ecad3

Elect Leaders API final

0293d14

minor change

3041d64

Added a function

7ca1b5d

Minor Change

f8484b2

removed binary file from pr

82e5a0c

PratRanj07 and others added 9 commits October 7, 2024 17:33

new comments

4cca9db

removing election type check

8841fe2

removing merge conflicts

5944466

changelog.md changes

856b953

new changes

46ce5a3

Fixing failing tests

9f71d23

Moved top level error code from response to event

ad09687

changed topic partition result structure

2960d89

style fix

b49e0db

airlock-confluentinc bot force-pushed the dev_electLeaders branch from 3557490 to b49e0db Compare October 7, 2024 12:03

PratRanj07 requested a review from emasab October 7, 2024 13:21

pranavrth requested changes Oct 8, 2024

View reviewed changes

remove null partitions check

be366c8

emasab requested changes Oct 8, 2024

View reviewed changes

requested changes

8020ab0

airlock-confluentinc bot force-pushed the dev_electLeaders branch from bbae851 to d8f9d83 Compare October 8, 2024 19:59

emasab requested changes Oct 9, 2024

View reviewed changes

PratRanj07 and others added 2 commits October 9, 2024 13:59

introduction md formatting

c62e9cf

Merge pull request #73 from confluentinc/dev_electLeaders

50f6d1e

Dev elect leaders

airlock-confluentinc bot force-pushed the dev_electLeaders branch from d8f9d83 to 50f6d1e Compare October 9, 2024 08:37

PratRanj07 and others added 3 commits October 9, 2024 14:16

requested changes

271a8bd

Merge branch 'airlock/dev_electLeaders' into dev_electLeaders

894ec1c

Merge pull request #74 from confluentinc/dev_electLeaders

9b40da8

requested changes

PratRanj07 requested review from emasab and pranavrth October 9, 2024 08:57

partitions cnt print

55dd966

emasab approved these changes Oct 9, 2024

View reviewed changes

pranavrth approved these changes Oct 9, 2024

View reviewed changes

PratRanj07 merged commit 0cd23c6 into master Oct 9, 2024
3 checks passed

PratRanj07 deleted the dev_electLeaders branch October 9, 2024 11:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added Public API for Preferred/Unclean Leader Election on Topic Partitions #4845

Added Public API for Preferred/Unclean Leader Election on Topic Partitions #4845

PratRanj07 commented Sep 11, 2024

pranavrth commented Sep 16, 2024

pranavrth left a comment

pranavrth Sep 17, 2024

confluent-cla-assistant bot commented Sep 19, 2024

pranavrth left a comment

pranavrth commented Sep 30, 2024

pranavrth commented Sep 30, 2024

pranavrth left a comment

pranavrth Oct 1, 2024

pranavrth left a comment

pranavrth left a comment

pranavrth left a comment

pranavrth left a comment

pranavrth left a comment

pranavrth Oct 8, 2024

emasab left a comment

emasab left a comment

pranavrth left a comment

Added Public API for Preferred/Unclean Leader Election on Topic Partitions #4845

Added Public API for Preferred/Unclean Leader Election on Topic Partitions #4845

Conversation

PratRanj07 commented Sep 11, 2024

pranavrth commented Sep 16, 2024

pranavrth left a comment

Choose a reason for hiding this comment

pranavrth Sep 17, 2024

Choose a reason for hiding this comment

confluent-cla-assistant bot commented Sep 19, 2024

pranavrth left a comment

Choose a reason for hiding this comment

pranavrth commented Sep 30, 2024

pranavrth commented Sep 30, 2024

pranavrth left a comment

Choose a reason for hiding this comment

pranavrth Oct 1, 2024

Choose a reason for hiding this comment

pranavrth left a comment

Choose a reason for hiding this comment

pranavrth left a comment

Choose a reason for hiding this comment

pranavrth left a comment

Choose a reason for hiding this comment

pranavrth left a comment

Choose a reason for hiding this comment

pranavrth left a comment

Choose a reason for hiding this comment

pranavrth Oct 8, 2024

Choose a reason for hiding this comment

emasab left a comment

Choose a reason for hiding this comment

emasab left a comment

Choose a reason for hiding this comment

pranavrth left a comment

Choose a reason for hiding this comment