…ata Gradle plugin to 1.11.1 (apache#14131)

Reviewers: Divij Vaidya <diviv@amazon.com>

…pache#14026)

Reviewers: Divij Vaidya <diviv@amazon.com>

Co-authored-by: phuchong.tran <phuchong.tran@servicenow.com>

…he#14278)

Reviewers: Yash Mayya <yash.mayya@gmail.com>, Chris Egerton <chrise@aiven.io>

…API endpoint (apache#14244)

Reviewers: Greg Harris <greg.harris@aiven.io>

Reviewers: Divij Vaidya <diviv@amazon.com>

…ime and size. (apache#13561)

This change introduces a remote log segment segment retention cleanup mechanism.

RemoteLogManager runs retention cleanup activity tasks on each leader replica. It assesses factors such as overall size and retention duration, subsequently removing qualified segments from remote storage. This process also involves adjusting the log-start-offset within the UnifiedLog accordingly. It also cleans up the segments which have epochs earlier than the earliest leader epoch in the current leader. 

Co-authored-by: Satish Duggana <satishd@apache.org>
Co-authored-by: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>

Reviewers: Jun Rao <junrao@gmail.com>, Divij Vaidya <diviv@amazon.com, Luke Chen <showuon@gmail.com>, Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Christo Lolov <lolovc@amazon.com>, Jorge Esteban Quilcate Otoya <quilcate.jorge@gmail.com>, Alexandre Dupriez <alexandre.dupriez@gmail.com>, Nikhil Ramakrishnan <ramakrishnan.nikhil@gmail.com>

 Use "add-exports" only when jdk >= 16 in minikdc

Reviewers: Greg Harris <greg.harris@aiven.io>

Reviewers: Divij Vaidya <diviv@amazon.com>

Reviewers: Mickael Maison <mickael.maison@gmail.com>

…apache#14282)

When starting up a controller for the first time (i.e., with an empty log), it is possible for
MetadataLoader to publish an empty MetadataImage before the activation records of the controller
have been written. While this is not a bug, it could be confusing. This patch closes that gap by
waiting for at least one controller record to be committed before the MetadataLoader starts publishing
images.

Reviewers: Colin P. McCabe <cmccabe@apache.org>

…gning partitions (apache#14127)

This PR adds the following changes to the `TopicBasedRemoteLogMetadataManager`

1. Added a guard in RemoteLogMetadataCache so that the incoming request can be served from the cache iff the corresponding user-topic-partition is initalized
2. Improve error handling in ConsumerTask thread so that is not killed when there are errors in reading the internal topic
3. ConsumerTask initialization should handle the case when there are no records to read
and some other minor changes

Added Unit Tests for the changes

Co-authored-by: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>

Reviewers: Luke Chen <showuon@gmail.com>, Jorge Esteban Quilcate Otoya <quilcate.jorge@gmail.com>, Christo Lolov <lolovc@amazon.com>, Satish Duggana <satishd@apache.org>

Fixed the conflict causing the jenkins job failure from AK-> CCS.
Below are the PR which caused the conflict.

In AK
apache#14286
apache#13676

In CCS
#901

For #901 and apache#14286 conflict
I have kept the versioning changes of CCS in files
       gradle.properties
       settings.gradle
       streams/quickstart/java/pom.xml
       streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
       streams/quickstart/pom.xml
       tests/kafkatest/__init__.py