Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

{DO NOT MERGE] ccs jenkins fix #944

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

{DO NOT MERGE] ccs jenkins fix #944

wants to merge 12 commits into from

Conversation

honshu
Copy link
Member

@honshu honshu commented Aug 27, 2023

Fixed the conflict causing the jenkins job failure from AK-> CCS.
Below are the PR which caused the conflict.

In AK
apache#14286
apache#13676

In CCS
#901

For #901 and apache#14286 conflict
I have kept the versioning changes of CCS in files
settings.gradle
streams/quickstart/java/pom.xml
streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
streams/quickstart/pom.xml
tests/kafkatest/init.py

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

bmscomp and others added 12 commits August 24, 2023 19:29
…ata Gradle plugin to 1.11.1 (apache#14131)

Reviewers: Divij Vaidya <diviv@amazon.com>
…pache#14026)

Reviewers: Divij Vaidya <diviv@amazon.com>

Co-authored-by: phuchong.tran <phuchong.tran@servicenow.com>
…he#14278)

Reviewers: Yash Mayya <yash.mayya@gmail.com>, Chris Egerton <chrise@aiven.io>
…API endpoint (apache#14244)

Reviewers: Greg Harris <greg.harris@aiven.io>
Reviewers: Divij Vaidya <diviv@amazon.com>
…ime and size. (apache#13561)

This change introduces a remote log segment segment retention cleanup mechanism.

RemoteLogManager runs retention cleanup activity tasks on each leader replica. It assesses factors such as overall size and retention duration, subsequently removing qualified segments from remote storage. This process also involves adjusting the log-start-offset within the UnifiedLog accordingly. It also cleans up the segments which have epochs earlier than the earliest leader epoch in the current leader. 

Co-authored-by: Satish Duggana <satishd@apache.org>
Co-authored-by: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>

Reviewers: Jun Rao <junrao@gmail.com>, Divij Vaidya <diviv@amazon.com, Luke Chen <showuon@gmail.com>, Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Christo Lolov <lolovc@amazon.com>, Jorge Esteban Quilcate Otoya <quilcate.jorge@gmail.com>, Alexandre Dupriez <alexandre.dupriez@gmail.com>, Nikhil Ramakrishnan <ramakrishnan.nikhil@gmail.com>
 Use "add-exports" only when jdk >= 16 in minikdc

Reviewers: Greg Harris <greg.harris@aiven.io>
Reviewers: Mickael Maison <mickael.maison@gmail.com>
…apache#14282)

When starting up a controller for the first time (i.e., with an empty log), it is possible for
MetadataLoader to publish an empty MetadataImage before the activation records of the controller
have been written. While this is not a bug, it could be confusing. This patch closes that gap by
waiting for at least one controller record to be committed before the MetadataLoader starts publishing
images.

Reviewers: Colin P. McCabe <cmccabe@apache.org>
…gning partitions (apache#14127)

This PR adds the following changes to the `TopicBasedRemoteLogMetadataManager`

1. Added a guard in RemoteLogMetadataCache so that the incoming request can be served from the cache iff the corresponding user-topic-partition is initalized
2. Improve error handling in ConsumerTask thread so that is not killed when there are errors in reading the internal topic
3. ConsumerTask initialization should handle the case when there are no records to read
and some other minor changes

Added Unit Tests for the changes

Co-authored-by: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>

Reviewers: Luke Chen <showuon@gmail.com>, Jorge Esteban Quilcate Otoya <quilcate.jorge@gmail.com>, Christo Lolov <lolovc@amazon.com>, Satish Duggana <satishd@apache.org>
Fixed the conflict causing the jenkins job failure from AK-> CCS.
Below are the PR which caused the conflict.

In AK
apache#14286
apache#13676

In CCS
#901

For #901 and apache#14286 conflict
I have kept the versioning changes of CCS in files
       gradle.properties
       settings.gradle
       streams/quickstart/java/pom.xml
       streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
       streams/quickstart/pom.xml
       tests/kafkatest/__init__.py
@honshu honshu requested review from a team as code owners August 27, 2023 18:38
@cla-assistant
Copy link

cla-assistant bot commented Aug 27, 2023

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 11 committers have signed the CLA.

❌ gharris1727
❌ bmscomp
❌ satishd
❌ showuon
❌ Phuc-Hong-Tran
❌ yashmayya
❌ abhijeetk88
❌ see-quick
❌ honshu
❌ mumrah
❌ mimaison
You have signed the CLA already but the status is still pending? Let us recheck it.

@honshu
Copy link
Member Author

honshu commented Aug 28, 2023

Conflicts :

➜  kafka git:(fix-master) ✗ git d
diff --cc gradle.properties
index 716e248096,a303a31419..0000000000
--- a/gradle.properties
+++ b/gradle.properties
@@@ -23,8 -23,8 +23,12 @@@ group=org.apache.kafk
  #  - streams/quickstart/pom.xml
  #  - streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
  #  - streams/quickstart/java/pom.xml
++<<<<<<< HEAD
 +version=7.6.0-0-ccs
++=======
+ version=3.7.0-SNAPSHOT
++>>>>>>> upstream/trunk
  scalaVersion=2.13.11
  task=build
 -org.gradle.jvmargs=-Xmx2g -Xss4m -XX:+UseParallelGC
 +org.gradle.jvmargs=-Xmx2g -Xss100m -XX:+UseParallelGC
  org.gradle.parallel=true
diff --cc settings.gradle
index 0f9a3066b1,79af7b84c3..0000000000
--- a/settings.gradle
+++ b/settings.gradle
@@@ -13,6 -13,40 +13,43 @@@
  // See the License for the specific language governing permissions and
  // limitations under the License.
  
++<<<<<<< HEAD
++=======
+ plugins {
+     id 'com.gradle.enterprise' version '3.14.1'
+     id 'com.gradle.common-custom-user-data-gradle-plugin' version '1.11.1'
+ }
+ 
+ def isGithubActions = System.getenv('GITHUB_ACTIONS') != null
+ def isJenkins = System.getenv('JENKINS_URL') != null
+ def isCI = isGithubActions || isJenkins
+ 
+ gradleEnterprise {
+     server = "https://ge.apache.org"
+     buildScan {
+         capture { taskInputFiles = true }
+         uploadInBackground = !isCI
+         publishAlways()
+         publishIfAuthenticated()
+         obfuscation {
+             // This obfuscates the IP addresses of the build machine in the build scan.
+             // Alternatively, the build scan will provide the hostname for troubleshooting host-specific issues.
+             ipAddresses { addresses -> addresses.collect { address -> "0.0.0.0"} }
+         }
+     }
+ }
+ 
+ buildCache {
+     local {
+         enabled = !isCI
+     }
+ 
+     remote(gradleEnterprise.buildCache) {
+         enabled = false
+     }
+ }
+ 
++>>>>>>> upstream/trunk
  include 'clients',
      'connect:api',
      'connect:basic-auth-extension',
diff --cc streams/quickstart/java/pom.xml
index 02d9d9fc1a,d4dbf26827..0000000000
--- a/streams/quickstart/java/pom.xml
+++ b/streams/quickstart/java/pom.xml
@@@ -26,7 -26,7 +26,11 @@@
      <parent>
          <groupId>org.apache.kafka</groupId>
          <artifactId>streams-quickstart</artifactId>
++<<<<<<< HEAD
 +        <version>7.6.0-0-ccs</version>
++=======
+         <version>3.7.0-SNAPSHOT</version>
++>>>>>>> upstream/trunk
          <relativePath>..</relativePath>
      </parent>
  
diff --cc streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
index 3649ed1f4a,74351ac786..0000000000
--- a/streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
+++ b/streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
@@@ -29,7 -29,7 +29,11 @@@
  
      <properties>
          <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
++<<<<<<< HEAD
 +        <kafka.version>7.6.0-0-ccs</kafka.version>
++=======
+         <kafka.version>3.7.0-SNAPSHOT</kafka.version>
++>>>>>>> upstream/trunk
          <slf4j.version>1.7.36</slf4j.version>
      </properties>
  
diff --cc streams/quickstart/pom.xml
index 1a5c7aa5f6,69b5f5fb42..0000000000
--- a/streams/quickstart/pom.xml
+++ b/streams/quickstart/pom.xml
@@@ -22,7 -22,7 +22,11 @@@
      <groupId>org.apache.kafka</groupId>
      <artifactId>streams-quickstart</artifactId>
      <packaging>pom</packaging>
++<<<<<<< HEAD
 +    <version>7.6.0-0-ccs</version>
++=======
+     <version>3.7.0-SNAPSHOT</version>
++>>>>>>> upstream/trunk
  
      <name>Kafka Streams :: Quickstart</name>
  
diff --cc tests/kafkatest/__init__.py
index cc68bca890,9ab162964c..0000000000
--- a/tests/kafkatest/__init__.py
+++ b/tests/kafkatest/__init__.py
@@@ -21,5 -21,5 +21,10 @@@
  #
  # Instead, in development branches, the version should have a suffix of the form ".devN"
  #
++<<<<<<< HEAD
 +# For example, when Kafka is at version 1.0.0-0, this should be something like "1.0.0-0.dev0"
 +__version__ = '7.6.0-0.dev0'
++=======
+ # For example, when Kafka is at version 1.0.0-SNAPSHOT, this should be something like "1.0.0.dev0"
+ __version__ = '3.7.0.dev0'
++>>>>>>> upstream/trunk

@jolshan
Copy link
Member

jolshan commented Sep 15, 2023

I believe you did this correctly, but for everyone's awareness, we should not pull in apache#13676

@kowshik
Copy link
Member

kowshik commented Sep 15, 2023

@jolshan Just wanted to be sure I understood things right. Are you saying it is OK to pull in apache#13676 into this repository (in this PR), but we should omit it in the subsequent merge with ce-kafka:master?

@jolshan
Copy link
Member

jolshan commented Sep 19, 2023

@kowshik We should not pull it into any confluentinc repo (this one or ce-kafka)
(fwiw, this was addressed in a previous sync, so we should just not choose any conflict resolution that brings in these files)

@jolshan
Copy link
Member

jolshan commented Sep 19, 2023

See revert here: 4fc716a

Confluent does not have the licensing to use gradle enterprise, so we should not have this code in any of our repos.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.