Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Kafka java driver multi shotover tests #1588

Merged
merged 1 commit into from
May 20, 2024

Conversation

rukai
Copy link
Member

@rukai rukai commented Apr 23, 2024

This PR brings us closer to full support for the java driver.
The java driver integration tests that are currently disabled are now enabled but some of the test cases within those integration tests are still disabled.
This is done by calling a minimal_test_suite function and calling it when running the java driver.

The fixes to shotover was to handle prefetching topic metadata for fetches using the newer protocol version used by the java driver.
The new protocol specifies topics by their id instead of their name.
Previously we were only prefetching topic metadata by name but now we also do it by topic id.
This involved:

  1. finding a list of all topic id's in the current message batch. (route_requests method)
  2. Including these topic id's in the metadata request (get_metadata_of_topics method)
    • I chose to revert to a lower api_version when not dealing with topic id's, this isnt required but I figured we may as well support the older kafka versions since it doesn't add much complexity.

@rukai rukai force-pushed the fix_kafka_java_multi_shotover_tests branch from 8be38c2 to 643a8fc Compare May 1, 2024 21:59
@rukai rukai force-pushed the fix_kafka_java_multi_shotover_tests branch from 643a8fc to f6b57db Compare May 17, 2024 01:23
Copy link

codspeed-hq bot commented May 17, 2024

CodSpeed Performance Report

Merging #1588 will not alter performance

Comparing rukai:fix_kafka_java_multi_shotover_tests (e7ab9c9) with main (e69fdac)

Summary

✅ 37 untouched benchmarks

@rukai rukai force-pushed the fix_kafka_java_multi_shotover_tests branch 3 times, most recently from 225d0cd to 296ee6f Compare May 17, 2024 05:12
@rukai rukai force-pushed the fix_kafka_java_multi_shotover_tests branch from 296ee6f to e7ab9c9 Compare May 17, 2024 05:54
@rukai rukai marked this pull request as ready for review May 17, 2024 06:19
@rukai rukai requested a review from conorbros May 17, 2024 06:39
@rukai rukai merged commit c1331f8 into shotover:main May 20, 2024
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants