CORE-3092 support metadata v9 #22669

michael-redpanda · 2024-07-31T14:00:37Z

Adds support for metadata API version 8

Backports Required

Release Notes

Improvements

Added support for Metadata API v8

tests/docker/ducktape-deps/sarama-examples

tests/go/sarama/produce_test/main.go

src/v/kafka/server/tests/metadata_test.cc

BenPope · 2024-07-31T16:23:32Z

src/v/kafka/server/tests/metadata_test.cc

+    security::server_first_message send_scram_client_first(
+      kafka::client::transport& client,
+      const security::client_first_message& client_first) {
+        kafka::sasl_authenticate_request client_first_req;
+        {
+            auto msg = client_first.message();
+            client_first_req.data.auth_bytes = bytes(msg.cbegin(), msg.cend());
+        }
+        auto client_first_resp = client.dispatch(client_first_req).get0();
+        BOOST_REQUIRE_EQUAL(
+          client_first_resp.data.error_code, kafka::error_code::none);
+        return security::server_first_message(
+          client_first_resp.data.auth_bytes);
+    }
+
+    security::server_final_message send_scram_client_final(
+      kafka::client::transport& client,
+      const security::client_final_message& client_final) {
+        kafka::sasl_authenticate_request client_last_req;
+        {
+            auto msg = client_final.message();
+            client_last_req.data.auth_bytes = bytes(msg.cbegin(), msg.cend());
+        }
+        auto client_last_resp = client.dispatch(client_last_req).get0();
+
+        BOOST_REQUIRE_EQUAL(
+          client_last_resp.data.error_code, kafka::error_code::none);
+        return security::server_final_message(
+          std::move(client_last_resp.data.auth_bytes));
+    }
+
+    void do_sasl_handshake(kafka::client::transport& client) {
+        kafka::sasl_handshake_request req;
+        req.data.mechanism = security::scram_sha256_authenticator::name;
+
+        auto resp = client.dispatch(req).get0();
+        BOOST_REQUIRE_EQUAL(resp.data.error_code, kafka::error_code::none);
+    }
+
+    void authn_kafka_client(
+      kafka::client::transport& client,
+      const ss::sstring& username,
+      const ss::sstring& password) {
+        do_sasl_handshake(client);
+        const auto nonce = random_generators::gen_alphanum_string(130);
+        const security::client_first_message client_first(username, nonce);
+        const auto server_first = send_scram_client_first(client, client_first);
+
+        BOOST_REQUIRE(
+          std::string_view(server_first.nonce()).starts_with(nonce));
+        BOOST_REQUIRE_GE(
+          server_first.iterations(), security::scram_sha256::min_iterations);
+        security::client_final_message client_final(
+          bytes("n,,"), server_first.nonce());
+        auto salted_password = security::scram_sha256::hi(
+          bytes(password.cbegin(), password.cend()),
+          server_first.salt(),
+          server_first.iterations());
+        client_final.set_proof(security::scram_sha256::client_proof(
+          salted_password, client_first, server_first, client_final));
+
+        auto server_final = send_scram_client_final(client, client_final);
+        BOOST_REQUIRE(!server_final.error());
+
+        auto server_key = security::scram_sha256::server_key(salted_password);
+        auto server_sig = security::scram_sha256::server_signature(
+          server_key, client_first, server_first, client_final);
+
+        BOOST_REQUIRE_EQUAL(server_final.signature(), server_sig);
+    }
+};


There's a lot duplication with kafka/client/sasl_client, was it too hard to extract?

I know :(. That uses a shared_broker_t rather than the raw transport to handle network I/O.

vbotbuildovich · 2024-07-31T16:43:22Z

new failures in https://buildkite.com/redpanda/redpanda/builds/52317#0191095c-d1a2-4a43-aa67-cf5158cb09b1:

"rptest.tests.archival_test.ArchivalTest.test_single_partition_leadership_transfer.cloud_storage_type=CloudStorageType.ABS"

new failures in https://buildkite.com/redpanda/redpanda/builds/52317#0191095c-d1a3-4a66-a94d-568835d9a0df:

"rptest.tests.simple_e2e_test.SimpleEndToEndTest.test_consumer_interruption"

new failures in https://buildkite.com/redpanda/redpanda/builds/52317#01910973-7913-4972-87b2-1e2e8b8af7d7:

"rptest.tests.compatibility.sarama_test.SaramaTest.test_sarama_interceptors"

new failures in https://buildkite.com/redpanda/redpanda/builds/52326#01910a78-c894-43f5-a037-9b7f5a1d3db7:

"rptest.tests.simple_e2e_test.SimpleEndToEndTest.test_consumer_interruption"

new failures in https://buildkite.com/redpanda/redpanda/builds/52326#01912307-28fb-41da-95c8-8f8180506541:

"rptest.tests.simple_e2e_test.SimpleEndToEndTest.test_consumer_interruption"

new failures in https://buildkite.com/redpanda/redpanda/builds/52491#01912459-05fa-4121-a8fa-d6979efeb931:

"rptest.tests.simple_e2e_test.SimpleEndToEndTest.test_consumer_interruption"

vbotbuildovich · 2024-07-31T17:01:06Z

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/52317#0191095c-d1a2-4a43-aa67-cf5158cb09b1

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/52317#0191095c-d1a3-4a66-a94d-568835d9a0df

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/52326#01910a78-c894-43f5-a037-9b7f5a1d3db7

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/52326#01910a78-c893-4658-851f-3ca4331a8a40

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/52326#01910a79-d337-4578-aa7b-31ef0b209a1e

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/53405#01917c10-174e-4760-8f68-9606a7c6612a

michael-redpanda · 2024-07-31T19:11:38Z

Force push b5df3f7:

Updated sarama verion
s/get0/get/
Fixed up intercepters example test due to change in behavior of sarama

BenPope · 2024-08-05T14:49:33Z

CI Failures:

https://redpandadata.atlassian.net/browse/CORE-1531

If `include_topic_authorized_operations` is not set to true in the request, the `topic_authorized_operations` field in the topic metadata response must be the expected default value of -2147483648. Signed-off-by: Michael Boquard <michael@redpanda.com>

We do not need to audit authz checks in situations when returning back a bitfield representing the operations the authenticated user is permitted to perform. Signed-off-by: Michael Boquard <michael@redpanda.com>

michael-redpanda · 2024-08-05T19:20:40Z

Force push 6631b7f:

Added a comment to not update metadata to v11 or beyond without also including support for DescribeCluster v0

michael-redpanda · 2024-08-05T19:20:56Z

Force push 96c9f0c:

Rebased dev, maybe that'll fix the weird CI failures

michael-redpanda · 2024-08-08T14:22:41Z

/ci-repeat 1
release
skip-unit
dt-log-level=trace
skip-redpanda-build
tests/rptest/tests/raft_availability_test.py::RaftAvailabilityTest.test_two_nodes_down

Signed-off-by: Michael Boquard <michael@redpanda.com>

michael-redpanda · 2024-08-22T20:42:12Z

Force push:

f18bd51
- Switched metadata from v9 -> v8
- There is some sort of issue with the java kafka library where it acts differently with consumer groups when metadata v9 is in use causing tests to fail. The implementation of metadata v9 is fine (there's really no difference between v8->v9), but there's something else happening which needs to be better understood before we go to v9
2783b87
- Backed out sarama changes - they require v9

BenPope

LGTM

Signed-off-by: Michael Boquard <michael@redpanda.com>

michael-redpanda · 2024-08-23T13:21:18Z

Force push b68d8d4:

Fixed kafka fixture tests

vbotbuildovich · 2024-08-23T16:41:31Z

/backport v24.2.x

dotnwat · 2024-08-26T21:35:16Z

src/v/kafka/server/handlers/metadata.h

+// IMPORTANT: Do not bump support to v11 (or beyond) unless DescribeCluster v0
+// has been implemented.  v11 drops support for the authorized operations list
+// and moves those lists to DCv0
+// Keep this at v8.  Moving to v9 appears to cause issues with the Kafka Java
+// Client


michael-redpanda requested a review from a team July 31, 2024 14:00

michael-redpanda self-assigned this Jul 31, 2024

michael-redpanda requested review from BenPope and removed request for a team July 31, 2024 14:00

michael-redpanda requested a review from a team as a code owner July 31, 2024 14:00

michael-redpanda requested review from jackietung-redpanda and andijcr and removed request for a team July 31, 2024 14:00

github-actions bot added the area/redpanda label Jul 31, 2024

BenPope reviewed Jul 31, 2024

View reviewed changes

michael-redpanda force-pushed the CORE-3092-Support-metadata-v9 branch from f6e5555 to b5df3f7 Compare July 31, 2024 19:11

michael-redpanda requested a review from BenPope July 31, 2024 19:11

BenPope previously approved these changes Aug 1, 2024

View reviewed changes

andijcr previously approved these changes Aug 1, 2024

View reviewed changes

michael-redpanda dismissed stale reviews from andijcr and BenPope via 6631b7f August 5, 2024 19:18

michael-redpanda force-pushed the CORE-3092-Support-metadata-v9 branch from b5df3f7 to 6631b7f Compare August 5, 2024 19:18

michael-redpanda added 2 commits August 5, 2024 15:18

kafka: Permit override of auditing

46aafdd

We do not need to audit authz checks in situations when returning back a bitfield representing the operations the authenticated user is permitted to perform. Signed-off-by: Michael Boquard <michael@redpanda.com>

michael-redpanda force-pushed the CORE-3092-Support-metadata-v9 branch from 6631b7f to 96c9f0c Compare August 5, 2024 19:18

michael-redpanda requested review from BenPope and andijcr August 5, 2024 19:20

andijcr previously approved these changes Aug 6, 2024

View reviewed changes

BenPope previously approved these changes Aug 6, 2024

View reviewed changes

michael-redpanda requested a review from oleiman August 7, 2024 15:47

kafka: Update minimum metadata API to v9

0cccb5d

Signed-off-by: Michael Boquard <michael@redpanda.com>

michael-redpanda dismissed stale reviews from BenPope and andijcr via f18bd51 August 22, 2024 20:23

michael-redpanda force-pushed the CORE-3092-Support-metadata-v9 branch 2 times, most recently from f18bd51 to 2783b87 Compare August 22, 2024 20:33

michael-redpanda requested review from BenPope and andijcr August 22, 2024 20:42

BenPope previously approved these changes Aug 23, 2024

View reviewed changes

kafka/tests: Added fixture tests for metadata

b68d8d4

Signed-off-by: Michael Boquard <michael@redpanda.com>

michael-redpanda dismissed BenPope’s stale review via b68d8d4 August 23, 2024 13:20

michael-redpanda force-pushed the CORE-3092-Support-metadata-v9 branch from 2783b87 to b68d8d4 Compare August 23, 2024 13:20

michael-redpanda requested a review from BenPope August 23, 2024 13:21

andijcr approved these changes Aug 23, 2024

View reviewed changes

michael-redpanda merged commit 1cba765 into redpanda-data:dev Aug 23, 2024
18 checks passed

vbotbuildovich mentioned this pull request Aug 23, 2024

[v24.2.x] CORE-3092 support metadata v8 #23039

Merged

dotnwat reviewed Aug 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CORE-3092 support metadata v9 #22669

CORE-3092 support metadata v9 #22669

michael-redpanda commented Jul 31, 2024 •

edited

Loading

BenPope Jul 31, 2024

michael-redpanda Jul 31, 2024

vbotbuildovich commented Jul 31, 2024 •

edited

Loading

vbotbuildovich commented Jul 31, 2024 •

edited

Loading

michael-redpanda commented Jul 31, 2024

BenPope commented Aug 5, 2024

michael-redpanda commented Aug 5, 2024 •

edited

Loading

michael-redpanda commented Aug 5, 2024 •

edited

Loading

michael-redpanda commented Aug 8, 2024

michael-redpanda commented Aug 22, 2024 •

edited

Loading

BenPope left a comment

michael-redpanda commented Aug 23, 2024

vbotbuildovich commented Aug 23, 2024

dotnwat Aug 26, 2024

CORE-3092 support metadata v9 #22669

CORE-3092 support metadata v9 #22669

Conversation

michael-redpanda commented Jul 31, 2024 • edited Loading

Backports Required

Release Notes

Improvements

BenPope Jul 31, 2024

Choose a reason for hiding this comment

michael-redpanda Jul 31, 2024

Choose a reason for hiding this comment

vbotbuildovich commented Jul 31, 2024 • edited Loading

vbotbuildovich commented Jul 31, 2024 • edited Loading

michael-redpanda commented Jul 31, 2024

BenPope commented Aug 5, 2024

michael-redpanda commented Aug 5, 2024 • edited Loading

michael-redpanda commented Aug 5, 2024 • edited Loading

michael-redpanda commented Aug 8, 2024

michael-redpanda commented Aug 22, 2024 • edited Loading

BenPope left a comment

Choose a reason for hiding this comment

michael-redpanda commented Aug 23, 2024

vbotbuildovich commented Aug 23, 2024

dotnwat Aug 26, 2024

Choose a reason for hiding this comment

michael-redpanda commented Jul 31, 2024 •

edited

Loading

vbotbuildovich commented Jul 31, 2024 •

edited

Loading

vbotbuildovich commented Jul 31, 2024 •

edited

Loading

michael-redpanda commented Aug 5, 2024 •

edited

Loading

michael-redpanda commented Aug 5, 2024 •

edited

Loading

michael-redpanda commented Aug 22, 2024 •

edited

Loading