v23.3.21

Features

• #22957 Schema Registry: added support for the "verbose" query parameter on the schema compatibility checker endpoint by @pgellert in #23028

Bug Fixes

• Fixed a segfault that will occur if mTLS is in use and the auditing client is not configured correctly by @michael-redpanda in #23289
• Fixed possible oversized alloc in the DescribeLogsDir API handler by @michael-redpanda in #23292
• Fixes a bug that would cause retention to be applied to compact-only tiered storage topics. by @andrwng in #23361
• Fixes a regression in schema registry where concurrent requests could lead to memory faults by @oleiman in #23137
• Return NOT_COORDINATOR immediately in group transactional operations if the broker is aware that it is not the group coordinator. by @ztlpn in #23187
• #23108 Schema Registry: fixes a bug in the protobuf compatibility checks of nested fields where nested paths were incorrectly handled. by @pgellert in #23170
• #23222 Fixes a bug that resulted in persisting an invalid topic_properties configuration in the topic_table. by @WillemKauf in #23227

Improvements

• #23079 Schema Registry: Tolerate null metadata and ruleSet when registering or querying a schema by @BenPope in #23081
• #23108 Schema Registry: now reports more specific error messages for Avro and Protobuf schemas when they are incompatible with earlier schemas. by @pgellert in #23170
• PR #23238 [v23.3.x] config: mark cloud_storage_enabled() as needs_restart::yes by @WillemKauf

Full Changelog: v23.3.20...v23.3.21

v24.2.5

Features

• Adds admin API endpoint for enterprise feature info GET /v1/features/enterprise by @oleiman in #23368
• New REST API for mounting/unmounting topics by @bashtanov in #23209
• Schema Registry: verbose compatibility error reporting is now supported for JSON as well by @pgellert in #23299

Bug Fixes

• Fixed a segfault that will occur if mTLS is in use and the auditing client is not configured correctly by @michael-redpanda in #23288
• Fixed possible oversized alloc in the DescribeLogsDir API handler by @michael-redpanda in #23291
• Fixes a bug that resulted in persisting an invalid topic_properties configuration in the topic_table. by @WillemKauf in #23223
• Fixes a bug that would cause retention to be applied to compact-only tiered storage topics. by @andrwng in #23359
• Return NOT_COORDINATOR immediately in group transactional operations if the broker is aware that it is not the group coordinator. by @ztlpn in #23189
• #23316 Fixes crash when consuming from topic partitions with negative indexes by @bharathv in #23317

Improvements

• rpk connect now will be downloaded on the first run. by @r-vasquez in #23351
• rpk: introduce rpk connect install, rpk connect upgrade, and rpk connect uninstall, manual ways to manage rpk connect versioning. by @r-vasquez in #23351
• PR #23127 [v24.2.x] c/members_table: include members table version in the snapshot by @mmaslankaprv
• PR #23155 [v24.2.x] c/topic_config: Explicitly initialize non-class fields by @oleiman
• PR #23175 [v24.2.x] c/tm_stm: fix log format strings by @ztlpn
• PR #23236 [v24.2.x] config: mark cloud_storage_enabled() as needs_restart::yes by @WillemKauf
• PR #23241 [v24.2.x] Migrations storage improvements by @bashtanov
• PR #23242 [v24.2.x] cst/inv: Minor fixes for issues found during scrubber integration by @abhijat
• PR #23325 [v24.2.x] CORE-7268 fix migrated resources raft snapshot application by @bashtanov
• PR #23357 [v24.2.x] admin: add cloud_storage_cache_size check to config_multi_property_validation() by @WillemKauf

Full Changelog: v24.2.4...v24.2.5

v24.1.17

Features

• rpk: supports triggering on-demand partition balancer by @daisukebe in #23395

Bug Fixes

• Fixed a segfault that will occur if mTLS is in use and the auditing client is not configured correctly by @michael-redpanda in #23290
• Fixed possible oversized alloc in the DescribeLogsDir API handler by @michael-redpanda in #23293
• Fixes a bug that would cause retention to be applied to compact-only tiered storage topics. by @andrwng in #23360
• Return NOT_COORDINATOR immediately in group transactional operations if the broker is aware that it is not the group coordinator. by @ztlpn in #23188
• #23221 Fixes a bug that resulted in persisting an invalid topic_properties configuration in the topic_table. by @WillemKauf in #23226

Improvements

• rpk connect now will be downloaded on the first run. by @r-vasquez in #23352
• rpk: introduce rpk connect install, rpk connect upgrade, and rpk connect uninstall, manual ways to manage rpk connect versioning. by @r-vasquez in #23352
• PR #23128 [v24.1.x] c/members_table: include members table version in the snapshot by @mmaslankaprv
• PR #23235 [v24.1.x] CORE-7049 dt/rbac: wait for healthy cluster after restart by @pgellert
• PR #23237 [v24.1.x] config: mark cloud_storage_enabled() as needs_restart::yes by @WillemKauf
• PR #23362 [v24.1.x] admin: add cloud_storage_cache_size check to config_multi_property_validation() (manual backport) by @WillemKauf
• PR #23377 [v24.1.x] Remove some warnings and ignored errors from ducktape log by @ivotron

Full Changelog: v24.1.16...v24.1.17

v24.1.16

Features

• #22956 Schema Registry: added support for the "verbose" query parameter on the schema compatibility checker endpoint by @pgellert in #23027

Bug Fixes

• Fixes a regression in schema registry where concurrent requests could lead to memory faults by @oleiman in #23138
• #23109 Schema Registry: fixes a bug in the protobuf compatibility checks of nested fields where nested paths were incorrectly handled. by @pgellert in #23169
• rpk: fix a bug in rpk cloud login that prevented --no-profile from working when used with SSO login. by @r-vasquez in #23067

Improvements

• #23080 Schema Registry: Tolerate null metadata and ruleSet when registering or querying a schema by @BenPope in #23082
• #23109 Schema Registry: now reports more specific error messages for Avro and Protobuf schemas when they are incompatible with earlier schemas. by @pgellert in #23169

Full Changelog: v24.1.15...v24.1.16

v24.2.4

Bug Fixes

• Fixes a regression in schema registry where concurrent requests could lead to memory faults by @oleiman in #23136
• #23107 Schema Registry: fixes a bug in the protobuf compatibility checks of nested fields where nested paths were incorrectly handled. by @pgellert in #23110
• fixes not fencing subsequent producer requests after its transaction has been expired by @mmaslankaprv in #23152
• rpk: fix a bug in rpk cloud login that prevented --no-profile from working when used with SSO login. by @r-vasquez in #23066

Improvements

• #23077 Schema Registry: Tolerate null metadata and ruleSet when registering or querying a schema by @BenPope in #23078
• #23107 Schema Registry: now reports more specific error messages for Avro and Protobuf schemas when they are incompatible with earlier schemas. by @pgellert in #23110

Full Changelog: v24.2.3...v24.2.4

v24.2.3

Features

• Schema Registry: added support for the "verbose" query parameter on the schema compatibility checker endpoint by @pgellert in #23011

Bug Fixes

• Avoids a crash in Redpanda's layer that could be triggered by empty offset commits. by @andrwng in #22830
• Fix New segment does not line up with previous segment error for compacted topics. by @Lazin in #22924
• Fixes a bug in which empty offset commit requests could result in a crash. by @andrwng in #22895
• Fixes timequeries not being served to remote read replica topics with remote.read=false. by @WillemKauf in #22885
• Made EndTxnRequest fully idempotent by @mmaslankaprv in #22941
• #22936 Fixes a bug in metrics reporting where label values containing special characters (", \, \n) could lead to a malformed metrics response. by @pgellert in #22939
• fixes problem of a node being reported as alive after its node_id has changed by @mmaslankaprv in #22835
• fixes rare assertion that might have been triggered in the raft::heartbeat_manager by @mmaslankaprv in #23056

Improvements

• Added support for Metadata API v8 by @michael-redpanda in #23039
• Added vectorized_kafka_rpc_connections_rejected_rate_limit metric which counts incoming Kafka connections rejected due to the connection rate limit (if set), analogously to the existing vectorized_kafka_rpc_connections_rejected metric which counts rejected connections due to the hitting the open connection limit. by @travisdowns in #22816
• Allows DeleteRecords requests from Kafka clients or rpk topic trim-prefix to be called with truncation_offset <= start_offset without returning an error. The request is instead treated as a no-op. by @WillemKauf in #22945
• Allows the self-test to be completely compatible with a mixed version cluster, in the case of a rolling upgrade. by @WillemKauf in #22922
• Reduce the memory overhead of many small segments. by @rockwotj in #22966
• Return core assignments from health report in /v1/cluster/partitions admin API output. by @ztlpn in #22773
• Schema Registry: 5 new compatibility checks are added for protobuf (ONEOF_FIELD_REMOVED, MULTIPLE_FIELDS_MOVED_TO_ONEOF, REQUIRED_FIELD_{ADDED,REMOVED}, FIELD_NAMED_TYPE_CHANGED, MESSAGE_REMOVED) by @pgellert in #22858
• fixed spurious warning log entries that may be altering users by @mmaslankaprv in #22771
• optimized memory usage with large core count nodes by @mmaslankaprv in #22982

Full Changelog: v24.2.2...v24.2.3

v24.1.15

Bug Fixes

• Avoids a crash in Redpanda's layer that could be triggered by empty offset commits. by @andrwng in #22829
• Fix New segment does not line up with previous segment error for compacted topics. by @Lazin in #22926
• Fixes a bug in which empty offset commit requests could result in a crash. by @andrwng in #22896
• #20135 Fix an issue where transforms would miscalculate their initial start offset, leading to consuming the whole input topic. by @oleiman in #22736
• fixes problem of a node being reported as alive after its node_id has changed by @mmaslankaprv in #22910
• PR #22778 [v24.1.x] cluster: add serde_fields() to start_test_request by @WillemKauf
• PR #22781 [backport] [v24.1.x] miscellaneous idempotency fixes #22687 by @bharathv

Improvements

• Allows DeleteRecords requests from Kafka clients or rpk topic trim-prefix to be called with truncation_offset <= start_offset without returning an error. The request is instead treated as a no-op. by @WillemKauf in #22944
• Made leadership changes related with reconfiguration faster and less disruptive by @mmaslankaprv in #22769
• Reduce the memory overhead of many small segments. by @rockwotj in #22967
• Schema Registry: 5 new compatibility checks are added for protobuf (ONEOF_FIELD_REMOVED, MULTIPLE_FIELDS_MOVED_TO_ONEOF, REQUIRED_FIELD_{ADDED,REMOVED}, FIELD_NAMED_TYPE_CHANGED, MESSAGE_REMOVED) by @pgellert in #22859
• fixed spurious warning log entries that may be altering users by @mmaslankaprv in #22770
• PR #21620 [v24.1.x] [CORE-5632] cluster: Reset cloud storage metric in the cluster::partition::remove_persistent_state by @Lazin
• PR #22684 [v24.1.x] Logging improvements in cluster members subsystem by @mmaslankaprv
• PR #22703 [v24.1.x] schema_registry: Sync Swagger with docs by @BenPope
• PR #22759 [v24.1.x] Disable log cleanup policy for controller log by @mmaslankaprv
• PR #22768 [v24.1.x] CORE-5686 Decouple different raft group shutdown sequences by @mmaslankaprv
• PR #22789 [v24.1.x] config: update docs for cloud_storage_azure_hierarchical_namespace_enabled by @nvartolomei
• PR #22795 [v24.1.x] CORE-6479 dt/random_node_ops: suppress method_not_found log by @pgellert
• PR #22887 [v24.1.x] cluster: fix partition::may_read_from_cloud() for remote read replicas (manual backport) by @WillemKauf
• PR #22943 [backport] [24.1.x] tx: do not throw USE on expired transaction_ids by @bharathv
• PR #22947 [v24.1.x] [CORE-5542] s/segment_utils: pass abort source into compacted index reader by @mmaslankaprv
• PR #22959 [v24.1.x] c/members_table: fixed typo in applying remove node command by @mmaslankaprv
• PR #22960 [v24.1.x] k/group: do not replicate empty offsets commit batch by @mmaslankaprv
• PR #22987 [v24.1.x] kafka: deal with truncation_offset < 0 cases in prefix_truncate() by @WillemKauf
• PR #22993 [backport] [v24.1.x] rm_stm: fix race during leadership transfer by @bharathv

Full Changelog: v24.1.14...v24.1.15

v23.3.20

Bug Fixes

• fixes problem of a node being reported as alive after its node_id has changed by @mmaslankaprv in #22886
• PR #21587 [v23.3.x] cloud storage: Fix deleted_count in clean_up_at_start by @jcipar
• PR #22888 [v23.3.x] cluster: fix partition::may_read_from_cloud() for remote read replicas (manual backport) by @WillemKauf

Improvements

• Schema Registry: 5 new compatibility checks are added for protobuf (ONEOF_FIELD_REMOVED, MULTIPLE_FIELDS_MOVED_TO_ONEOF, REQUIRED_FIELD_{ADDED,REMOVED}, FIELD_NAMED_TYPE_CHANGED, MESSAGE_REMOVED) by @pgellert in #22860

Full Changelog: v23.3.19...v23.3.20

v24.2.2

Features

• rpk: Add rpk registry mode to manage the schema registry mode. by @r-vasquez in #22697

Bug Fixes

• Schema Registry: Fix Swagger for POST /compatibility/subjects/{subject}/versions/{version} by @BenPope in #22646
• PR #22685 [v24.2.x] cloud_storage: minor fixes in topic_mount_handler by @WillemKauf
• PR #22692 [v24.2.x] tests/availability: fix resuming of suspended nodes by @bashtanov
• PR #22757 [backport] [v24.2.x] miscellaneous idempotency fixes #22687 by @bharathv
• PR #22760 [v24.2.x] Disable log cleanup policy for controller log by @mmaslankaprv

Improvements

• Allow rpk cluster self-test start to run, even in a cluster with mixed versions of redpanda (before and after cloudcheck addition in 24.2.x). by @WillemKauf in #22698
• #22649 Schema Registry: Remove spurious log entry: No syntax specified for the proto file by @BenPope in #22650
• rpk topic describe-storage can be used now with internal topics. by @r-vasquez in #22490
• PR #22655 [v24.2.x] Shard placement table: add metrics by @ztlpn
• PR #22658 [v24.2.x] [CORE-6750] schema registry exclusiveMinumum/exclusiveMaximum double support by @andijcr
• PR #22659 [v24.2.x] rpk: update cloudcheck self-test docs by @WillemKauf
• PR #22671 [v24.2.x] cluster: remove cloud storage self-test checks by @WillemKauf
• PR #22682 [v24.2.x] cloud_storage: add topic_mount_manifest (manual backport) by @WillemKauf
• PR #22704 [v24.2.x] schema_registry: Sync Swagger with docs by @BenPope
• PR #22751 [v24.2.x] gha/transform-sdk: pin and centralize language versions by @rockwotj

Full Changelog: v24.2.1...v24.2.2

v24.1.14

Features

• #22603 Schema Registry: Support normalize=true by @BenPope in #22634

Bug Fixes

• Fixes a bug in the purger, which when used with ABS may attempt to download a directory Blob as a manifest. This would completely block the purger from purging partitions for a deleted topic. by @WillemKauf in #22541
• Fixes a bug in which the abs_parse_impl used in xml_sax_parser would throw an exception when processing a List Blobs response with a <BlobPrefix> element within <Blobs>, but without any <Blob> elements. by @WillemKauf in #22539
• Fixes a rare race condition triggering an assert (crash) with idempotent producers. by @bharathv in #22738
• Kafka client used by audit system will now properly handle DNS lookup failure errors by @michael-redpanda in #22525
• Schema Registry: Fix Swagger for POST /compatibility/subjects/{subject}/versions/{version} by @BenPope in #22645
• PR #21589 [v24.1.x] cloud storage: Fix deleted_count in clean_up_at_start by @jcipar
• PR #22322 [v24.1.x] Increased tolerance in fast addition tests by @mmaslankaprv
• PR #22537 [v24.1.x] cloud_storage: add error_outcome::operation_not_supported by @WillemKauf
• PR #22694 [v24.1.x] tests/availability: fix resuming of suspended nodes by @bashtanov
• PR #22700 [v24.1.x] admin: remove throw on unknown self test type in admin_server (manual backport) by @WillemKauf

Improvements

• #22603 Schema Registry: Improve AVRO Normalization by @BenPope in #22634
• #22628 rpk debug bundle: include the result of uname -a by @JFlath in #22630
• #22651 Schema Registry: Remove spurious log entry: No syntax specified for the proto file by @BenPope in #22652

Full Changelog: v24.1.13...v24.1.14