Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CORE-1620] Use log_reader in the ntp_archiver #24690

Open
wants to merge 3 commits into
base: dev
Choose a base branch
from
Open

[CORE-1620] Use log_reader in the ntp_archiver #24690

wants to merge 3 commits into from
+562 −45

Conversation

Lazin
Copy link
Contributor

@Lazin Lazin commented Jan 4, 2025

Current ntp_archiver implementation is reading data from segments directly. This makes it tied to storage implementation which makes it difficult to evolve the storage.

This PR introduces new upload mechanism based on storage::log_reader interface and offset_range_size method of the log. Instead of uploading segment files new implementation uploads offset ranges based on heuristics (it tries to produce segments of certain size frequently enough).

Currently, only normal uploads are handled this way. Compacted reupload and adjacent segment merging are not touched.

This is still work in progress. Unit tests that expect uploads to happen on a segment boundary will fail.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.3.x
  • v24.2.x
  • v24.1.x

Release Notes

  • none

Lazin added 3 commits January 4, 2025 10:24
@Lazin
storage: Improve logging in 'offset_range_size'method
bf402a7 
Signed-off-by: Evgeny Lazin <4lazin@gmail.com>
@Lazin
archival: Use dumb pointer in async_data_uploader
fb91d3b 
The ntp_archiver doesn't store the smart pointer so there is no point to
pass smart pointer into the async_data_uploader if async_data_uploader
is invoked by the ntp_archiver.

Signed-off-by: Evgeny Lazin <4lazin@gmail.com>
@Lazin
archiver: Implement upload based on async_data_uploader
7e2465b 
Only normal upload is implemented. The code uses async_data_uploader to
read data from storage layer using reader instead of reading files from
disk directly. The new implementation is a drop in replacement but it's
not swapped yet.

Signed-off-by: Evgeny Lazin <4lazin@gmail.com>
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Jan 4, 2025 •
edited
Loading

Retry command for Build#60291

please wait until all jobs are finished before running the slash command



/ci-repeat 1
tests/rptest/tests/e2e_shadow_indexing_test.py::ShadowIndexingInfiniteRetentionTest.test_segments_not_deleted@{"cloud_storage_type":2}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingLocalRetentionTest.test_shadow_indexing_default_local_retention@{"cloud_storage_type":1,"cluster_remote_write":true,"topic_remote_write":null}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingLocalRetentionTest.test_local_time_based_retention_is_overridden@{"cloud_storage_type":1,"local_retention_ms":3600000}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingLocalRetentionTest.test_shadow_indexing_default_local_retention@{"cloud_storage_type":1,"cluster_remote_write":true,"topic_remote_write":"true"}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingCloudRetentionTest.test_cloud_size_based_retention_application@{"cloud_storage_type":2}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingCloudRetentionTest.test_topic_recovery_retention_settings
tests/rptest/tests/read_replica_e2e_test.py::TestReadReplicaService.test_simple_end_to_end@{"cloud_storage_type_and_url_style":[1,"virtual_host"],"partition_count":10}
tests/rptest/tests/topic_creation_test.py::CreateSITopicsTest.topic_alter_config_test@{"incremental":false}
tests/rptest/tests/archive_retention_test.py::CloudArchiveRetentionTest.test_delete@{"cloud_storage_type":1,"retention_type":"retention.ms"}
tests/rptest/tests/cloud_storage_chunk_read_path_test.py::CloudStorageChunkReadTest.test_fallback_mode
tests/rptest/tests/read_replica_e2e_test.py::TestReadReplicaService.test_identical_lwms_after_delete_records@{"cloud_storage_type":1,"partition_count":5}
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_disabled_partitions@{"cloud_storage_type":1,"filesystem_catalog_mode":true}
tests/rptest/tests/e2e_iam_role_test.py::ShortLivedCredentialsTests.test_short_lived_credentials
tests/rptest/tests/cloud_storage_chunk_read_path_test.py::CloudStorageChunkReadTest.test_read_chunks
tests/rptest/tests/offset_for_leader_epoch_read_replica_test.py::OffsetForLeaderEpochReadReplicaTest.test_offset_for_leader_epoch
tests/rptest/tests/partition_force_reconfiguration_test.py::NodeWiseRecoveryTest.test_node_wise_recovery@{"dead_node_count":2}
tests/rptest/tests/delete_records_test.py::DeleteRecordsTest.test_delete_records_concurrent_truncations@{"cloud_storage_enabled":true,"truncate_point":"random_offset"}
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures@{"cloud_storage_type":1}
tests/rptest/tests/cluster_recovery_test.py::ClusterRecoveryTest.test_bootstrap_with_recovery
tests/rptest/tests/follower_fetching_test.py::FollowerFetchingTest.test_basic_follower_fetching@{"read_from_object_store":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"executed"},"use_alias":false},"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"prepared"},"use_alias":false},"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"out","stage":"executed"},"use_alias":false},"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":null,"use_alias":false},"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"executing"},"use_alias":false},"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"preparing"},"use_alias":false},"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"out","stage":"prepared"},"use_alias":false},"transfer_leadership":true}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_avro_schema@{"cloud_storage_type":1,"query_engine":"spark"}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_e2e_basic@{"cloud_storage_type":1,"filesystem_catalog_mode":false,"query_engine":"trino"}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_upload_after_external_update@{"cloud_storage_type":1}
tests/rptest/tests/datalake/transactions_test.py::DatalakeTransactionTests.test_with_transactions@{"cloud_storage_type":1,"compaction":false}
tests/rptest/tests/full_disk_test.py::LogStorageMaxSizeSI.test_stay_below_target_size@{"cleanup_policy":"compact","log_segment_size":1048576}
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndShadowIndexingTest.test_reset@{"cloud_storage_type":1}
tests/rptest/tests/idempotency_test.py::IdempotencySnapshotDelivery.test_recovery_after_snapshot_is_delivered
tests/rptest/tests/polaris_catalog_smoke_test.py::PolarisCatalogSmokeTest.test_connecting_to_catalog@{"cloud_storage_type":1,"with_tls":false}
tests/rptest/tests/redpanda_connect_test.py::RedpandaConnectTest.test_redpanda_connect_producer
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndShadowIndexingTest.test_reset_spillover@{"cloud_storage_type":1}
tests/rptest/tests/shadow_indexing_compacted_topic_test.py::TSWithAlreadyCompactedTopic.test_initial_upload
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[2,"virtual_host"],"test_case":{"name":"(TS_Read == True, SpilloverManifestUploaded == True)"}}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[2,"virtual_host"],"test_case":{"name":"(TS_Read == True, TS_TxRangeMaterialized == True)"}}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"virtual_host"],"test_case":{"name":"(TS_Delete == True, SpilloverManifestUploaded == True, TS_Spillover_ManifestDeleted == True)"}}
tests/rptest/tests/timequery_test.py::TimeQueryTest.test_timequery@{"batch_cache":false,"cloud_storage":true,"spillover":false}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"virtual_host"],"test_case":{"name":"(TS_Read == True, SpilloverManifestUploaded == True)"}}
tests/rptest/tests/timequery_test.py::TimeQueryTest.test_timequery_empty_local_log
tests/rptest/tests/timequery_test.py::TimeQueryTest.test_timequery_with_trim_prefix@{"cloud_storage":true,"spillover":false}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"virtual_host"],"test_case":{"name":"(TS_Read == True, TS_TxRangeMaterialized == True)"}}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[2,"virtual_host"],"test_case":{"name":"(TS_Delete == True, SpilloverManifestUploaded == True, TS_Spillover_ManifestDeleted == True)"}}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"path"],"test_case":{"name":"(TS_Read == True, AdjacentSegmentMergerReupload == True, SpilloverManifestUploaded == True)"}}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"path"],"test_case":{"name":"(TS_Read == True, TS_Timequery == True)"}}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_fast3@{"cloud_storage_type":1}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_fast1@{"cloud_storage_type":1}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_missing_partition@{"cloud_storage_type":1}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_admin_api_recovery@{"cloud_storage_type":1}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_time_based_retention@{"cloud_storage_type":1}
tests/rptest/tests/adjacent_segment_merging_test.py::AdjacentSegmentMergingTest.test_reupload_of_local_segments@{"acks":-1,"cloud_storage_type":1}
tests/rptest/tests/alter_topic_configuration_test.py::ShadowIndexingGlobalConfig.test_overrides_set
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_prevent_recovery@{"cloud_storage_type":1}
tests/rptest/tests/archival_test.py::ArchivalTest.test_compaction_cluster_config_change@{"cloud_storage_type":2}
tests/rptest/tests/archival_test.py::ArchivalTest.test_reconnect@{"cloud_storage_type":2}
tests/rptest/tests/alter_topic_configuration_test.py::AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node@{"incremental_update":false}
tests/rptest/tests/archival_test.py::ArchivalTest.test_connection_drop@{"cloud_storage_type":2}
tests/rptest/tests/archival_test.py::ArchivalTest.test_write@{"cloud_storage_type_and_url_style":[1,"virtual_host"]}
tests/rptest/tests/archival_test.py::ArchivalTest.test_isolate@{"cloud_storage_type":2}
tests/rptest/tests/archival_test.py::ArchivalTest.test_retention_archival_coordination@{"acks":1,"cloud_storage_type":2}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_many_partitions@{"check_mode":"check_manifest_and_segment_metadata","cloud_storage_type":1}
tests/rptest/tests/cloud_topics_test.py::CloudTopicsTest.test_reconciler_uploads@{"cloud_storage_type":2}
tests/rptest/tests/cloud_storage_inventory_test.py::CloudStorageInventoryTest.test_load_inventory_report
tests/rptest/tests/archival_test.py::ArchivalTest.test_timeboxed_uploads@{"acks":-1,"cloud_storage_type":2}
tests/rptest/tests/archival_test.py::ArchivalTest.test_timeboxed_uploads@{"acks":1,"cloud_storage_type":2}
tests/rptest/tests/cluster_metadata_upload_test.py::ClusterMetadataUploadTest.test_uploads_after_wipe@{"cloud_storage_type":2}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_creating_and_listing_migrations
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_list_mountable_topics
tests/rptest/tests/delete_records_test.py::DeleteRecordsTest.test_delete_records_segment_deletion@{"cloud_storage_enabled":true,"truncate_point":"random_offset"}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingCloudRetentionTest.test_cloud_retention_deleted_segments_count@{"cloud_storage_type":2}
tests/rptest/tests/shadow_indexing_admin_api_test.py::SIAdminApiTest.test_topic_recovery_request_validation
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_missing_topic_manifest@{"cloud_storage_type":1}
tests/rptest/tests/shadow_indexing_admin_api_test.py::SIAdminApiTest.test_bucket_validation@{"cloud_storage_type":1}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingLocalRetentionTest.test_local_time_based_retention_is_overridden@{"cloud_storage_type":1,"local_retention_ms":-1}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingLocalRetentionTest.test_shadow_indexing_default_local_retention@{"cloud_storage_type":1,"cluster_remote_write":false,"topic_remote_write":"true"}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingLocalRetentionTest.test_shadow_indexing_non_default_local_retention@{"cloud_storage_type":1}
tests/rptest/tests/read_replica_e2e_test.py::TestReadReplicaService.test_simple_end_to_end@{"cloud_storage_type_and_url_style":[2,"virtual_host"],"partition_count":10}
tests/rptest/tests/topic_creation_test.py::CreateSITopicsTest.test_shadow_indexing_mode
tests/rptest/tests/archive_retention_test.py::CloudArchiveRetentionTest.test_delete@{"cloud_storage_type":2,"retention_type":"retention.ms"}
tests/rptest/tests/read_replica_e2e_test.py::TestReadReplicaService.test_writes_forbidden@{"cloud_storage_type":1,"partition_count":10}
tests/rptest/tests/delete_records_test.py::DeleteRecordsTest.test_delete_records_concurrent_truncations@{"cloud_storage_enabled":true,"truncate_point":"at_high_watermark"}
tests/rptest/tests/follower_fetching_test.py::FollowerFetchingTest.test_with_leadership_transfers
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndShadowIndexingTest.test_write@{"cloud_storage_type":1}
tests/rptest/tests/e2e_iam_role_test.py::AWSRoleFetchTests.test_write
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_recovery_mode@{"cloud_storage_type":1,"filesystem_catalog_mode":true}
tests/rptest/tests/recovery_mode_test.py::RecoveryModeTest.test_rolling_restart
tests/rptest/tests/cloud_storage_chunk_read_path_test.py::CloudStorageChunkReadTest.test_prefetch_chunks@{"prefetch":3}
tests/rptest/tests/cloud_storage_scrubber_test.py::CloudStorageScrubberTest.test_scrubber@{"cloud_storage_type":1}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":null,"use_alias":false},"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"executing"},"use_alias":false},"transfer_leadership":false}
tests/rptest/tests/cloud_storage_chunk_read_path_test.py::CloudStorageChunkReadTest.test_read_when_chunk_api_disabled
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"preparing"},"use_alias":false},"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"out","stage":"prepared"},"use_alias":false},"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"executed"},"use_alias":false},"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"prepared"},"use_alias":false},"transfer_leadership":true}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_e2e_basic@{"cloud_storage_type":1,"filesystem_catalog_mode":false,"query_engine":"spark"}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"out","stage":"executed"},"use_alias":false},"transfer_leadership":true}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_topic_lifecycle@{"cloud_storage_type":1,"filesystem_catalog_mode":false}
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndHydrationTimeoutTest.test_hydration_completes_when_consumer_killed
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndShadowIndexingTest.test_reset_from_cloud@{"cloud_storage_type":1}
tests/rptest/tests/e2e_topic_recovery_test.py::EndToEndTopicRecovery.test_restore_with_config_batches@{"cloud_storage_type":2,"num_messages":2}
tests/rptest/tests/datalake/partition_movement_test.py::PartitionMovementTest.test_cross_core_movements@{"cloud_storage_type":1}
tests/rptest/tests/offset_for_leader_epoch_archival_test.py::OffsetForLeaderEpochArchivalTest.test_querying_archive
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndSpilloverTest.test_spillover@{"cloud_storage_type":1}
tests/rptest/tests/test_si_cache_space_leak.py::ShadowIndexingCacheSpaceLeakTest.test_si_cache@{"concurrency":2,"message_size":10000,"num_messages":100000}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[2,"virtual_host"],"test_case":{"name":"(TS_Read == True, TS_Timequery == True)"}}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"path"],"test_case":{"name":"(TS_Delete == True, SpilloverManifestUploaded == True, TS_Spillover_ManifestDeleted == True)"}}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"path"],"test_case":{"name":"(TS_Read == True, SpilloverManifestUploaded == True)"}}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"path"],"test_case":{"name":"(TS_Read == True, TS_TxRangeMaterialized == True)"}}
tests/rptest/tests/shadow_indexing_compacted_topic_test.py::ShadowIndexingCompactedTopicTest.test_upload@{"cloud_storage_type":2}
tests/rptest/tests/topic_delete_test.py::TopicDeleteCloudStorageTest.drop_lifecycle_marker_test@{"cloud_storage_type":1}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_fast2@{"cloud_storage_type":1}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[2,"virtual_host"],"test_case":{"name":"(TS_Read == True, AdjacentSegmentMergerReupload == True, SpilloverManifestUploaded == True)"}}
tests/rptest/tests/timequery_test.py::TimeQueryTest.test_timequery_with_spillover_gc_delayed
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_missing_segment@{"cloud_storage_type":1}
tests/rptest/tests/upgrade_test.py::UpgradeFromPriorFeatureVersionCloudStorageTest.test_rolling_upgrade@{"cloud_storage_type":1}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"virtual_host"],"test_case":{"name":"(TS_Read == True, TS_Timequery == True)"}}
tests/rptest/tests/adjacent_segment_merging_test.py::AdjacentSegmentMergingTest.test_reupload_of_local_segments@{"acks":1,"cloud_storage_type":1}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_many_partitions@{"check_mode":"check_manifest_existence","cloud_storage_type":2}
tests/rptest/tests/archival_test.py::ArchivalTest.test_compaction_topic_config_change@{"cloud_storage_type":2}
tests/rptest/tests/archival_test.py::ArchivalTest.test_one_node_reconnect@{"cloud_storage_type":2}
tests/rptest/tests/archival_test.py::ArchivalTest.test_connection_flicker@{"cloud_storage_type":2}
tests/rptest/tests/archival_test.py::ArchivalTest.test_retention_archival_coordination@{"acks":-1,"cloud_storage_type":2}
tests/rptest/tests/archival_test.py::ArchivalTest.test_write@{"cloud_storage_type_and_url_style":[2,"virtual_host"]}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_many_partitions@{"check_mode":"no_check","cloud_storage_type":1}
tests/rptest/tests/archival_test.py::ArchivalTest.test_all_partitions_leadership_transfer@{"cloud_storage_type":2}
tests/rptest/tests/archival_test.py::ArchivalTest.test_timeboxed_uploads@{"acks":0,"cloud_storage_type":2}
tests/rptest/tests/archival_test.py::ArchivalTest.test_single_partition_leadership_transfer@{"cloud_storage_type":2}
tests/rptest/tests/cluster_config_test.py::ClusterConfigAzureSharedKey.test_live_shared_key_change@{"cloud_storage_type":2}
tests/rptest/tests/cluster_metadata_upload_test.py::ClusterMetadataUploadTest.test_uploads_after_restart@{"cloud_storage_type":2}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_conflicting_migrations
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_creating_when_cluster_misconfigured1
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_higher_level_migration_api
tests/rptest/tests/delete_records_test.py::DeleteRecordsTest.test_delete_records_segment_deletion@{"cloud_storage_enabled":true,"truncate_point":"at_segment_boundary"}
tests/rptest/tests/ot_consistency_test.py::OffsetTranslatorConsistencyTest.test_offset_translator_state_consistent
tests/rptest/tests/remote_label_test.py::RemoteLabelsTest.test_clusters_share_bucket@{"cloud_storage_type":1}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingCloudRetentionTest.test_cloud_size_based_retention@{"cloud_storage_type":2}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingCloudRetentionTest.test_cloud_time_based_retention@{"cloud_storage_type":2}
tests/rptest/tests/shadow_indexing_admin_api_test.py::SIAdminApiTest.test_topic_recovery_on_leader
tests/rptest/tests/topic_creation_test.py::CreateTopicsTest.test_no_log_bloat_when_recreating_existing_topics
tests/rptest/tests/topic_creation_test.py::CreateTopicsTest.test_create_topic_with_single_configuration_property
tests/rptest/tests/shadow_indexing_firewall_test.py::ShadowIndexingFirewallTest.test_consume_from_blocked_s3@{"cloud_storage_type":1}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_no_data@{"cloud_storage_type":1}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_vcluster_id@{"cloud_storage_type":1}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_empty_segments@{"cloud_storage_type":1}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_size_based_retention@{"cloud_storage_type":1}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingLocalRetentionTest.test_local_time_based_retention_is_overridden@{"cloud_storage_type":2,"local_retention_ms":3600000}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingLocalRetentionTest.test_shadow_indexing_default_local_retention@{"cloud_storage_type":2,"cluster_remote_write":true,"topic_remote_write":null}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingLocalRetentionTest.test_shadow_indexing_default_local_retention@{"cloud_storage_type":2,"cluster_remote_write":true,"topic_remote_write":"true"}
tests/rptest/tests/topic_creation_test.py::CreateSITopicsTest.test_shadow_indexing_mode_persistence
tests/rptest/tests/read_replica_e2e_test.py::TestReadReplicaService.test_simple_end_to_end@{"cloud_storage_type_and_url_style":[1,"path"],"partition_count":10}
tests/rptest/tests/archive_retention_test.py::CloudArchiveRetentionTest.test_delete@{"cloud_storage_type":1,"retention_type":"retention.bytes"}
tests/rptest/tests/read_replica_e2e_test.py::TestReadReplicaService.test_identical_hwms@{"cloud_storage_type":1,"partition_count":5}
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_disabled_partitions@{"cloud_storage_type":1,"filesystem_catalog_mode":false}
tests/rptest/tests/cloud_storage_chunk_read_path_test.py::CloudStorageChunkReadTest.test_prefetch_chunks@{"prefetch":5}
tests/rptest/tests/e2e_iam_role_test.py::STSRoleFetchTests.test_write
tests/rptest/tests/cloud_storage_chunk_read_path_test.py::CloudStorageChunkReadTest.test_read_when_segment_size_smaller_than_chunk_size
tests/rptest/tests/cloud_storage_usage_test.py::CloudStorageUsageTest.test_cloud_storage_usage_reporting
tests/rptest/tests/timequery_test.py::TestReadReplicaTimeQuery.test_timequery
tests/rptest/tests/partition_force_reconfiguration_test.py::NodeWiseRecoveryTest.test_node_wise_recovery@{"dead_node_count":1}
tests/rptest/tests/datalake/coordinator_retention_test.py::CoordinatorRetentionTest.test_retention@{"cloud_storage_type":1}
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures@{"cloud_storage_type":2}
tests/rptest/tests/delete_records_test.py::DeleteRecordsTest.test_delete_records_concurrent_truncations@{"cloud_storage_enabled":true,"truncate_point":"one_below_high_watermark"}
tests/rptest/tests/cluster_recovery_test.py::ClusterRecoveryTest.test_basic_controller_snapshot_restore
tests/rptest/tests/follower_fetching_test.py::FollowerFetchingTest.test_basic_follower_fetching@{"read_from_object_store":false}
tests/rptest/tests/usage_test.py::UsageTestCloudStorageMetrics.test_usage_manager_cloud_storage
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"out","stage":"preparing"},"use_alias":false},"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":null,"use_alias":true},"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"executing"},"use_alias":true},"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"preparing"},"use_alias":true},"transfer_leadership":false}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"executed"},"use_alias":true},"transfer_leadership":true}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"in","stage":"prepared"},"use_alias":true},"transfer_leadership":true}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_e2e_basic@{"cloud_storage_type":1,"filesystem_catalog_mode":true,"query_engine":"spark"}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_migrated_topic_data_integrity@{"params":{"cancellation":{"dir":"out","stage":"executing"},"use_alias":false},"transfer_leadership":true}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_topic_lifecycle@{"cloud_storage_type":1,"filesystem_catalog_mode":true}
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndShadowIndexingTest.test_reset@{"cloud_storage_type":2}
tests/rptest/tests/e2e_topic_recovery_test.py::EndToEndTopicRecovery.test_restore_with_config_batches@{"cloud_storage_type":1,"num_messages":2}
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndThrottlingTest.test_throttling@{"cloud_storage_type":1}
tests/rptest/tests/e2e_shadow_indexing_test.py::EndToEndShadowIndexingTest.test_reset_spillover@{"cloud_storage_type":2}
tests/rptest/tests/shadow_indexing_compacted_topic_test.py::ShadowIndexingCompactedTopicTest.test_upload@{"cloud_storage_type":1}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[2,"virtual_host"],"test_case":{"name":"(TS_Read == True, TS_Timequery == True, SpilloverManifestUploaded == True)"}}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"path"],"test_case":{"name":"(TS_Read == True, TS_TxRangeMaterialized == True, SpilloverManifestUploaded == True)"}}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[2,"virtual_host"],"test_case":{"name":"(TS_Read == True, SegmentRolledByTimeout == True)"}}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"virtual_host"],"test_case":{"name":"(TS_Read == True, TS_Timequery == True, SpilloverManifestUploaded == True)"}}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"virtual_host"],"test_case":{"name":"(TS_Read == True, SegmentRolledByTimeout == True)"}}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_fast1@{"cloud_storage_type":2}
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"path"],"test_case":{"name":"(TS_Read == True, AdjacentSegmentMergerReupload == True)"}}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_admin_api_recovery@{"cloud_storage_type":2}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_missing_partition@{"cloud_storage_type":2}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_fast3@{"cloud_storage_type":2}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_time_based_retention@{"cloud_storage_type":2}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_prevent_recovery@{"cloud_storage_type":2}
tests/rptest/tests/adjacent_segment_merging_test.py::AdjacentSegmentMergingTest.test_reupload_of_local_segments@{"acks":-1,"cloud_storage_type":2}
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_many_partitions@{"check_mode":"no_check","cloud_storage_type":2}
tests/rptest/tests/alter_topic_configuration_test.py::ShadowIndexingGlobalConfig.test_overrides_remove
tests/rptest/tests/archival_test.py::ArchivalTest.test_compaction_topic_config_change@{"cloud_storage_type":1}
tests/rptest/tests/archival_test.py::ArchivalTest.test_one_node_reconnect@{"cloud_storage_type":1}
tests/rptest/tests/archival_test.py::ArchivalTest.test_connection_flicker@{"cloud_storage_type":1}
tests/rptest/tests/archival_test.py::ArchivalTest.test_retention_archival_coordination@{"acks":-1,"cloud_storage_type":1}
tests/rptest/tests/archival_test.py::ArchivalTest.test_all_partitions_leadership_transfer@{"cloud_storage_type":1}
tests/rptest/tests/archival_test.py::ArchivalTest.test_write@{"cloud_storage_type_and_url_style":[1,"path"]}
tests/rptest/tests/archival_test.py::ArchivalTest.test_single_partition_leadership_transfer@{"cloud_storage_type":1}
tests/rptest/tests/archival_test.py::ArchivalTest.test_timeboxed_uploads@{"acks":0,"cloud_storage_type":1}
tests/rptest/tests/cluster_metadata_upload_test.py::ClusterMetadataUploadTest.test_uploads_after_restart@{"cloud_storage_type":1}
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_inbound_existing_topic
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_conflicting_names
tests/rptest/tests/delete_records_test.py::DeleteRecordsTest.test_delete_records_bounds_checking@{"cloud_storage_enabled":true}
tests/rptest/tests/delete_records_test.py::DeleteRecordsTest.test_delete_records_empty_or_missing_topic_or_partition@{"cloud_storage_enabled":true}
tests/rptest/tests/delete_records_test.py::DeleteRecordsTest.test_delete_records_segment_deletion@{"cloud_storage_enabled":true,"truncate_point":"one_below_high_watermark"}
tests/rptest/tests/delete_records_test.py::DeleteRecordsTest.test_delete_records_topic_policy_change@{"cloud_storage_enabled":true}
tests/rptest/tests/delete_records_test.py::DeleteRecordsTest.test_delete_records_with_transactions@{"cloud_storage_enabled":true}
tests/rptest/tests/offset_for_leader_epoch_archival_test.py::OffsetForLeaderEpochArchivalTest.test_querying_remote_partitions@{"remote_reads":[false,true]}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingCloudRetentionTest.test_cloud_size_based_retention@{"cloud_storage_type":1}
tests/rptest/tests/retention_policy_test.py::ShadowIndexingCloudRetentionTest.test_cloud_time_based_retention@{"cloud_storage_type":1}
tests/rptest/tests/shadow_indexing_admin_api_test.py::SIAdminApiTest.test_topic_recovery_redirects_to_controller_leader
tests/rptest/tests/shadow_indexing_admin_api_test.py::SIAdminApiTest.test_bucket_validation@{"cloud_storage_type":2}
tests/rptest/tests/topic_creation_test.py::CreateTopicsTest.test_create_with_min_rf
tests/rptest/tests/topic_recovery_test.py::TopicRecoveryTest.test_missing_topic_manifest@{"cloud_storage_type":2}

@dotnwat dotnwat changed the title Use log_reader in the ntp_archiver [CORE-1620] Use log_reader in the ntp_archiver Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/redpanda
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Lazin @vbotbuildovich