CI Failure (BadLogLines) in CloudStorageScrubberTest.cloud_storage_scrubber_test

[Lazin]

https://buildkite.com/redpanda/redpanda/builds/37723#018ad613-2438-4768-9545-866943cce59f/6-4404

Module: rptest.tests.cloud_storage_scrubber_test
Class:  CloudStorageScrubberTest
Method: test_scrubber
Arguments:
{
  "cloud_storage_type=CloudStorageType": "ABS"
}

test_id:    rptest.tests.cloud_storage_scrubber_test.CloudStorageScrubberTest.test_scrubber.cloud_storage_type=CloudStorageType.ABS
status:     FAIL
run time:   1 minute 52.903 seconds



<BadLogLines nodes=docker-rp-18(16),docker-rp-9(16),docker-rp-10(16) example="ERROR 2023-09-27 10:19:17,594 [shard 1:main] cloud_storage - partition_manifest.cc:1009 - [{kafka/topic-wzsbluyaqk/0}] New replacement segment does not line up with previous segment: {anomaly_types: {{offset_gap}}, new_segment: {o=1906-1975 t={timestamp: 1695809874587}-{timestamp: 1695809953457}}, previous_segment: {{is_compacted: false, size_bytes: 1153440, base_offset: 1766, committed_offset: 1835, base_timestamp: {timestamp: 1695809874556}, max_timestamp: {timestamp: 1695809874569}, delta_offset: 16, ntp_revision: 23, archiver_term: 1, segment_term: 1, delta_offset_end: 16, sname_format: {v3}, metadata_size_hint: 0}}}">
--
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run
  | data = self.run_test()
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test
  | return self.test_context.function(self.test)
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
  | return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  | File "/root/tests/rptest/services/cluster.py", line 142, in wrapped
  | redpanda.raise_on_bad_logs(
  | File "/root/tests/rptest/services/redpanda.py", line 1244, in raise_on_bad_logs
  | raise BadLogLines(bad_lines)
  | rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-18(16),docker-rp-9(16),docker-rp-10(16) example="ERROR 2023-09-27 10:19:17,594 [shard 1:main] cloud_storage - partition_manifest.cc:1009 - [{kafka/topic-wzsbluyaqk/0}] New replacement segment does not line up with previous segment: {anomaly_types: {{offset_gap}}, new_segment: {o=1906-1975 t={timestamp: 1695809874587}-{timestamp: 1695809953457}}, previous_segment: {{is_compacted: false, size_bytes: 1153440, base_offset: 1766, committed_offset: 1835, base_timestamp: {timestamp: 1695809874556}, max_timestamp: {timestamp: 1695809874569}, delta_offset: 16, ntp_revision: 23, archiver_term: 1, segment_term: 1, delta_offset_end: 16, sname_format: {v3}, metadata_size_hint: 0}}}">

[rystsov]

https://buildkite.com/redpanda/redpanda/builds/37765

[VladLazar]

False alarm. The test removes a segment for the manifest in order for the scrubber to detect it. A re-upload tries to add a segment next to the removed one and the STM refuses in order to avoid creating a gap. The log line needs silencing.