CI Failure (BLL: Failed to hydrate chunk start 0, error: NotFound) in ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy

[vbotbuildovich]

https://buildkite.com/redpanda/vtools/builds/12842

Module: rptest.tests.e2e_shadow_indexing_test
Class: ShadowIndexingWhileBusyTest
Method: test_create_or_delete_topics_while_busy
Arguments: {
    "short_retention": true,
    "cloud_storage_type": 1
}

test_id:    ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy
status:     FAIL
run time:   914.928 seconds

<BadLogLines nodes=ip-172-31-15-134(10) example="ERROR 2024-04-12 10:24:40,464 [shard 1:fetc] cloud_storage - [fiber4776 4d0945a0/kafka/topic-rqfxuuiaxi/18_52/2922-2961-20977360-1-v1.log.1] - segment_chunk_api.cc:226 - Failed to hydrate chunk start 0, error: NotFound">
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 177, in wrapped
    redpanda.raise_on_bad_logs(
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1523, in raise_on_bad_logs
    lsearcher.search_logs(_searchable_nodes)
  File "/home/ubuntu/redpanda/tests/rptest/services/utils.py", line 156, in search_logs
    raise BadLogLines(bad_loglines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=ip-172-31-15-134(10) example="ERROR 2024-04-12 10:24:40,464 [shard 1:fetc] cloud_storage - [fiber4776 4d0945a0/kafka/topic-rqfxuuiaxi/18_52/2922-2961-20977360-1-v1.log.1] - segment_chunk_api.cc:226 - Failed to hydrate chunk start 0, error: NotFound">

JIRA Link: CORE-2352

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12857

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12865

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12890

[nvartolomei]

There seem to be some race conditions between archival (write path) and cloud storage (read path). E.g. the one leading to Failed to hydrate chunk start 0, error: NotFound. The archiver does garbage collection from the cloud while we still try to read the segment. I believe we have 2 (3?) flavors of this problem: one in the context of RRR (harder to solve #17857), (follower fetching too?), and a local one (likely easier to solve) which is the one in the issue.

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/13071

[nvartolomei]

Fixed by #17828