CI Failure (Expected to delete some chunk files during seq-cons, none deleted) in CloudStorageChunkReadTest. test_read_chunks

[dotnwat]

https://buildkite.com/redpanda/redpanda/builds/51463#0190aaf1-31cb-40ed-9878-f6c9bcd62031

Module: rptest.tests.cloud_storage_chunk_read_path_test
Class:  CloudStorageChunkReadTest
Method: test_read_chunks

AssertionError('Expected to delete some chunk files during seq-cons, none deleted')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 105, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/cloud_storage_chunk_read_path_test.py", line 264, in test_read_chunks
    raise ex
  File "/root/tests/rptest/tests/cloud_storage_chunk_read_path_test.py", line 250, in test_read_chunks
    assert rm_chunks.deleted_chunks > 0, "Expected to delete some chunk files during seq-cons, none deleted"
AssertionError: Expected to delete some chunk files during seq-cons, none deleted

JIRA Link: CORE-5591

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/15600
*https://buildkite.com/redpanda/redpanda/builds/51485
*https://buildkite.com/redpanda/redpanda/builds/51484
*https://buildkite.com/redpanda/redpanda/builds/51484
*https://buildkite.com/redpanda/redpanda/builds/51483
*https://buildkite.com/redpanda/redpanda/builds/51485
*https://buildkite.com/redpanda/redpanda/builds/51486
*https://buildkite.com/redpanda/redpanda/builds/51487
*https://buildkite.com/redpanda/redpanda/builds/51488
*https://buildkite.com/redpanda/redpanda/builds/51487
*https://buildkite.com/redpanda/vtools/builds/15613
*https://buildkite.com/redpanda/redpanda/builds/51496
*https://buildkite.com/redpanda/redpanda/builds/51488
*https://buildkite.com/redpanda/vtools/builds/15614

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/15627
*https://buildkite.com/redpanda/redpanda/builds/51531
*https://buildkite.com/redpanda/redpanda/builds/51526
*https://buildkite.com/redpanda/vtools/builds/15625
*https://buildkite.com/redpanda/vtools/builds/15626

[abhijat]

After a certain point during test run, all possible chunks have been hydrated IE the file handles are open. The underlying files get deleted but the file handles remain open, so no chunk files are deleted from that point onwards. Possible fixes could include:

• adding a quiet period to allow chunk eviction before running the seq. consumer
• making the check more fine grained, IE assert chunks deleted > 0 only if some chunks were ever present in the cache during consumer scope, eg by looking at download metric
• making the check scope larger, IE assert some chunk is deleted during both the rand. consumer and seq. consumer run.

I opted for the last one in #21427 as the purpose behind deleting chunks was to test the re-materialization of chunks and the assertion in itself is not valuable, only the side effect of removing files is.