[v24.1.x] storage: housekeeping metrics

[nvartolomei]

Backport of PR #24007
Fixes #24035

[vbotbuildovich]

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/57759#0193062a-a7ab-4917-a739-fc77b025fc0d:

"rptest.tests.archive_retention_test.CloudArchiveRetentionTest.test_delete.cloud_storage_type=CloudStorageType.S3.retention_type=retention.ms"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/57811#01930900-b4cb-45b3-9141-4032b91fb530:

"rptest.tests.test_si_cache_space_leak.ShadowIndexingCacheSpaceLeakTest.test_si_cache.message_size=10000.num_messages=100000.concurrency=2"

[vbotbuildovich]

Retry command for Build#57759

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/archive_retention_test.py::CloudArchiveRetentionTest.test_delete@{"cloud_storage_type":1,"retention_type":"retention.ms"}

[Lazin]

/ci-repeat 1
tests/rptest/tests/archive_retention_test.py::CloudArchiveRetentionTest.test_delete@{"cloud_storage_type":1,"retention_type":"retention.ms"}

[nvartolomei]

/ci-repeat 1
skip-redpanda-build
tests/rptest/tests/archive_retention_test.py::CloudArchiveRetentionTest.test_delete@{"cloud_storage_type":1,"retention_type":"retention.ms"}

[nvartolomei]

/dt

[vbotbuildovich]

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/57811#01930900-b4cb-45b3-9141-4032b91fb530

[vbotbuildovich]

Retry command for Build#57811

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/test_si_cache_space_leak.py::ShadowIndexingCacheSpaceLeakTest.test_si_cache@{"concurrency":2,"message_size":10000,"num_messages":100000}

[lf-rep]

Force merge by Nicolae request