fix: Self Monitor storage volume fill-up despite retention rules

[hisarbalik]

Description

Changes proposed in this pull request (what was done and why):

• Recently we have observed on several clusters the self-monitor pod instances are getting evicted due to the error message Usage of EmptyDir volume exceeds the limit "100Mi". despite retention rules of 2 hours or 50MB of size. The TSDB storage size-based retention works in a way, it includes the WAL, checkpoint, m-mapped chunks, and persistent blocks, although TSDB counts all of those data to decide any deletion, WAL, checkpoints, and m-mapped chunks required for normal operation of TSDB. Only persistence blocks are deleted even if all those data blocks go beyond the configured retention size. The WAL segments can grow up to 128MB before compacting, and Prometheus will keep at least 3 WAL files. So to ensure Self Monitor doesn't get evicted due to volume exceeding the storage limit, we should increase volume size at 3 * WAL segment size + some space for the other data types.

Changes refer to particular issues, PRs or documents:

Traceability

• The PR is linked to a GitHub issue.
• The follow-up issues (if any) are linked in the Related Issues section.
• If the change is user-facing, the documentation has been adjusted.
• The feature is unit-tested.
• The feature is e2e-tested.