feat(v2): block cleaner

[aleks-p]

Adds a block cleaner package for tracking deleted blocks. The purpose is:

• preventing deleted blocks from re-entering the system because of retries / DLQ recovery
• remove deleted blocks from the bucket after a grace period

Some other details:

• block tombstones are stored in memory and in bolt db
• the Raft leader applies a Raft command on an interval
• when applying the command, each metastore instance removes expired tombstones from memory and disk
• the Raft leader also removes the block from the bucket when it applies the command

TODO:

• improve test coverage
• Analyze performance

[kolesnikovae]

Looks good overall, however, I think we need to make sure that only the current leader removes blocks from the object storage. I might be mistaken, but this is not the case in the current version

[aleks-p]

Looks good overall, however, I think we need to make sure that only the current leader removes blocks from the object storage. I might be mistaken, but this is not the case in the current version

I've reworked things a bit, the blockcleaner package now acts mainly as a store for the markers. The Raft command and handling is moved to the metastore package. Instead of a leader check we now check that the instance that submitted the command against the instance running the command (using a generated id for now, but could be switched to use raft server ids).

I've also reduced the number of transactions dramatically, deletion is done in a single update transaction. Still lacking extensive test coverage but looks more robust in a dev deployment.

[kolesnikovae]

LGTM