Store snapshot of manifest to object store

[ShiKaiWi]

Describe This Problem

Currently, the snapshot and normal updates of manifest are appended to WAL, which leads to such problems:

• Complexity of processing the concurrent writing to WAL of snapshotting and normal updating;
• One giant snapshot may exceed the limit of the length of one WAL entry;

Proposal

The basic idea of the proposal is simple: store the snapshot to the object store.

Additional Context

This is a breaking change #402.

[ShiKaiWi]

Overview

Here is a diagram to describe the new storage of manifest combined with WAL and Object Storage

                      ┌─────────────────────────────────────────────────────────────────┐ 
                      │                                                                 │ 
                      │    ┌────────┐                                                   │ 
                      │    │    3   │  /manifest/snapshot/{space_id}/{table_id}/current │ 
┌──────┐              │    └────────┘                                                   │ 
│  4   │              │         │                                                       │ 
├──────┘              │         │                                                       │ 
   3   │─────────┐    │         ▼                                                       │ 
├ ─ ─ ─          │    │    ┌────────┐                                                   │ 
   2   │─────────┼────┼───▶│Snapshot│     /manifest/snapshot/{space_id}/{table_id}/3    │ 
├ ─ ─ ─          │    │    └────────┘                                                   │ 
   1   │─────────┘    │                                                                 │ 
└ ─ ─ ─               └─────────────────────────────────────────────────────────────────┘ 
                                                                                          
  WAL                                          Object Storage                             
                                                                                          

Update

Just insert the new updates into the WAL.

Snapshot

• Generate the latest snapshot with previous snapshot if any and updates after that snapshot;
• Store the latest snapshot to as the /manifest/snapshot/{space_id}/{table_id}/{end_seq} into Object Storage;
• Overwrite the /manifest/snapshot/{space_id}/{table_id}/current to map the current snapshot to the path of the latest snapshot;
• Delete the logs included by the latest snapshot and previous snapshot in the Object Storage, and it doesn't matter if any error ocurr in this stage.

Recover

• Fetch the /manifest/snapshot/{space_id}/{table_id}/current file from Object Storage for the current snapshot file path;
• Fetch the current snapshot;
• Read the new logs after the current snapshot and combine the both two parts to generate the integrate meta data of the table;

[jiacai2050]

Overwrite the /manifest/snapshot/{space_id}/{table_id}/current to map the current snapshot to the path of the latest snapshot;

How will you ensure this operation is atomic?

[ShiKaiWi]

Overwrite the /manifest/snapshot/{space_id}/{table_id}/current to map the current snapshot to the path of the latest snapshot;

How will you ensure this operation is atomic?

I guess the ObjectStore trait should ensure that operation to be atomic.

Currently, for the real object storage service, e.g. AWS3, Aliyun OSS, the consistency model guarantees the put/update is atomic. As for the current "object store" based local file system in CeresDB, the atomic put/update is actually not ensured yet, but I guess it should provide such guarantee to meet the requirements by the ObjectStore trait.

Reference: S3 consistency model