feat: use OceanBase as object store #887

MichaelLeeHZ · 2023-05-10T11:31:37Z

Related Issues

Closes #

Detailed Changes

OceanBase is a highly reliable, distributed relational database. Currently, it can be used as a write-ahead log (WAL) for CeresDB. If OceanBase can be used as an object store for CeresDB, it would decrease CeresDB's dependencies and improve stability.
And this change set implements the trait ObjectStore based on OceanBase.

Test Plan

Add new unit tests for the new module.

components/object_store/Cargo.toml

components/object_store/src/obkv/meta.rs

components/object_store/src/obkv/util.rs

components/object_store/src/obkv/mod.rs

## Related Issues Closes #912 ## Detailed Changes Define a trait `CloudMultiPartUploadImpl` that can be implemented by cloud-based object stores, which refers to `object_store` in `arrow_rs`. ## Test Plan Test in #887 --------- Co-authored-by: WEI Xikai <ShiKaiWi@users.noreply.github.com>

analytic_engine/src/manifest/details.rs

analytic_engine/src/storage_options.rs

components/object_store/src/multipart.rs

analytic_engine/src/storage_options.rs

components/object_store/src/obkv/meta.rs

components/object_store/src/obkv/util.rs

components/object_store/src/obkv/mod.rs

jiacai2050 · 2023-06-02T02:16:35Z

@MachaelLee Could you write some design in PR description? Eg:

How a object's metadata is saved?
How data inside a object is saved? does it aligned?

components/table_kv/src/lib.rs

components/table_kv/src/memory.rs

components/table_kv/src/obkv.rs

components/object_store/src/obkv/mod.rs

Co-authored-by: WEI Xikai <ShiKaiWi@users.noreply.github.com>

MichaelLeeHZ · 2023-06-02T07:41:14Z

@MachaelLee Could you write some design in PR description? Eg:

How a object's metadata is saved?

How data inside a object is saved? does it aligned?

I will add some design in #912 later.

components/object_store/src/obkv/meta.rs

components/table_kv/src/lib.rs

components/object_store/src/obkv/mod.rs

Co-authored-by: WEI Xikai <ShiKaiWi@users.noreply.github.com>

components/object_store/src/obkv/meta.rs

components/object_store/src/obkv/mod.rs

components/object_store/src/obkv/meta.rs

components/object_store/src/obkv/mod.rs

ShiKaiWi

LGTM

MichaelLeeHZ requested a review from ShiKaiWi May 10, 2023 11:31

ShiKaiWi reviewed May 12, 2023

View reviewed changes

This was referenced May 22, 2023

Implement object store on Obkv #912

Closed

feat: Implement multipart upload of object store #913

Merged

MichaelLeeHZ marked this pull request as ready for review May 22, 2023 13:18

ShiKaiWi reviewed May 26, 2023

View reviewed changes

analytic_engine/src/manifest/details.rs Outdated Show resolved Hide resolved

analytic_engine/src/storage_options.rs Outdated Show resolved Hide resolved

components/object_store/src/multipart.rs Outdated Show resolved Hide resolved

MichaelLeeHZ force-pushed the feat_obkv_as_oss branch from da5174b to 3781394 Compare May 26, 2023 06:13

MichaelLeeHZ added 22 commits May 26, 2023 14:18

code

b86b0af

code

191f2f8

code

91ea7a1

resolving boxstream

2de52a3

compile

1dd9aaa

add some tests

5510b34

chore

162bb37

clippy

8396b11

delete println

92aee30

delete println

a6764f6

impl multipart

ff70574

multipart

3ecb18a

chore

ce1f7cf

meta

2edc0b3

put

1de1fde

implement put

37e0f76

unit test

60d8315

clippy

5927671

clippy

ebb697a

upload

da56e5e

add new method used only in ceresdb

e36287f

test

1d221c5

MichaelLeeHZ added 7 commits June 1, 2023 17:15

chore

d0761bb

estimate meta json size

e4707f2

configurable

afc72f5

current upload id

40e3eac

pick shard table

a3ef761

fmt

7a9ce91

current upload id

1554ce7

ShiKaiWi reviewed Jun 2, 2023

View reviewed changes

batch delete

bd6061c

ShiKaiWi reviewed Jun 2, 2023

View reviewed changes

MichaelLeeHZ and others added 2 commits June 2, 2023 14:53

rename

25f9249

Update analytic_engine/src/storage_options.rs

7c09c4e

Co-authored-by: WEI Xikai <ShiKaiWi@users.noreply.github.com>

MichaelLeeHZ added 2 commits June 2, 2023 15:53

chore

c4e6a55

fmt

a8609e0

ShiKaiWi reviewed Jun 2, 2023

View reviewed changes

MichaelLeeHZ and others added 4 commits June 2, 2023 17:11

chore

5e87445

Update components/object_store/src/obkv/mod.rs

3359350

Co-authored-by: WEI Xikai <ShiKaiWi@users.noreply.github.com>

fmt

5641ef9

convered parts

472271a

ShiKaiWi reviewed Jun 5, 2023

View reviewed changes

components/object_store/src/obkv/meta.rs Outdated Show resolved Hide resolved

components/object_store/src/obkv/meta.rs Outdated Show resolved Hide resolved

ShiKaiWi reviewed Jun 5, 2023

View reviewed changes

MichaelLeeHZ added 3 commits June 5, 2023 14:38

debug log: format cost time

5f0949d

remove unused code

94671a9

None data part

77997eb

ShiKaiWi reviewed Jun 5, 2023

View reviewed changes

components/object_store/src/obkv/mod.rs Outdated Show resolved Hide resolved

backtrace

957ce6d

ShiKaiWi approved these changes Jun 5, 2023

View reviewed changes

MichaelLeeHZ merged commit 96fc8ab into apache:main Jun 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: use OceanBase as object store #887

feat: use OceanBase as object store #887

MichaelLeeHZ commented May 10, 2023 •

edited by ShiKaiWi

Loading

jiacai2050 commented Jun 2, 2023 •

edited

Loading

MichaelLeeHZ commented Jun 2, 2023

ShiKaiWi left a comment

feat: use OceanBase as object store #887

feat: use OceanBase as object store #887

Conversation

MichaelLeeHZ commented May 10, 2023 • edited by ShiKaiWi Loading

Related Issues

Detailed Changes

Test Plan

jiacai2050 commented Jun 2, 2023 • edited Loading

MichaelLeeHZ commented Jun 2, 2023

ShiKaiWi left a comment

Choose a reason for hiding this comment

MichaelLeeHZ commented May 10, 2023 •

edited by ShiKaiWi

Loading

jiacai2050 commented Jun 2, 2023 •

edited

Loading