feat: init metric engine structure #1554

jiacai2050 · 2024-08-21T09:52:41Z

Rationale

See #1558

Detailed Changes

Add a new sub directory horaedb, all source codes for metric engine are under it.

Test Plan

Add a new ci.

ShiKaiWi · 2024-09-10T15:06:27Z

horaedb/metric_engine/src/storage.rs

+
+/// Time-aware merge storage interface.
+#[async_trait]
+pub trait TMStorage {


TM is confusing. How about writing as the full name TimeMergeStorage?

Rachelint · 2024-09-11T06:25:37Z

horaedb/metric_engine/src/storage.rs

+
+/// Time-aware merge storage interface.
+#[async_trait]
+pub trait TimeMergeStorage {


I think may be just define the xxxRequest? And we fill it when we impl it.

Rachelint · 2024-09-11T06:35:17Z

horaedb/metric_engine/src/storage.rs

+        range: TimeRange,
+        predicates: Vec<Predicate>,


Seems time range is a predicate.
And I think maybe we can just process the filter and projection by reusing realted fileds in old ReadRequest?
It seems not sure, may just define a empty ScanRequest in this sketch.

Seems time range is a predicate.

TimeRange is special predicate, it must be included in every request.

And I think maybe we can just process the filter and projection by reusing realted fileds in old ReadRequest?

No code reuse for now.

It seems not sure, may just define a empty ScanRequest in this sketch.

The current one show basic usage, if we leave empty here, other dev have no idea what the query looks like.

horaedb/metric_engine/src/types.rs

horaedb/metric_engine/src/storage.rs

Rachelint · 2024-09-11T06:43:38Z

horaedb/metric_engine/src/manifest.rs

+pub struct Manifest {}
+
+impl Manifest {
+    pub fn new(id: u64) -> Self {


The id means?

UUID, it can be used to construct path for store it.

Rachelint · 2024-09-11T06:49:38Z

horaedb/metric_engine/src/sst.rs

+pub struct SSTable {
+    pub sst_id: u64,
+    pub storage: ObjectStoreRef,
+}


It seems storage should not be part of SSTable? The struct seems better to include the metadata(e.g. id, path)
Maybe we just define a empty struct in this sketch?

OK, I will just leave an id in it.

Rachelint

LGTM

ShiKaiWi · 2024-09-11T09:55:08Z

LGTM

## Rationale See apache#1558 ## Detailed Changes Add a new sub directory `horaedb`, all source codes for metric engine are under it. ## Test Plan Add a new ci.

refactor: partitioned_lock's elaboration (apache#1540) Extended the `try_new` interface while keeping the old one for compatibility. * Implemented the `try_new_suggest_cap` method, while changing the old `try_new` method to `try_new_bit_len` to ensure compatibility. * Modified structs and functions that call old interfaces. * Added new unit tests * Passed CI test --------- Co-authored-by: chunhao.ch <chunhao@antgroup.com> feat: support INSERT INTO SELECT (apache#1536) Close apache#557. When generating the insert logical plan, alse generate the select logical plan and store it in the insert plan. Then execute the select logical plan in the insert interpreter, convert the result records into RowGroup and then insert it. CI refactor: insert select to stream mode (apache#1544) Close apache#1542 Do select and insert procedure in stream way. CI test. --------- Co-authored-by: jiacai2050 <dev@liujiacai.net> fix(comment): update error documentation comment for remote engine service (apache#1548) Updating an error comment in the code to reflect the correct service name is needed. No need refactor: manifest error code (apache#1546) fix: sequence overflow when dropping a table using a message queue as WAL (apache#1550) Fix the issue of sequence overflow when dropping a table using a message queue as WAL. close apache#1543 Check the maximum value of sequence to prevent overflow. CI. feat: Add a new disk-based WAL implementation for standalone deployment (apache#1552) 1. Added a struct `Segment` responsible for reading and writing segment files, and it records the offset of each record. 2. Add a struct SegmentManager responsible for managing all segments, including: 1. Reading all segments from the folder upon creation. 2. Writing only to the segment with the largest ID. 3. Maintaining a cache where segments not in the cache are closed, while segments in the cache have their files open and are memory-mapped using mmap. 3. Implement the `WalManager` trait. Unit tests. chore: upgrade object store version (apache#1541) The object store version is upgraded to 0.10.1 to prepare for access to opendal - Impl AsyncWrite for ObjectStoreMultiUpload - Impl MultipartUpload for ObkvMultiPartUpload - Adapt new api on query writing path - Existing tests --------- Co-authored-by: jiacai2050 <dev@liujiacai.net> feat: use opendal to access underlying storage (apache#1557) Use opendal to access the object store, thus unifying the access method of the underlying storage. - use opendal to access s3/oss/local file - Existed tests feat: add metric engine rfc (apache#1558) RFC for next metric engine. No need. chore: update link (apache#1561) I noticed that the previous repository has been archived, maybe it would be better to update the new link chore(horaemeta): add building docs (apache#1562) feat: Implementing cross-segment read/write for WAL based on local disk (apache#1556) Improving WAL based on local disk. This is a follow-up task for apache#1552. 1. Make MAX_FILE_SIZE configurable. 2. Allocate enough space when creating a segment to avoid remapping when appending to the segment. 3. Add `MultiSegmentLogIterator` to enable cross-segment reading. 4. When writing, if the current segment has insufficient space, create a new segment and write to the new segment. Unit test. chore: fix doc links (apache#1565) fix: disable layered memtable in overwrite mode (apache#1533) Layered memtable is only designed for append mode table now, and it shouldn't be used in overwrite mode table. - Make default values in config used. - Add `enable` field to control layered memtable's on/off. - Add check to prevent invalid options during table create/alter. - Add related it cases. Test manually. Following cases are considered: Check and intercept the invalid table options during table create/alter - enable layered memtable but mutable switch threshold is 0 - enable layered memtable for overwrite mode table Table options new field `layered_enable`'s default value when it is not found in pb - false, when whole `layered_memtable_options` not exist - false, when `layered_memtable_options` exist, and `mutable_segment_switch_threshold` == 0 - true, when `layered_memtable_options` exist, and `mutable_segment_switch_threshold` > 0 feat: init metric engine structure (apache#1554) See apache#1558 Add a new sub directory `horaedb`, all source codes for metric engine are under it. Add a new ci. feat: Implement delete operation for WAL based on local storage (apache#1566) Currently the WAL based on the local disk does not support the delete function. This PR implements that functionality. This is a follow-up task of apache#1552 and apache#1556. 1. For each `Segment`, add a hashmap to record the minimum and maximum sequence numbers of all tables within that segment. During `delete` and `write` operations, this hashmap will be updated. During read operations, logs will be filtered based on this hashmap. 2. During the `delete` operation, based on the aforementioned hashmap, if all logs of all tables in a read-only segment (a segment that is not currently being written to) are marked as deleted, the segment file will be physically deleted from the disk. Unit test, TSBS and running a script locally that repeatedly inserts data, forcibly kills, and restarts the database process to test persistence. fix: support to compat the old layered memtable options (apache#1568) We introduce the explicit flag to control should we enable layered memtable, but it has some compatibility problem when upgrading from old version. This pr add an option to support compating the old layered memtable on/off control method. Add an option to support compating the old layered memtable on/off control method. Manually. chore: record replay cost in log (apache#1569) 1. Add replay cost in log 2. Remove verbose http log 3. Recover default to shard based, which is faster in most wal implementation. fix: logs might be missed during RegionBased replay in the WAL based on local disk (apache#1570) In RegionBased replay, a batch of logs is first scanned from the WAL, and then replayed on various tables using multiple threads. This approach works fine for WALs based on tables, as the logs for each table are clustered together. However, in a WAL based on local disk, the logs for each table may be scattered across different positions within the batch. During multi-threaded replay, it is possible that for a given table, log2 is replayed before log1, resulting in missed logs. 1. Modify `split_log_batch_by_table` function to aggregate all logs for a table together. 2. Modify `tableBatch` struct to change a single range into a `Vec<Range>`. Manual testing. fix format.

github-actions bot added the feature New feature or request label Aug 21, 2024

feat: add metric engine structure

8eb0fd0

jiacai2050 force-pushed the feat-init-me branch from 7274ce8 to 8eb0fd0 Compare September 4, 2024 09:08

jiacai2050 added 2 commits September 9, 2024 17:38

add sst module

84156fe

fix comments

3380b89

jiacai2050 force-pushed the feat-init-me branch from b606aa4 to 3f46e6c Compare September 10, 2024 13:30

refactor, add TMStorage

c40d3d8

jiacai2050 force-pushed the feat-init-me branch 2 times, most recently from 729c11c to fdf93d4 Compare September 10, 2024 13:44

add Makefile

4aa0602

jiacai2050 force-pushed the feat-init-me branch from fdf93d4 to 4aa0602 Compare September 10, 2024 13:50

jiacai2050 marked this pull request as ready for review September 10, 2024 13:52

jiacai2050 requested review from ShiKaiWi, chunshao90, tisonkun and Rachelint September 10, 2024 13:55

add manifest

710c087

jiacai2050 force-pushed the feat-init-me branch from af69a70 to 710c087 Compare September 10, 2024 14:09

ShiKaiWi reviewed Sep 10, 2024

View reviewed changes

add record batch stream

f5a0a67

Rachelint reviewed Sep 11, 2024

View reviewed changes

fix CR

9ae567f

Rachelint previously approved these changes Sep 11, 2024

View reviewed changes

jiacai2050 dismissed Rachelint’s stale review via cba26dc September 11, 2024 09:44

replace custom predicate with datafusion expr

e665307

jiacai2050 force-pushed the feat-init-me branch from cba26dc to e665307 Compare September 11, 2024 09:50

ShiKaiWi approved these changes Sep 11, 2024

View reviewed changes

jiacai2050 merged commit 7ccf97e into apache:main Sep 11, 2024
6 checks passed

jiacai2050 deleted the feat-init-me branch September 11, 2024 10:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: init metric engine structure #1554

feat: init metric engine structure #1554

jiacai2050 commented Aug 21, 2024 •

edited

Loading

ShiKaiWi Sep 10, 2024

jiacai2050 Sep 11, 2024

Rachelint Sep 11, 2024

jiacai2050 Sep 11, 2024

Rachelint Sep 11, 2024

jiacai2050 Sep 11, 2024

Rachelint Sep 11, 2024

jiacai2050 Sep 11, 2024

Rachelint Sep 11, 2024

jiacai2050 Sep 11, 2024

Rachelint left a comment

ShiKaiWi commented Sep 11, 2024

feat: init metric engine structure #1554

feat: init metric engine structure #1554

Conversation

jiacai2050 commented Aug 21, 2024 • edited Loading

Rationale

Detailed Changes

Test Plan

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rachelint left a comment

Choose a reason for hiding this comment

ShiKaiWi commented Sep 11, 2024

jiacai2050 commented Aug 21, 2024 •

edited

Loading