Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test #339

Closed
wants to merge 68 commits into from
Closed

test #339

Changes from 1 commit
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
9992a5e
Import LevelDB
flowbehappy Mar 6, 2019
783e07b
DeltaMerge Engine POC
flowbehappy Jan 30, 2019
ea34ed2
add delta merge engine test (#18)
lidezhu Apr 4, 2019
87bca17
Add junction for concurrent hash map
flowbehappy Apr 3, 2019
13e1020
DeltaMerge Storage Engine implementation
flowbehappy Jun 14, 2019
e8e6e13
done DM integration with query layer
flowbehappy May 24, 2019
c86350d
Auto gc dm data
flowbehappy May 24, 2019
6d59553
DiskValueSpace: read page by iterator
flowbehappy Jun 14, 2019
4e4f3e1
parallelize segment input stream creation
flowbehappy May 29, 2019
58fcd61
Using DB::DM namespace
flowbehappy Jun 4, 2019
17c2ced
Bug fix: remove INVALID
flowbehappy Jun 6, 2019
ff39a6c
Log operations and add status
flowbehappy Jun 6, 2019
968b6dd
bug fix: placed_delta_rows -> placed_delta_deletes
flowbehappy Jun 6, 2019
bd64b70
Add check to DeltaMerge Storage
flowbehappy Jun 10, 2019
4ef6073
Remove useless code
flowbehappy Jun 10, 2019
60b9d60
Remove pk_columns cache in delta, and bug fix
flowbehappy Jun 11, 2019
4493d95
Bug fix: out of bound exception in DiskValueSpace#read
flowbehappy Jun 11, 2019
678a47e
Remove useless commented code
flowbehappy Jun 12, 2019
466acd6
[FLASH-290] Test cases for DeltaMerge and Gen code coverage report (#82)
JaySon-Huang Jun 14, 2019
abc84d8
Page storage bug fix (#87)
JaySon-Huang Jun 24, 2019
83f505b
[FLASH-318/319/343/344] PageStorage: RefPages, MVCC && Snapshot read …
JaySon-Huang Aug 2, 2019
0e1de61
[DM] Support range read and lock optimization (#140)
flowbehappy Aug 5, 2019
277641a
[FLASH-406] CI run gtest cases (#163)
JaySon-Huang Aug 8, 2019
c8f0112
fix gtest compile error
JaySon-Huang Aug 5, 2019
8a2ccc6
fix bug: PageEntriesVersionSetWithDelta::compactDeltaAndBase should N…
JaySon-Huang Aug 14, 2019
d0f3495
temporary disable some tests for DM (#184)
JaySon-Huang Aug 16, 2019
221387a
Small fix for gtests after rebase to master
JaySon-Huang Aug 26, 2019
ac4165f
Atomic read/write on DeltaMergeStore level (#172)
flowbehappy Aug 27, 2019
0ff12ed
[FLASH-386] DeltaMerge DDL support (#190)
JaySon-Huang Aug 28, 2019
e60b3d3
[FLASH-455] Remove level db dependency (#206)
marsishandsome Aug 28, 2019
8f2a5f5
[FLASH-466] improve code coverage (#219)
marsishandsome Sep 6, 2019
fa5bc51
DM: support rough index in Storage level. (#233)
flowbehappy Sep 12, 2019
a36a2b3
[FLASH-461/471] Basic read/write StorageDeltaMerge's table through Ra…
JaySon-Huang Sep 12, 2019
60e6856
fix compile error
flowbehappy Sep 16, 2019
5e82056
[FLASH-469/452/462] DM: Rename column, read with specify tso, schema-…
JaySon-Huang Sep 18, 2019
0d6d19a
[FLASH-409] Fix page not found exception (#238)
flowbehappy Sep 19, 2019
f0f9197
fix bug: RefPage of PageEntriesVersionSetWithDelta (#246)
JaySon-Huang Sep 24, 2019
d5ceccf
[FLASH-470/516] DM learner read and check gc safe point (#249)
JaySon-Huang Sep 26, 2019
f2875a5
[FLASH-426] Eliminate data copy of Segment split/merge (#258)
flowbehappy Sep 27, 2019
7d6c7d3
Binary compatibility version for DeltaMerge's chunk; Fix disappear of…
JaySon-Huang Sep 27, 2019
1007394
[FLASH-528] Fix PageStorage error: Directory not empty (#263)
JaySon-Huang Sep 30, 2019
98d3d1b
[FLASH-541] Read ranges indicated by mvcc_query_info (#267)
JaySon-Huang Oct 11, 2019
43d0037
[FLASH-546] Fix exception after reload PageStorage from disk (#274)
JaySon-Huang Oct 12, 2019
a0773cd
[FLASH-296] [DM] Use Background Threads To Run Delta Merge Operations…
flowbehappy Oct 19, 2019
b5df908
Bug fix: placeUpsert offset is not right. Also copy the cache when me…
flowbehappy Oct 21, 2019
553b8dd
Bug fix: Segment#split could use out of range split point.
flowbehappy Oct 21, 2019
435ed78
[FLASH-544]Add test cases for DM (#262)
leiysky Oct 22, 2019
7879cfe
[FLASH-548/579] Engine DeltaMerge coprocessor integration (#287)
JaySon-Huang Oct 23, 2019
73f1fe7
fix compile error after rebase
JaySon-Huang Oct 23, 2019
5d9798e
[FLASH-453]Support default value of CAST expression (#286)
leiysky Oct 29, 2019
c5d3482
[FLASH-575] Improve the performance of stream operations in DM (#296)
flowbehappy Oct 30, 2019
680c5ff
[Flash-600] PageStorage GC only remove obsoleted data (#297)
JaySon-Huang Oct 30, 2019
cd50d94
fix bug: init of variables
JaySon-Huang Oct 30, 2019
86489a4
[FLASH-629] Add concurrency for wait index while doing learner read (…
JaySon-Huang Nov 12, 2019
eb741e8
[FLASH-549/550/626] Remove obsoleted data in Engine DeltaMerge (#292)
JaySon-Huang Nov 12, 2019
99008ed
[FLASH-463]Flush regions into DM as soon as raft commands processed (…
leiysky Nov 12, 2019
30fd832
[FLASH-654] Add gc callback for PageStorage (#319)
JaySon-Huang Nov 12, 2019
7bc6612
[FLASH-660] Check region version while doing learner read (#321)
JaySon-Huang Nov 20, 2019
514801c
Enhance PageStorage gc (#325)
JaySon-Huang Nov 20, 2019
58283b5
Fix non-global-context exception while running unit test
JaySon-Huang Nov 22, 2019
d64370b
[FLASH-617] Make cast when applying DDL (#328)
leiysky Nov 25, 2019
48a2cd6
[FLASH-668/477] Predicator push down (#329)
JaySon-Huang Nov 26, 2019
fac6098
add ReorganizeBlockInputStream for reorganize the boundary of blocks …
JaySon-Huang Nov 27, 2019
4da2f8a
[FLASH-622] DMFile implementation (#327)
flowbehappy Nov 27, 2019
55e71a9
Add DMCleanReadRows event and adjust log level
flowbehappy Nov 28, 2019
9838315
Try do range filter in DeltaValueSpace
flowbehappy Nov 28, 2019
c53eb14
[FLASH-736] Illegal delta data, the next row is expected larger than …
flowbehappy Nov 29, 2019
1076984
Enable rough set index in DMFile when data type is not Handle
flowbehappy Nov 30, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Auto gc dm data
flowbehappy committed Dec 3, 2019
commit c86350dc24a0ec9d307bccc5efd77ba1eccb4562
17 changes: 15 additions & 2 deletions dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp
Original file line number Diff line number Diff line change
@@ -9,6 +9,7 @@
#include <Storages/DeltaMerge/DeltaMergeHelpers.h>
#include <Storages/DeltaMerge/DeltaMergeStore.h>
#include <Storages/DeltaMerge/SegmentReadTaskPool.h>

namespace DB
{

@@ -17,13 +18,18 @@ namespace ErrorCodes
extern const int LOGICAL_ERROR;
} // namespace ErrorCodes

DeltaMergeStore::DeltaMergeStore(const Context & db_context,
DeltaMergeStore::DeltaMergeStore(Context & db_context,
const String & path_,
const String & name,
const ColumnDefines & columns,
const ColumnDefine & handle,
const Settings & settings_)
: path(path_), storage_pool(path), table_name(name), table_handle_define(handle), settings(settings_)
: path(path_),
storage_pool(path),
table_name(name),
table_handle_define(handle),
background_pool(db_context.getBackgroundPool()),
settings(settings_)
{
// We use Int64 to store handle.
if (!table_handle_define.type->equals(*EXTRA_HANDLE_COLUMN_TYPE))
@@ -62,6 +68,13 @@ DeltaMergeStore::DeltaMergeStore(const Context & db_context,
segment_id = segment->nextSegmentId();
}
}

gc_handle = background_pool.addTask([this] { return storage_pool.gc(); });
}

DeltaMergeStore::~DeltaMergeStore()
{
background_pool.removeTask(gc_handle);
}

void DeltaMergeStore::write(const Context & db_context, const DB::Settings & db_settings, const Block & to_write)
8 changes: 6 additions & 2 deletions dbms/src/Storages/DeltaMerge/DeltaMergeStore.h
Original file line number Diff line number Diff line change
@@ -5,6 +5,7 @@
#include <Storages/DeltaMerge/DMContext.h>
#include <Storages/DeltaMerge/Segment.h>
#include <Storages/DeltaMerge/StoragePool.h>
#include <Storages/MergeTree/BackgroundProcessingPool.h>
#include <Storages/Page/PageStorage.h>

namespace DB
@@ -33,12 +34,13 @@ class DeltaMergeStore
// size_t segment_delta_cache_limit_bytes = 16 * MB;
};

DeltaMergeStore(const Context & db_context, //
DeltaMergeStore(Context & db_context, //
const String & path_,
const String & name,
const ColumnDefines & columns,
const ColumnDefine & handle,
const Settings & settings_);
~DeltaMergeStore();

void write(const Context & db_context, const DB::Settings & db_settings, const Block & block);

@@ -86,7 +88,9 @@ class DeltaMergeStore
ColumnDefines table_columns;
ColumnDefine table_handle_define;

DataTypePtr table_handle_original_type;
DataTypePtr table_handle_original_type;
BackgroundProcessingPool & background_pool;
BackgroundProcessingPool::TaskHandle gc_handle;

Settings settings;

16 changes: 16 additions & 0 deletions dbms/src/Storages/DeltaMerge/StoragePool.cpp
Original file line number Diff line number Diff line change
@@ -17,5 +17,21 @@ StoragePool::StoragePool(const String & path)
max_meta_page_id = get_max_page_id(meta_storage);
}

bool StoragePool::gc(const Seconds try_gc_period)
{
std::lock_guard<std::mutex> lock(mutex);

Timepoint now = Clock::now();
if (now < (last_try_gc_time.load() + try_gc_period))
return false;
last_try_gc_time = now;

bool ok = false;

ok |= meta_storage.gc();
ok |= data_storage.gc();
ok |= log_storage.gc();

return ok;
}
} // namespace DB
15 changes: 15 additions & 0 deletions dbms/src/Storages/DeltaMerge/StoragePool.h
Original file line number Diff line number Diff line change
@@ -1,14 +1,23 @@
#pragma once

#include <atomic>
#include <chrono>

#include <Storages/Page/PageStorage.h>

namespace DB
{

static const std::chrono::seconds DELTA_MERGE_GC_PERIOD(60);

class StoragePool
{
public:
using Clock = std::chrono::system_clock;
using Timepoint = Clock::time_point;
using Duration = Clock::duration;
using Seconds = std::chrono::seconds;

StoragePool(const String & path);

PageId maxLogPageId() { return max_log_page_id; }
@@ -23,6 +32,8 @@ class StoragePool
PageStorage & data() { return data_storage; }
PageStorage & meta() { return meta_storage; }

bool gc(const Seconds try_gc_period = DELTA_MERGE_GC_PERIOD);

private:
std::atomic<PageId> max_log_page_id;
std::atomic<PageId> max_data_page_id;
@@ -31,5 +42,9 @@ class StoragePool
PageStorage log_storage;
PageStorage data_storage;
PageStorage meta_storage;

std::atomic<Timepoint> last_try_gc_time = Clock::now();

std::mutex mutex;
};
} // namespace DB