[Optimize][Delete] Simplify the delete process to make it fast #3191

xy720 · 2020-03-25T03:38:37Z

#3190
Our current DELETE strategy reuses the LoadChecker framework. Loadchecker runs jobs in different stages by polling them in every 5 seconds.

There are four stages of a load job, Pending/ETL/Loading/Quorum_finish, each of them is allocated to a LoadChecker. Four example, if a load job is submitted, it will be initialized to the pending state, then wait for running by the Pending LoadChecker. After the pending job is runned, its stage will change to ETL stage, and then wait for running by the next LoadChecker(ETL). Because interval time of the LoadChecker is 5s, in worst case, a pending job need to wait for 20s during its life cycle.

In particular, the DELETE jobs do not need to wait for polling, they can run the pushTask() function directly to delete. In this commit, I add a delete handler to concurrently processing delete tasks. All delete tasks will push to BE immediately, not required to wait for LoadCheker, without waiting for 2 LoadChecker(delete job started in LOADING state), at most 10s will be save(5s per LoadCheker). The delete process now is synchronized and users get response only after the delete finished or be canceled. If a delete is running over a certain period of time, it will be cancelled with a timeout exception.

当前Doris的删除策略是走的LoadChecker路线，Delete Job 被视为一般的Load Job，而一般的Load Job需要经过一次LoadChecker的轮询才能改变自己的状态。

Load Job的状态变化顺序为PENDING->ETL->LOADING->QUORUM_FINISHED，因为LoadCheker的轮询间隔为5秒，因此最坏情况下Load Job需要等待20s时间。

但是Delete Job一初始化即为LOADING状态，甚至它根本不需要状态，直接提交Push Task后等待就可以了，因此这次提交旨在将Delete Job从LoadChecker的系统中分离出来，节省不需要浪费的时间。

最好情况下，让Delete Job直接push task，可以节省5～10s时间，在提交任务后，根据此次提交涉及的tablet数量指定等待超时时间，这样同步删除的方式使一个Delete Job被取消或者完成时用户能够立刻收到消息，最后我们只要持久化已完成的Delete Job，供用户show delete命令查找即可。

kangpinghuang · 2020-03-26T02:48:43Z

could you add some performance optimization result description?

fe/src/main/java/org/apache/doris/load/DeleteHandler.java

morningman · 2020-04-11T13:43:30Z

fe/src/main/java/org/apache/doris/load/DeleteHandler.java

+                    partition.getId(), partitionName,
+                    -1, 0, deleteConditions);
+            deleteJob = new DeleteJob(transactionId, deleteInfo);
+            idToDeleteJob.put(deleteJob.getTransactionId(), deleteJob);


If you put deleteInfo in idToDeleteJob, you need to make sure that the deleteInfo will be cleaned finally, even if any exception is thrown.
So I think you should clear the deleteInfo in finally block.

added a new try finally block surround outside this logic

fe/src/main/java/org/apache/doris/task/DeleteJob.java

fe/src/main/java/org/apache/doris/load/DeleteHandler.java

morningman · 2020-04-14T12:03:18Z

fe/src/main/java/org/apache/doris/persist/EditLog.java

                    DeleteInfo info = (DeleteInfo) journal.getData();
-                    Load load = catalog.getLoadInstance();
-                    load.replayDelete(info, catalog);
+                    DeleteHandler deleteHandler = catalog.getDeleteHandler();


You can not use the origin OP_FINISH_SYNC_DELETE, because they are different operations.

Now add a new operation OP_FINISH_DELETE

morningman · 2020-04-14T12:04:15Z

fe/src/main/java/org/apache/doris/qe/ShowExecutor.java


-        Load load = catalog.getLoadInstance();
-        List<List<Comparable>> deleteInfos = load.getDeleteInfosByDb(dbId, true);
+        DeleteHandler deleteHandler = catalog.getDeleteHandler();


You should also show the delete info from Load, or after we upgrade the Doris, the old delete info can not be seen.

fe/src/main/java/org/apache/doris/task/DeleteJob.java

morningman · 2020-04-14T12:08:10Z

fe/src/main/java/org/apache/doris/task/DeleteJob.java

+    }
+
+    public boolean addFinishedReplica(long tabletId, Replica replica) {
+        TabletDeleteInfo tDeleteInfo = tabletDeleteInfoMap.get(tabletId);


After changing tabletDeleteInfoMap to ConncurrentHashMap, you should use putIfAbsent to perform the atomic operation here.

morningman · 2020-04-14T12:37:35Z

fe/src/main/java/org/apache/doris/load/DeleteHandler.java

+        }
+    }
+
+    public boolean cancelJob(DeleteJob job, CancelType cancelType, String reason) {


This method return boolean, but you never use it.
I think return true means cancel succeed(txn failed), and return false means cancel failed(txn succeed).
And the caller should use this return value to decide whether to return user success or failure.

And if transaction is COMMITTED but not VISIBLE, you should return a transaction id to user, so that user can use that it to check transaction's state.

fe/src/main/java/org/apache/doris/load/DeleteHandler.java

morningman · 2020-04-15T13:52:38Z

fe/src/main/java/org/apache/doris/qe/StmtExecutor.java

            DdlExecutor.execute(context.getCatalog(), (DdlStmt) parsedStmt, originStmt);
            context.getState().setOk();
+        } catch (QueryStateException e) {
+            context.getState().setOk(0L, 0, e.getMessage());


QueryStateException should derived from UserException.

Better to just create a QueryState inside the QueryStateException, and here you can just call context.setState(e.getQueryState());. If other people use this exception, he will know how to use it.

morningman · 2020-04-15T14:01:39Z

fe/src/main/java/org/apache/doris/task/DeleteJob.java

+    public boolean addFinishedReplica(long tabletId, Replica replica) {
+        tabletDeleteInfoMap.putIfAbsent(tabletId, new TabletDeleteInfo(tabletId));
+        TabletDeleteInfo tDeleteInfo =  tabletDeleteInfoMap.get(tabletId);
+        synchronized (tDeleteInfo) {


No need to use synchronized here, I think you can just use a ConcurrentSet in TabletDeleteInfo

morningman

LGTM

…e#3191) Our current DELETE strategy reuses the LoadChecker framework. LoadChecker runs jobs in different stages by polling them in every 5 seconds. There are four stages of a load job, Pending/ETL/Loading/Quorum_finish, each of them is allocated to a LoadChecker. Four example, if a load job is submitted, it will be initialized to the Pending state, then wait for running by the Pending LoadChecker. After the pending job is ran, its stage will change to ETL stage, and then wait for running by the next LoadChecker(ETL). Because interval time of the LoadChecker is 5s, in worst case, a pending job need to wait for 20s during its life cycle. In particular, the DELETE jobs do not need to wait for polling, they can run the pushTask() function directly to delete. In this commit, I add a delete handler to concurrently processing delete tasks. All delete tasks will push to BE immediately, not required to wait for LoadCheker, without waiting for 2 LoadChecker(delete job started in LOADING state), at most 10s will be save(5s per LoadCheker). The delete process now is synchronized and users get response only after the delete finished or be canceled. If a delete is running over a certain period of time, it will be cancelled with a timeout exception. NOTICE: this CL upgrade FE meta version to 82

xy720 force-pushed the new_branch branch from b5dc0c3 to 7139a38 Compare April 7, 2020 11:59

morningman reviewed Apr 8, 2020

View reviewed changes

morningman requested changes Apr 11, 2020

View reviewed changes

morningman requested changes Apr 14, 2020

View reviewed changes

morningman changed the title ~~DeleteV2~~ [Optimize][Delete] Simplify the delete process to make it fast Apr 14, 2020

xy720 added 14 commits April 15, 2020 12:35

commit 1: deleteV2

e9910f3

commit 2: delete handler

000982e

commit 3: delete handler

1bc3679

commit 3: delete handler

739a0bb

commit 4

927aadd

commit 5

9530aae

commit 6

933a452

commit 7

f442fdb

commit 8

52bc87a

commit 9

2f79ea5

commit 10

0654799

commit 11

1571157

commit 12

ef29b1a

commit 13

5f9cb8d

xy720 force-pushed the new_branch branch from f7da159 to 5f9cb8d Compare April 15, 2020 10:49

xy720 added 2 commits April 15, 2020 21:02

commit 14

bd9f6df

commit 15

cf027fe

morningman reviewed Apr 15, 2020

View reviewed changes

commit 16

9a068e0

morningman approved these changes Apr 15, 2020

View reviewed changes

morningman merged commit b29cb9d into apache:master Apr 16, 2020

EmmyMiao87 mentioned this pull request Aug 17, 2020

Release Notes 0.13.0 #4370

Closed

[Optimize][Delete] Simplify the delete process to make it fast #3191

[Optimize][Delete] Simplify the delete process to make it fast #3191

Uh oh!

Conversation

xy720 commented Mar 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kangpinghuang commented Mar 26, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xy720 commented Mar 25, 2020 •

edited

Loading