Grouped execution support for JOINs with Hive connector #8951

haozhun · 2017-09-11T20:10:16Z

With the right table organization (e.g. bucketing in Hive), it is possible to process a subset of data at a time for JOINs. This reduces the amount of memory needed to hold the hash table.

haozhun · 2017-09-22T23:41:01Z

rebased, @martint

First 20 commits (all commits up to "Make SqlTaskExecution work with LocalExecutionPlan instead of Fragment") should be pretty easy to review.

The 21st commit ("Make SqlTaskExecution work with LocalExecutionPlan instead of Fragment") is a complex one. But it's well tested. Let me know whenever you have any questions.

martint

Done with these commits. Had some comments, especially on the one about refactoring TestBufferingSplitSource:

Remove unused field from HivePartitionKey
Add initialCount parameter to ReferenceCount constructor
Mark LookupJoinOperatorFactory as UsedByGeneratedCode
Improve TestMetadataManager to avoid stuck test
Make TestBackgroundSplitLoader not use DirectExecutor
Refactor TestBufferingSplitSource
Add javadoc for guarantee provided by SourcePartitionedScheduler
Add javadoc to clarify ConnectorSplitSource.isFinished requirement

martint · 2017-09-25T19:02:16Z

presto-tests/src/test/java/com/facebook/presto/tests/TestMetadataManager.java

@@ -86,7 +88,12 @@ public void testMetadataIsClearedAfterQueryCanceled()

        // wait until query starts running
        while (true) {
-            if (queryManager.getQueryInfo(queryId).getState() == RUNNING) {


Add comment to commit message explaining why it was getting stuck.

martint · 2017-09-25T19:03:35Z

presto-hive/src/test/java/com/facebook/presto/hive/TestBackgroundSplitLoader.java

@@ -69,7 +70,7 @@
    private static final Path RETURNED_PATH = new Path(SAMPLE_PATH);
    private static final Path FILTERED_PATH = new Path(SAMPLE_PATH_FILTERED);

-    private static final Executor EXECUTOR = directExecutor();


Commit message needs explanation of motivation to not use direct executor.

martint · 2017-09-25T19:04:32Z

presto-hive/src/test/java/com/facebook/presto/hive/TestBackgroundSplitLoader.java

-        List<ConnectorSplit> connectorSplits = hiveSplitSource.getNextBatch(1).get();
-        assertEquals(1, connectorSplits.size());
-        assertEquals(RETURNED_PATH.toString(), ((HiveSplit) connectorSplits.get(0)).getPath());
+    private List<String> drainHiveSplitSource(HiveSplitSource hiveSplitSource)


Argument could just be named source for simplicity.

And the method could be named drain, too

martint · 2017-09-25T19:06:51Z

presto-hive/src/test/java/com/facebook/presto/hive/TestBackgroundSplitLoader.java

+        ImmutableList.Builder<String> paths = ImmutableList.builder();
+        while (true) {
+            List<ConnectorSplit> splits = hiveSplitSource.getNextBatch(100).get();
+            for (ConnectorSplit connectorSplit : splits) {


Variable could be named split for simplicity.

martint · 2017-09-25T19:10:02Z

presto-main/src/test/java/com/facebook/presto/split/MockSplitSource.java


 public class MockSplitSource
        implements SplitSource
 {
    private static final Split SPLIT = new Split(new ConnectorId("test"), new ConnectorTransactionHandle() {}, new MockConnectorSplit());
+    private static final SettableFuture<List<Split>> COMPLETED_FUTURE = SettableFuture.create();


Futures.immediateFuture(null) ?

BTW, this can, technically, result in a memory leak. I went looking at Guava's docs and was surprised to see there's no explicit statement of whether a ListenableFuture (and, in particular, SettableFuture) is expected to not hold on to the reference to the listener once completed.

Futures.immediateFuture cannot be used because it returns a ListenableFuture, not a SettableFuture. We discussed this in person.

martint · 2017-09-25T19:54:33Z

presto-main/src/test/java/com/facebook/presto/split/MockSplitSource.java

+
+    private void doGetNextBatch()
+    {
+        if (splitsProduced >= totalSplits) {


splitsProduce can never be > than totalSplits, right? I'd change the check to == and add a checkState to help catch any logic errors that could cause it to go over.

martint · 2017-09-25T20:02:23Z

presto-main/src/test/java/com/facebook/presto/split/TestBufferingSplitSource.java

            throws Exception
    {
-        MockSplitSource mockSource = new MockSplitSource(1, 25);
+        MockSplitSource mockSource = new MockSplitSource().setBatchSize(1).increaseAvailableSplits(25).atSplitCompletion(FINISH);


Place each .setXXX on a separate line. It makes this easier to read:

MockSplitSource mockSource = new MockSplitSource() .setBatchSize(1) .increaseAvailableSplits(25) .atSplitCompletion(FINISH);

martint · 2017-09-25T20:09:52Z

presto-main/src/test/java/com/facebook/presto/split/TestBufferingSplitSource.java

+    {
+        ListenableFuture<?> result;
+        switch (nextBatchCall) {
+            case SINGLE_ARGUMENT:


What's this? It doesn't seem to serve any purpose in this PR, so either remove it or move it to the PR that's going to need it.

We talked about this offline. I'm avoiding structural changes to this file in the future commits so that those will be easier to review. As you have realized, reviewing this commit is quite involving despite it made little material change.

martint · 2017-09-25T20:12:30Z

presto-main/src/test/java/com/facebook/presto/split/TestBufferingSplitSource.java

+        }
+    }
+
+    public void testDriverGroups()


martint · 2017-09-25T20:14:23Z

...o-main/src/main/java/com/facebook/presto/execution/scheduler/SourcePartitionedScheduler.java

+     * In the event that no ordinary split is available from the underlying SplitSource,
+     * a synthesized EmptySplit will be scheduled.
+     * This at-least-one-split guarantee is provided on a per-SplitSource basis,
+     * not per-DriverGroup basis.


I don't think there's a "DriverGroup" concept at this point, so defer this part of the javadoc until it's introduced.

I'll move this commit (Add javadoc for guarantee provided by SourcePartitionedScheduler) after Support addressable split group in SplitSource/Manager

martint

Done with a few more:

Rename TableLayout.PartitioningColumn to StreamPartitioningColumn
Rename HiveSplitSource.finished to noMoreSplits
Rename OperatorFactory.close to noMoreOperators
Rename DriverFactory.close to noMoreOperators
Document guarantee on invocation of OperatorFactory.noMoreOperators
Add SplitManager to PlanFragmenter
Rename NodePartitioning to TablePartitioning
Add parameter to ConnectorSplitManager.getSplits

martint · 2017-09-25T20:25:47Z

presto-main/src/main/java/com/facebook/presto/metadata/TableLayout.java

@@ -80,7 +80,7 @@ public TableLayoutHandle getHandle()
                        nodePartitioning.getPartitioningColumns()));
    }

-    public Optional<Set<ColumnHandle>> getPartitioningColumns()


I'd mention, in the commit message, that this should've probably been renamed when stream properties were added and this is just correcting that oversight.

I suppose you meant to say "when node properties were added"

martint · 2017-09-25T20:27:32Z

presto-hive/src/main/java/com/facebook/presto/hive/BackgroundHiveSplitLoader.java

@@ -216,9 +216,9 @@ private void invokeFinishedIfNecessary()
            try {


Explain motivation in commit message. I.e., what's wrong with "finished"?

martint · 2017-09-25T20:30:17Z

presto-benchmark/src/main/java/com/facebook/presto/benchmark/HashBuildAndJoinBenchmark.java

@@ -81,7 +81,7 @@ from lineitem join orders using (orderkey)
        driversBuilder.add(hashBuilder);
        DriverFactory hashBuildDriverFactory = new DriverFactory(0, true, false, driversBuilder.build(), OptionalInt.empty());
        Driver hashBuildDriver = hashBuildDriverFactory.createDriver(taskContext.addPipelineContext(0, true, false).addDriverContext());
-        hashBuildDriverFactory.close();
+        hashBuildDriverFactory.noMoreDriver();


noMoreDrivers?

Also, commit message incorrectly refers to noMoreOperators

martint · 2017-09-25T20:30:51Z

presto-main/src/main/java/com/facebook/presto/operator/DriverFactory.java

@@ -27,7 +26,6 @@
 import static java.util.Objects.requireNonNull;

 public class DriverFactory
-        implements Closeable


Did you mean to put this in the previous commit?

martint · 2017-09-25T20:31:26Z

presto-main/src/main/java/com/facebook/presto/operator/DriverFactory.java

    public synchronized void noMoreDriver()
    {
-        if (!closed) {


Unrelated change?

I moved it to Rename DriverFactory.close to noMoreDrivers. It's still an unrelated change. But this file is touched in that commit. And I don't think something this minor need a stand-alone commit.

martint · 2017-09-25T20:33:39Z

presto-main/src/main/java/com/facebook/presto/sql/planner/PlanFragmenter.java

@@ -65,7 +66,7 @@ private PlanFragmenter()
    {
    }

-    public static SubPlan createSubPlans(Session session, Metadata metadata, Plan plan)
+    public static SubPlan createSubPlans(Session session, Metadata metadata, SplitManager splitManager, Plan plan)


Why? This change warrants an explanation of the motivation, especially since it's being added without anything using it.

martint · 2017-09-25T20:43:41Z

presto-accumulo/src/main/java/com/facebook/presto/accumulo/AccumuloSplitManager.java

@@ -57,7 +57,7 @@ public AccumuloSplitManager(
    }


I'd change the title of the commit message to "Pass split scheduling strategy to ConnectorSplitManager.getSplits"

martint · 2017-09-25T20:46:56Z

presto-accumulo/src/main/java/com/facebook/presto/accumulo/AccumuloSplitManager.java

@@ -57,7 +57,7 @@ public AccumuloSplitManager(
    }

    @Override
-    public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHandle, ConnectorSession session, ConnectorTableLayoutHandle layout)


I'm wondering whether this should be encoded in the layout, but I can't say without seeing how, exactly, this is derived. My intuition is that this is property of the plan shape, therefore, determined during planning/optimization (vs a runtime scheduling knob)

martint · 2017-09-25T20:48:19Z

presto-spi/src/main/java/com/facebook/presto/spi/connector/ConnectorSplitManager.java

+    enum SplitSchedulingStrategy
+    {
+        ALL_AT_ONCE,
+        GROUPED,


GROUPED notion doesn't exist at this point, so it should be deferred to the commit that adds it.

martint · 2017-09-26T00:13:42Z

presto-main/src/main/java/com/facebook/presto/execution/DriverGroupId.java

+
+import static com.google.common.base.Preconditions.checkState;
+
+@JsonSerialize(using = DriverGroupId.Serializer.class)


Why do we need custom serializer/deserializer? (I haven't looked deeply at what it does)

We talked about this offline. You can't do @JsonValue Integer because JsonCreator won't be invoked for null value.

martint · 2017-09-26T19:33:31Z

presto-main/src/main/java/com/facebook/presto/split/BufferingSplitSource.java

@@ -85,4 +101,28 @@ public boolean isFinished()
    {
        return source.isFinished();
    }
+
+    private static class FetchSplitsResult


How is this part of the change related to making groups addressable?

Here, BufferingSplitSource is implementing the new method in SplitSource that takes SplitGroupId.

martint · 2017-09-26T19:35:33Z

presto-main/src/main/java/com/facebook/presto/execution/DriverGroupId.java

+@JsonDeserialize(using = DriverGroupId.Deserializer.class)
+public class DriverGroupId
+{
+    private final boolean grouped;


What's an ungrouped group?

martint · 2017-09-26T19:38:05Z

presto-main/src/main/java/com/facebook/presto/split/ConnectorAwareSplitSource.java

+    public ListenableFuture<SplitBatch> getNextBatch(DriverGroupId driverGroupId, int maxSize)
+    {
+        checkState(driverGroupId.isGrouped() == (splitSchedulingStrategy == SplitSchedulingStrategy.GROUPED));
+        ListenableFuture<ConnectorSplitBatch> nextBatch = toListenableFuture(


This is hard to read. Introduce a variable for the conditional expression.

martint · 2017-09-26T19:40:42Z

presto-main/src/main/java/com/facebook/presto/split/SplitSource.java

+    class SplitBatch
+    {
+        private final List<Split> splits;
+        private final boolean noMoreSplits;


Maybe call it lastBatch. It will make isNoMoreSplits sound better: isLastBatch

martint · 2017-09-26T20:11:38Z

presto-main/src/main/java/com/facebook/presto/operator/Driver.java

-        if (!sourceOperator.isPresent() || !sourceOperator.get().getSourceId().equals(sourceUpdate.getPlanNodeId())) {
-            return;
-        }
+        checkArgument(sourceOperator.isPresent() && sourceOperator.get().getSourceId().equals(sourceUpdate.getPlanNodeId()));


Add a message to the checkArgument call.

martint · 2017-09-26T22:51:44Z

presto-main/src/main/java/com/facebook/presto/execution/SqlTaskExecution.java

@@ -118,16 +118,52 @@ public static SqlTaskExecution createSqlTaskExecution(
            Executor notificationExecutor,
            QueryMonitor queryMonitor)
    {
+        LocalExecutionPlan localExecutionPlan;


I'd get rid of this constructor and have the caller (SqlTaskExecutionFactory?) convert {planner,fragment} -> local execution plan, since planner and fragment are not used by this class anymore.

martint

Some comments I had pending. They may not be relevant after our face-to-face discussion last week, but I didn't want them to get lost.

martint · 2017-09-27T21:32:36Z

presto-main/src/main/java/com/facebook/presto/TaskSource.java

        this.noMoreSplits = noMoreSplits;
    }

+    public TaskSource(
+            PlanNodeId planNodeId,


Arguments can all go on the same line

martint · 2017-09-27T21:36:58Z

presto-main/src/main/java/com/facebook/presto/TaskSource.java

@@ -28,19 +29,30 @@
 {
    private final PlanNodeId planNodeId;
    private final Set<ScheduledSplit> splits;
+    private final Set<DriverGroupId> noMoreSplitsForDriverGroup;


The name of this field is misleading given its type. Maybe call it groupsWithNoMoreSplits or groupsWithoutMoreSplits?

martint · 2017-09-28T00:27:22Z

presto-main/src/main/java/com/facebook/presto/execution/SqlTaskExecution.java

    @GuardedBy("this")
-    private final ConcurrentMap<PlanNodeId, TaskSource> pendingSplits = new ConcurrentHashMap<>();
+    private final Map<PlanNodeId, SplitsForPlanNode> pendingSplitsMap;


pendingSplitsMap is not a good name. Maybe call this pendingSplitsPerSource

martint · 2017-09-28T00:32:40Z

presto-main/src/main/java/com/facebook/presto/sql/planner/LocalExecutionPlanner.java

            PlanNode plan,
            List<Symbol> outputLayout,
            Map<Symbol, Type> types,
+            List<PlanNodeId> partitionedSourceOrder,


What's "partitionedSourceOrder"? Specifically, what does "order" mean in this context?

It is the order in which you must start partitioned splits to avoid deadlocks... so for a collocated join, you must start all build splits before you schedule any probes.

martint · 2017-09-28T18:10:05Z

presto-main/src/main/java/com/facebook/presto/execution/SqlTaskExecution.java

@@ -128,6 +135,18 @@ public static SqlTaskExecution createSqlTaskExecution(
                        fragment.getPartitioningScheme(),
                        fragment.getPartitionedSources(),
                        outputBuffer);
+
+                for (DriverFactory driverFactory : localExecutionPlan.getDriverFactories()) {


What's the purpose of these validations? They just seem to be ensuring that LocalExecutionPlanner is doing its job. If it's to catch potential bugs in that class, I'd move them into that class.

martint · 2017-09-29T21:10:08Z

presto-main/src/main/java/com/facebook/presto/execution/SqlTaskExecution.java

+    }
+
+    // Splits for a particular plan node and driver group combination
+    class SplitsForDriverGroupInPlanNode


There's nothing related to driver, plan node or group in this class. It seems to be a container for related splits. A name like SplitGroup might be more appropriate

martint · 2017-09-29T21:12:45Z

presto-main/src/main/java/com/facebook/presto/execution/SqlTaskExecution.java

+    class SplitsForDriverGroupInPlanNode
+    {
+        private Set<ScheduledSplit> splits = new HashSet<>();
+        private SplitsState state = INITIALIZED;


What's the purpose of the INITIALIZED state? The only significant checks either do state == SPLITS_ADDED || state == INITIALIZED or look for NO_MORE_SPLITS. Also, I'm not sure why we need a FINISHED state. It's never used.

martint · 2017-09-29T21:15:46Z

presto-main/src/main/java/com/facebook/presto/execution/SqlTaskExecution.java

+            }
+        }
+
+        public void markAsCleanedUp()


What's this for? It doesn't do anything other than set state to FINISHED, which is never checked or used.

It effectively closes. I added checks.

martint · 2017-09-29T21:21:38Z

presto-main/src/main/java/com/facebook/presto/execution/SqlTaskExecution.java

+    }
+
+    // Splits for a particular plan node (all driver groups)
+    class SplitsForPlanNode


Tag these classes with @NotThreadSafe

martint · 2017-09-29T21:22:01Z

presto-main/src/main/java/com/facebook/presto/execution/SqlTaskExecution.java

+    // Splits for a particular plan node (all driver groups)
+    class SplitsForPlanNode
+    {
+        private final Map<DriverGroupId, SplitsForDriverGroupInPlanNode> map = new HashMap<>();


map is not a good name. Maybe perGroup?

martint

These commits look good:

Remove unused field from HivePartitionKey
Fix theoretical busy loop in HiveSplitLoader/Source
Add initialCount parameter to ReferenceCount constructor
Mark LookupJoinOperatorFactory as UsedByGeneratedCode
Improve TestMetadataManager to avoid stuck test
Make TestBackgroundSplitLoader not use DirectExecutor
Refactor TestBufferingSplitSource
Add javadoc to clarify ConnectorSplitSource.isFinished requirement
Rename TableLayout.PartitioningColumn to StreamPartitioningColumn
Rename HiveSplitSource.finished to noMoreSplits
Rename OperatorFactory.close to noMoreOperators
Document guarantee on invocation of OperatorFactory.noMoreOperators
Rename NodePartitioning to TablePartitioning

martint · 2017-10-04T23:01:32Z

presto-main/src/main/java/com/facebook/presto/execution/SqlQueryExecution.java

@@ -367,7 +367,7 @@ private PlanRoot doAnalyzeQuery()
        stateMachine.setOutput(output);

        // fragment the plan
-        SubPlan subplan = PlanFragmenter.createSubPlans(stateMachine.getSession(), metadata, plan);
+        SubPlan subplan = PlanFragmenter.createSubPlans(stateMachine.getSession(), metadata, splitManager, plan);


I forget if we discussed this, but why isn't all the necessary information available from table metadata? If that's the case, the fragmenter shouldn't have to rely on SplitManager.

martint · 2017-10-04T23:18:19Z

presto-spi/src/main/java/com/facebook/presto/spi/connector/ConnectorSplitManager.java

+
+    enum SplitSchedulingStrategy
+    {
+        ALL_AT_ONCE,


Instead of ALL_AT_ONCE, SINGLE_GROUP? "All at once" seems to imply they will all get scheduled at the same time.

maybe "ungrouped"? all at once is confusing to me because I thought this was the execution strategy (e.g. phased).

haozhun · 2017-10-06T19:10:28Z

I merged commits that @martint approved.

haozhun · 2017-11-16T01:24:00Z

Comments addressed

This allows ConnectorSplitManager implementation to return a different SplitSource depending on whether addressable split groups are needed.

ConnectorNodePartitioningProvider.listPartitionHandles lists all PartitionHandles that belong to a ConnectorPartitioningHandle. This commit is a step toward supporting addressable splits.

PlanFragmenter needs to have acceess to NodePartitioningManager to know if splits of a table can be discovered in an addressable fashion.

This code was useful when Driver can have multiple source node. This capability has been removed for a long time. This commit removes artifact left over from back then.

Previously, it takes in Fragment, and invokes LocalExecutionPlanner to get the LocalExecutionPlan. However, it continues to look into some properties in Fragment. This commit adds the additional meta properties into LocalExecutionPlan so that SqlTaskExecution don't look at Fragment any more (except to turn it into LocalExecutionPlan). This commit also adds a constructor that takes LocalExecutionPlan directly for improved testability.

This commit brings the concept of driver groups to tasks, pipelines, drivers, and operators on workers. In particular, changes are applied to SqlTask, SqlTaskExecution, Driver/OperatorFactory, Pipeline/DriverContext.

Operators that share state across drivers need to be aware of grouped execution in order to manage lifecycle correctly. LocalExchange is one such operators. LocalExchangeSinkOperator and LocalExchangeSourceOperator share page buffer across drivers.

There were some methods in PartitionedLookupSourceFactory that isn't part of LookupSourceFactory interfaces. In a later commit, I need to add a delegator for PartitionedLookupSourceFactory. It doesn't work because existing code tries to downcast LookupSourceFactory to PartitionedLookupSourceFactory, which doesn't work anymore with the delegator.

The test shuts down the executor after every single test case to terminate any outstanding threads. This can lead to excessive logging of RejectionExecutionHandler, which in turn leads to Travis failure.

sopel39 · 2017-12-13T10:55:29Z

@haozhun It would be great if you could provide some brief explanation for community how grouped execution is implemented. This change touches various components (planner, execution, etc) and it is quite large, so such description would be very helpful.

Especially some information about:

what new concepts were introduced to Presto codebase?
how planning is affected?
how execution pipeline is affected?
what are the most important considerations when using grouping?

@findepi @kokosing @losipiuk any other question ideas?

FYI: @kbajda @mattsfuller

haozhun self-assigned this Sep 11, 2017

facebook-github-bot added the CLA Signed label Sep 11, 2017

haozhun force-pushed the bbb branch 4 times, most recently from 1215c6e to 63c7eca Compare September 12, 2017 00:22

haozhun changed the title ~~[WIP] Process a subset of buckets at a time for JOINs~~ Grouped execution support for JOINs with Hive connector Sep 12, 2017

haozhun force-pushed the bbb branch from 63c7eca to 926634a Compare September 12, 2017 04:11

haozhun assigned martint and unassigned haozhun Sep 12, 2017

haozhun force-pushed the bbb branch from 926634a to 2ee514c Compare September 18, 2017 04:24

haozhun requested a review from martint September 18, 2017 04:24

haozhun force-pushed the bbb branch from 2ee514c to 9cf7f11 Compare September 22, 2017 23:40

martint reviewed Sep 25, 2017

View reviewed changes

martint reviewed Sep 26, 2017

View reviewed changes

haozhun force-pushed the bbb branch 3 times, most recently from 387cccb to 47dc5b4 Compare September 28, 2017 01:03

martint reviewed Oct 2, 2017

View reviewed changes

haozhun force-pushed the bbb branch from 47dc5b4 to 0ba83d0 Compare October 3, 2017 21:50

martint reviewed Oct 4, 2017

View reviewed changes

haozhun force-pushed the bbb branch 2 times, most recently from 3ee9398 to 83807cd Compare October 6, 2017 19:06

haozhun force-pushed the bbb branch 3 times, most recently from 5c0332f to 97c8938 Compare October 9, 2017 21:29

haozhun assigned dain Oct 9, 2017

haozhun force-pushed the bbb branch from 983e326 to a3db0a7 Compare November 16, 2017 01:23

dain approved these changes Nov 21, 2017

View reviewed changes

dain assigned haozhun and unassigned martint and dain Nov 22, 2017

haozhun force-pushed the bbb branch 2 times, most recently from 9e134fb to 65ecd9e Compare December 9, 2017 00:27

haozhun added 16 commits December 8, 2017 17:48

Pass split scheduling strategy to ConnectorSplitManager.getSplits

7fa23b9

This allows ConnectorSplitManager implementation to return a different SplitSource depending on whether addressable split groups are needed.

Inline PlanFragmenter.createSubPlans overload

62eb3de

Add PartitionHandle and listing method to SPI

d62e7f8

ConnectorNodePartitioningProvider.listPartitionHandles lists all PartitionHandles that belong to a ConnectorPartitioningHandle. This commit is a step toward supporting addressable splits.

Support addressable split group in SplitSource/Manager

7d2de34

Add NodePartitioningManager argument to PlanFragmenter

9260794

PlanFragmenter needs to have acceess to NodePartitioningManager to know if splits of a table can be discovered in an addressable fashion.

Add comments and rename fields in Driver

d665ad5

Make Driver only accept matching SourceUpdate

4879643

This code was useful when Driver can have multiple source node. This capability has been removed for a long time. This commit removes artifact left over from back then.

Add worker execution support for grouped execution

4e55aad

This commit brings the concept of driver groups to tasks, pipelines, drivers, and operators on workers. In particular, changes are applied to SqlTask, SqlTaskExecution, Driver/OperatorFactory, Pipeline/DriverContext.

Add coordinator execution support for grouped execution

fa3e1b7

Add planning support for grouped execution

d247075

Fix potential excessive logging in TestHashJoinOperator

751d10e

The test shuts down the executor after every single test case to terminate any outstanding threads. This can lead to excessive logging of RejectionExecutionHandler, which in turn leads to Travis failure.

Add grouped execution support to hash join operators

b6c6b80

Add grouped scheduling support to Hive connector

b42914d

haozhun force-pushed the bbb branch from 65ecd9e to b42914d Compare December 9, 2017 01:48

haozhun merged commit b42914d into prestodb:master Dec 9, 2017

wenleix mentioned this pull request Dec 21, 2018

[Design] Recoverable Grouped Execution #12124

Closed

wenleix mentioned this pull request Mar 5, 2019

[Design] Exchange Materialization #12387

Closed

wenleix mentioned this pull request Apr 8, 2019

Support partial merge pushdown #12611

Merged

		@@ -216,9 +216,9 @@ private void invokeFinishedIfNecessary()
		try {


		import static com.google.common.base.Preconditions.checkState;

		@JsonSerialize(using = DriverGroupId.Serializer.class)

Grouped execution support for JOINs with Hive connector #8951

Grouped execution support for JOINs with Hive connector #8951

Conversation

haozhun commented Sep 11, 2017

haozhun commented Sep 22, 2017 • edited Loading

martint left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haozhun Sep 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haozhun Sep 26, 2017 • edited Loading

Choose a reason for hiding this comment

martint left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martint left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martint left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haozhun commented Oct 6, 2017

haozhun commented Nov 16, 2017

sopel39 commented Dec 13, 2017 • edited Loading

haozhun commented Sep 22, 2017 •

edited

Loading

haozhun Sep 26, 2017 •

edited

Loading

haozhun Sep 26, 2017 •

edited

Loading

sopel39 commented Dec 13, 2017 •

edited

Loading