Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Grouped execution support for JOINs with Hive connector #8951

Merged
merged 16 commits into from
Dec 9, 2017
Merged

Grouped execution support for JOINs with Hive connector #8951

merged 16 commits into from
Dec 9, 2017

Commits on Dec 9, 2017

  1. Pass split scheduling strategy to ConnectorSplitManager.getSplits 

    This allows ConnectorSplitManager implementation to return a different
SplitSource depending on whether addressable split groups are needed.
    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    7fa23b9 View commit details
    Browse the repository at this point in the history

  2. Inline PlanFragmenter.createSubPlans overload

    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    62eb3de View commit details
    Browse the repository at this point in the history

  3. Add PartitionHandle and listing method to SPI 

    ConnectorNodePartitioningProvider.listPartitionHandles lists all
PartitionHandles that belong to a ConnectorPartitioningHandle.

This commit is a step toward supporting addressable splits.
    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    d62e7f8 View commit details
    Browse the repository at this point in the history

  4. Support addressable split group in SplitSource/Manager

    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    7d2de34 View commit details
    Browse the repository at this point in the history

  5. Add NodePartitioningManager argument to PlanFragmenter 

    PlanFragmenter needs to have acceess to NodePartitioningManager to know
if splits of a table can be discovered in an addressable fashion.
    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    9260794 View commit details
    Browse the repository at this point in the history

  6. Add comments and rename fields in Driver

    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    d665ad5 View commit details
    Browse the repository at this point in the history

  7. Make Driver only accept matching SourceUpdate 

    This code was useful when Driver can have multiple source node.
This capability has been removed for a long time.
This commit removes artifact left over from back then.
    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    4879643 View commit details
    Browse the repository at this point in the history

  8. Make SqlTaskExecution work with LocalExecutionPlan instead of Fragment 

    Previously, it takes in Fragment, and invokes LocalExecutionPlanner to get
the LocalExecutionPlan. However, it continues to look into some properties
in Fragment.

This commit adds the additional meta properties into LocalExecutionPlan
so that SqlTaskExecution don't look at Fragment any more (except to turn
it into LocalExecutionPlan). This commit also adds a constructor that
takes LocalExecutionPlan directly for improved testability.
    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    0309ca4 View commit details
    Browse the repository at this point in the history

  9. Add worker execution support for grouped execution 

    This commit brings the concept of driver groups to tasks, pipelines,
drivers, and operators on workers.

In particular, changes are applied to SqlTask, SqlTaskExecution,
Driver/OperatorFactory, Pipeline/DriverContext.
    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    4e55aad View commit details
    Browse the repository at this point in the history

  10. Add coordinator execution support for grouped execution

    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    fa3e1b7 View commit details
    Browse the repository at this point in the history

  11. Add planning support for grouped execution

    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    d247075 View commit details
    Browse the repository at this point in the history

  12. Add grouped execution support to local exchange operators 

    Operators that share state across drivers need to be aware of grouped
execution in order to manage lifecycle correctly.

LocalExchange is one such operators. LocalExchangeSinkOperator and
LocalExchangeSourceOperator share page buffer across drivers.
    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    dd7e506 View commit details
    Browse the repository at this point in the history

  13. Move methods in PartitionedLookupSourceFactory to its interface 

    There were some methods in PartitionedLookupSourceFactory that isn't part of
LookupSourceFactory interfaces. In a later commit, I need to add a delegator
for PartitionedLookupSourceFactory. It doesn't work because existing code
tries to downcast LookupSourceFactory to PartitionedLookupSourceFactory,
which doesn't work anymore with the delegator.
    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    9c60025 View commit details
    Browse the repository at this point in the history

  14. Fix potential excessive logging in TestHashJoinOperator 

    The test shuts down the executor after every single test case to terminate
any outstanding threads. This can lead to excessive logging of
RejectionExecutionHandler, which in turn leads to Travis failure.
    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    751d10e View commit details
    Browse the repository at this point in the history

  15. Add grouped execution support to hash join operators

    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    b6c6b80 View commit details
    Browse the repository at this point in the history

  16. Add grouped scheduling support to Hive connector

    @haozhun
    haozhun committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    b42914d View commit details
    Browse the repository at this point in the history