Add basic implementation of Unnest operator #27
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
@mbasmanova has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
|
This pull request was exported from Phabricator. Differential Revision: D30264256 |
kgpai
approved these changes
Aug 12, 2021
| @@ -34,4 +34,53 @@ const std::vector<std::shared_ptr<const PlanNode>>& ExchangeNode::sources() | |||
| return EMPTY_SOURCES; | |||
| } | |||
|
|
|||
| UnnestNode::UnnestNode( | |||
There was a problem hiding this comment.
Basic Question, does Unnest here mean hive explode or does it mean something else ?
There was a problem hiding this comment.
| std::vector<TypePtr> types; | ||
|
|
||
| for (const auto& variable : replicateVariables_) { | ||
| names.emplace_back(variable->name()); |
There was a problem hiding this comment.
nit: Might I suggest std::vector<Pair<string, TypePtr>> so we can use an iterator to access both instead of an index ?
There was a problem hiding this comment.
Type constructor takes two vectors, hence, no flexibility here.
| ChannelIndex outputChannel = 0; | ||
| for (const auto& variable : unnestNode->replicateVariables()) { | ||
| identityProjections_.emplace_back( | ||
| inputType->getChildIdx(variable->name()), outputChannel++); |
There was a problem hiding this comment.
Can you explain what ChannelIndex, childIdx here are ?
There was a problem hiding this comment.
It helps to think about input data as a table made of a columns. Each column has a name and an ordinal zero-based position called channel. inputType is a RowType made of of column types. childIdx = channel and ChannelIndex is a typedef for the type of channel.
using ChannelIndex = uint32_t;
| // Create "indices" buffer to repeat rows as many times as there are elements | ||
| // in the array. | ||
| BufferPtr repeatedIndices = | ||
| AlignedBuffer::allocate<vector_size_t>(numElements, pool()); |
There was a problem hiding this comment.
Is it possible for vector_size_t to be 0 here ? Should we also do some some max size check ?
There was a problem hiding this comment.
vector_size_t is a type, hence, cannot be zero, but numElements can be zero. Nice catch. Will update the code to handle that case properly.
velox/exec/tests/UnnestTest.cpp
Outdated
| assertQuery( | ||
| op, | ||
| std::vector<std::shared_ptr<TempFilePath>>{}, | ||
| "SELECT c0, x FROM tmp, UNNEST(ARRAY[0, 1, 2]) as t(x)"); |
There was a problem hiding this comment.
This initial cut of UNNEST doesn't support maps.
|
This pull request was exported from Phabricator. Differential Revision: D30264256 |
|
@kgpai Krishna, thank you for review. I replied to questions and updated the code to handle numElements == 0 explicitly. Would you take another look? |
|
This pull request was exported from Phabricator. Differential Revision: D30264256 |
|
This pull request was exported from Phabricator. Differential Revision: D30264256 |
Summary: This is an initial cut which supports only one unnest column of type ARRAY and doesn't support WITH ORDINALITY clause. Pull Request resolved: facebookincubator/velox#27 Differential Revision: D30264256 Pulled By: mbasmanova fbshipit-source-id: adf0c138d0a48a37437eb11e628c90532b21bdf5
|
This pull request was exported from Phabricator. Differential Revision: D30264256 |
|
@mbasmanova merged this pull request in 545a610. |
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Jun 29, 2022
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Jun 30, 2022
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Jul 20, 2022
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Aug 2, 2022
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Aug 12, 2022
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Aug 22, 2022
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Sep 7, 2022
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Sep 26, 2022
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format Removed special handling for avg (facebookincubator#31) [OPPRO-173] Make batch size configurable (facebookincubator#32) support dwrf format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Oct 26, 2022
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format Removed special handling for avg (facebookincubator#31) [OPPRO-173] Make batch size configurable (facebookincubator#32) support dwrf format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Nov 8, 2022
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format Removed special handling for avg (facebookincubator#31) [OPPRO-173] Make batch size configurable (facebookincubator#32) support dwrf format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Nov 8, 2022
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format Removed special handling for avg (facebookincubator#31) [OPPRO-173] Make batch size configurable (facebookincubator#32) support dwrf format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Nov 22, 2022
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format Removed special handling for avg (facebookincubator#31) [OPPRO-173] Make batch size configurable (facebookincubator#32) support dwrf format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Dec 15, 2022
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format Removed special handling for avg (facebookincubator#31) [OPPRO-173] Make batch size configurable (facebookincubator#32) support dwrf format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Jan 6, 2023
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format Removed special handling for avg (facebookincubator#31) [OPPRO-173] Make batch size configurable (facebookincubator#32) support dwrf format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Jan 12, 2023
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format Removed special handling for avg (facebookincubator#31) [OPPRO-173] Make batch size configurable (facebookincubator#32) support dwrf format
PHILO-HE
pushed a commit
to PHILO-HE/velox
that referenced
this pull request
Feb 3, 2023
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format Removed special handling for avg (facebookincubator#31) [OPPRO-173] Make batch size configurable (facebookincubator#32) support dwrf format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Feb 24, 2023
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format Removed special handling for avg (facebookincubator#31) [OPPRO-173] Make batch size configurable (facebookincubator#32) support dwrf format
liujiayi771
pushed a commit
to liujiayi771/velox
that referenced
this pull request
Mar 3, 2023
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format Removed special handling for avg (facebookincubator#31) [OPPRO-173] Make batch size configurable (facebookincubator#32) support dwrf format
liujiayi771
pushed a commit
to liujiayi771/velox
that referenced
this pull request
Mar 9, 2023
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format Removed special handling for avg (facebookincubator#31) [OPPRO-173] Make batch size configurable (facebookincubator#32) support dwrf format
liujiayi771
pushed a commit
to liujiayi771/velox
that referenced
this pull request
Apr 1, 2023
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format Removed special handling for avg (facebookincubator#31) [OPPRO-173] Make batch size configurable (facebookincubator#32) support dwrf format
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Apr 23, 2023
…incubator#27) * Filter validation for Parquet reader at runtime * Style * Style * Format Removed special handling for avg (facebookincubator#31) [OPPRO-173] Make batch size configurable (facebookincubator#32) support dwrf format
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an initial cut which supports only one unnest column of type ARRAY and doesn't support WITH ORDINALITY clause.