fix(presto): Handle ROW data stored as string #10456
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
betodealmeida
added
!deprecated-label:bug
betodealmeida
commented
Jul 28, 2020
| @@ -653,12 +656,15 @@ def expand_data( # pylint: disable=too-many-locals | |||
| # expand columns; we append them to the left so they are added | |||
| # immediately after the parent | |||
| expanded = get_children(column) | |||
| to_process.extendleft((column, level) for column in expanded) | |||
| to_process.extendleft((column, level) for column in expanded[::-1]) | |||
There was a problem hiding this comment.
I reversed expanded so that sub-columns are added in the right order, otherwise we get:
a ROW(b, c) => a, a.c, a.b
khtruong
approved these changes
Jul 28, 2020
Codecov Report
@@ Coverage Diff @@
## master #10456 +/- ##
==========================================
+ Coverage 70.60% 72.04% +1.44%
==========================================
Files 601 620 +19
Lines 32329 36465 +4136
Branches 3275 3695 +420
==========================================
+ Hits 22826 26272 +3446
- Misses 9397 10075 +678
- Partials 106 118 +12
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
dpgaspar
changed the title
betodealmeida
added a commit
that referenced
this pull request
Jul 28, 2020
* Handle ROW data stored as string * Use destringify * Fix mypy * Fix mypy with cast * Bypass pylint
betodealmeida
pushed a commit
to lyft/incubator-superset
that referenced
this pull request
Jul 28, 2020
fix(presto): Handle ROW data stored as string (apache#10456)
jinhyukchang
pushed a commit
to lyft/incubator-superset
that referenced
this pull request
Aug 28, 2020
* Handle ROW data stored as string * Use destringify * Fix mypy * Fix mypy with cast * Bypass pylint
jinhyukchang
added a commit
to lyft/incubator-superset
that referenced
this pull request
Aug 28, 2020
[VIZ-1979] [Backporting] fix(presto): Handle ROW data stored as string (apache#10456)
auxten
pushed a commit
to auxten/incubator-superset
that referenced
this pull request
Nov 20, 2020
* Handle ROW data stored as string * Use destringify * Fix mypy * Fix mypy with cast * Bypass pylint
cccs-rc
pushed a commit
to CybercentreCanada/superset
that referenced
this pull request
Mar 6, 2024
* Handle ROW data stored as string * Use destringify * Fix mypy * Fix mypy with cast * Bypass pylint
mistercrunch
added
🏷️ bot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY
When serializing nested types we convert them to string, which breaks the
expand_datain Presto.I added a check in
expand_dataso that when the data associated with a nested column (typesARRAYandROW) is a string, it gets deserialized back into the original data.BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
A Presto query selecting an
ARRAYand aROWwithout thePRESTO_EXPAND_DATAfeature flag:With the feature enabled, data should be displayed similar to how BigQuery does, with arrays expanded into multiple lines, and rows expanded into multiple columns. This is currently broken:
This PR fixes the problem, and the data gets expanded correctly:
TEST PLAN
Tested with a simple query (see above), and added a unit test covering the problem.
ADDITIONAL INFORMATION