[REFACTOR] Move DF reformat from StatementExecutionManagerImpl to QueryResultWriterImpl #701
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Louis Chu <clingzhi@amazon.com>
noCharger
requested review from
dai-chen,
rupal-bq,
vmmusings,
penghuo,
seankao-az,
anirudha,
kaituo and
YANG-DB
as code owners
noCharger
added
enhancement
dai-chen
approved these changes
Sep 28, 2024
| statementsExecutionManager.executeStatement(flintStatement) | ||
| val startTime = System.currentTimeMillis() | ||
| val df = statementsExecutionManager.executeStatement(flintStatement) | ||
| queryResultWriter.reformatDataFrame(df, flintStatement, startTime) |
There was a problem hiding this comment.
just wonder why query result writer has to expose this new API and cannot do this in writeDataFrame API? Anything happen after this and before writeDataFrame called?
There was a problem hiding this comment.
just wonder why query result writer has to expose this new API and cannot do this in
writeDataFrameAPI? Anything happen after this and before writeDataFrame called?
Reformat data will call the 'collect' API on the driver node to trigger an execution. If onlyspark.sql() is called without an execution triggerred on the same thread, Spark will not process the query execution, and the thread seems to be idle.
Verified this in IT that query will not able to process in this case.
assertion failed: Timeout occurred after 60000 milliseconds waiting for query result.
java.lang.AssertionError: assertion failed: Timeout occurred after 60000 milliseconds waiting for query result.
Add more comments in 19ac42b
Signed-off-by: Louis Chu <clingzhi@amazon.com>
seankao-az
approved these changes
Sep 30, 2024
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Sep 30, 2024
…ryResultWriterImpl (#701) * Refactor query result writer Signed-off-by: Louis Chu <clingzhi@amazon.com> * Add more scala doc and update sbt Signed-off-by: Louis Chu <clingzhi@amazon.com> --------- Signed-off-by: Louis Chu <clingzhi@amazon.com> (cherry picked from commit d76e0cd) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Sep 30, 2024
…ryResultWriterImpl (#701) * Refactor query result writer Signed-off-by: Louis Chu <clingzhi@amazon.com> * Add more scala doc and update sbt Signed-off-by: Louis Chu <clingzhi@amazon.com> --------- Signed-off-by: Louis Chu <clingzhi@amazon.com> (cherry picked from commit d76e0cd) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
noCharger
pushed a commit
that referenced
this pull request
Sep 30, 2024
…ryResultWriterImpl (#701) (#716) * Refactor query result writer * Add more scala doc and update sbt --------- (cherry picked from commit d76e0cd) Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
noCharger
added a commit
to noCharger/opensearch-spark
that referenced
this pull request
Oct 2, 2024
…l to QueryResultWriterImpl (opensearch-project#701) (opensearch-project#715)" This reverts commit 74397d8.
noCharger
added a commit
to noCharger/opensearch-spark
that referenced
this pull request
Oct 2, 2024
…l to QueryResultWriterImpl (opensearch-project#701) (opensearch-project#716)" This reverts commit daa33e5.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Move DF reformat from
StatementExecutionManagerImpltoQueryResultWriterImplso that people use customStatementExecutionManagerstill can work with defaultQueryResultWriterImpland write result to OpenSearch result index for custom data source.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.