[#886] fix(mr): MR Client may lost data or throw exception when rss.storage.type without MEMORY.#887
Merged
roryqi merged 2 commits intoapache:masterfrom May 19, 2023
Merged
Conversation
…age.type without MEMORY.
roryqi
changed the title
Contributor
|
You can use mockClient to test this issue like |
roryqi
changed the title
Codecov Report
@@ Coverage Diff @@
## master #887 +/- ##
============================================
+ Coverage 56.20% 57.06% +0.86%
- Complexity 2150 2197 +47
============================================
Files 327 313 -14
Lines 15905 14084 -1821
Branches 1247 1306 +59
============================================
- Hits 8939 8037 -902
+ Misses 6466 5606 -860
+ Partials 500 441 -59
... and 43 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
3 tasks
roryqi
approved these changes
May 19, 2023
Contributor
roryqi
left a comment
roryqi
left a comment
There was a problem hiding this comment.
LGTM, thanks @zhengchenyu , wait for CI. Merged to master and branch-0.7.
roryqi
pushed a commit
that referenced
this pull request
May 19, 2023
…torage.type without MEMORY. (#887) ### What changes were proposed in this pull request? Make sure finishShuffle after send all shuffle data. ### Why are the changes needed? If type without MEMORY, some data will never flush. ### How was this patch tested? I test in two mode: * Tez local debug mode * MR on yarn mode Add new UT Co-authored-by: zhengchenyu001 <zhengchenyu001@ke.com>
roryqi
added a commit
that referenced
this pull request
May 19, 2023
Contributor
|
Branch 0.7 has conflicts when I cherry-pick this pr. Considering this fix is used for storage without MEMORY. These storage types isn't recommended for production environment. This pr won't be merged to branch 0.7. |
xianjingfeng
pushed a commit
to xianjingfeng/uniffle
that referenced
this pull request
Jun 20, 2023
…ion when rss.storage.type without MEMORY. (apache#887)" This reverts commit 4423b43.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Make sure finishShuffle after send all shuffle data.
Why are the changes needed?
If type without MEMORY, some data will never flush.
How was this patch tested?
I test in two mode:
Add new UT