[BUG] Fix potential memory leak when encountering disk unhealthy

[zuston]

What changes were proposed in this pull request?

Fix potential memory leak when encountering disk unhealthy

Why are the changes needed?

When encountering disk unhealthy and exceed the timeout of pendingDataShuffleFlushEvent, it will release memory. But in current codebase, it wont release the data reference and cause the memory leak.

Does this PR introduce any user-facing change?

No

How was this patch tested?

[zuston]

@xianjingfeng @jerqi PTAL . Does this is a critical bug? I found the shuffle-server is always oom when encountering disk unhealthy(like reaching the highwatermark).

[roryqi]

Could you explain why this cause memory leak?

[codecov-commenter]

Codecov Report

Merging #370 (86f8320) into master (ad51341) will increase coverage by 0.72%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master     #370      +/-   ##
============================================
+ Coverage     58.01%   58.73%   +0.72%     
- Complexity     1361     1581     +220     
============================================
  Files           171      193      +22     
  Lines          9006    10843    +1837     
  Branches        787      947     +160     
============================================
+ Hits           5225     6369    +1144     
- Misses         3449     4099     +650     
- Partials        332      375      +43     

Impacted Files	Coverage Δ
...org/apache/uniffle/server/ShuffleFlushManager.java	79.14% <100.00%> (+0.22%)	⬆️
...org/apache/spark/shuffle/writer/AddBlockEvent.java	0.00% <0.00%> (ø)
.../org/apache/spark/shuffle/writer/WriterBuffer.java	93.18% <0.00%> (ø)
...java/org/apache/hadoop/mapred/SortWriteBuffer.java	90.90% <0.00%> (ø)
...pache/hadoop/mapreduce/task/reduce/RssFetcher.java	90.90% <0.00%> (ø)
...pache/spark/shuffle/writer/WriteBufferManager.java	83.82% <0.00%> (ø)
.../java/org/apache/spark/shuffle/RssSparkConfig.java	93.61% <0.00%> (ø)
...pache/hadoop/mapreduce/task/reduce/RssShuffle.java	0.00% <0.00%> (ø)
...che/spark/shuffle/writer/BufferManagerOptions.java	75.67% <0.00%> (ø)
...org/apache/spark/shuffle/RssSparkShuffleUtils.java	68.42% <0.00%> (ø)
... and 14 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[zuston]

Could you explain why this cause memory leak?

It releases memory but not release the data reference. Right?

[xianjingfeng]

Could you explain why this cause memory leak?

It releases memory but not release the data reference. Right?

It seems yes. I think it is the reason of #164

[xianjingfeng]

LGTM

[roryqi]

Got it. LGTM, thanks @zuston @xianjingfeng

[zuston]

Maybe this PR should be cherry-picked to release-0.6.0-RC2.

[roryqi]

Maybe this PR should be cherry-picked to release-0.6.0-RC2.

You can cherry-pick this pr to branch-0.6. And reply the mail to the release manager in the dev@uniffle.apache.org.

[kaijchen]

Maybe this PR should be cherry-picked to release-0.6.0-RC2.

Thanks @zuston for the fix. Yes I think this is an important fix, I'll backport this into branch-0.6 and start a v0.6.0-rc3 release.