[#1472][part-5] Use UnpooledByteBufAllocator to obtain accurate ByteBuf sizes to fix inaccurate usedMemory issue causing OOM by rickyma · Pull Request #1534 · apache/uniffle

rickyma · 2024-02-21T06:52:32Z

What changes were proposed in this pull request?

When we use UnpooledByteBufAllocator to allocate off-heap ByteBuf, Netty directly requests off-heap memory from the operating system instead of allocating it according to pageSize and chunkSize. This way, we can obtain the exact ByteBuf size during the pre-allocation of memory, avoiding distortion of metrics such as usedMemory.

Moreover, we have restored the code submission of the PR #1521. We ensure that there is sufficient direct memory for the Netty server during decoding sendShuffleDataRequest by taking into account the encodedLength of ByteBuf in advance during the pre-allocation of memory, thus avoiding OOM during decoding sendShuffleDataRequest.

Since we are not using PooledByteBufAllocator, the PR #1524 is no longer needed.

Why are the changes needed?

A sub PR for: #1519

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing UTs.

rickyma · 2024-02-21T06:59:17Z

@jerqi @zuston PTAL.

codecov-commenter · 2024-02-21T07:01:16Z

Codecov Report

Attention: 43 lines in your changes are missing coverage. Please review.

Comparison is base (59aa30d) 54.17% compared to head (d84cf09) 55.12%.

Files	Patch %	Lines
...niffle/server/netty/ShuffleServerNettyHandler.java	0.00%	15 Missing ⚠️
...he/uniffle/server/buffer/ShuffleBufferManager.java	18.75%	10 Missing and 3 partials ⚠️
...client/impl/grpc/ShuffleServerGrpcNettyClient.java	0.00%	7 Missing ⚠️
...pache/uniffle/server/ShuffleServerGrpcService.java	20.00%	3 Missing and 1 partial ⚠️
.../common/netty/protocol/SendShuffleDataRequest.java	0.00%	2 Missing ⚠️
...ava/org/apache/uniffle/common/util/NettyUtils.java	75.00%	1 Missing ⚠️
...pache/uniffle/server/NettyDirectMemoryTracker.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #1534      +/-   ##
============================================
+ Coverage     54.17%   55.12%   +0.94%     
- Complexity     2822     2824       +2     
============================================
  Files           435      415      -20     
  Lines         24501    22155    -2346     
  Branches       2074     2079       +5     
============================================
- Hits          13274    12212    -1062     
+ Misses        10397     9179    -1218     
+ Partials        830      764      -66

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

rickyma · 2024-02-21T07:02:03Z

Why is Rust CI always failed?

github-actions · 2024-02-21T07:15:10Z

Test Results

2 438 files +1 2 438 suites +1 4h 43m 24s ⏱️ + 1m 4s
823 tests ±0 822 ✅ +1 1 💤 ±0 0 ❌ ±0
9 743 runs +1 9 729 ✅ +2 14 💤 ±0 0 ❌ ±0

Results for commit d84cf09. ± Comparison against base commit 59aa30d.

♻️ This comment has been updated with latest results.

zuston · 2024-02-21T07:30:53Z

What's the initial motivation of using PooledByteBufAllocator? @leixm

XuQianJin-Stars · 2024-02-21T09:18:58Z

In the previous implementation, the pool cache was not used either.

XuQianJin-Stars · 2024-02-21T09:39:28Z

server/src/main/java/org/apache/uniffle/server/buffer/ShuffleBufferManager.java

  protected Map<String, Map<Integer, AtomicLong>> shuffleSizeMap = JavaUtils.newConcurrentMap();

  public ShuffleBufferManager(ShuffleServerConf conf, ShuffleFlushManager shuffleFlushManager) {
+    this.nettyServerEnabled = conf.get(ShuffleServerConf.NETTY_SERVER_PORT) >= 0;


Why not use RPC_SERVER_TYPE=GRPC_NETTY to judge?

XuQianJin-Stars · 2024-02-21T09:44:49Z

server/src/main/java/org/apache/uniffle/server/buffer/ShuffleBufferManager.java


  public ShuffleBufferManager(ShuffleServerConf conf, ShuffleFlushManager shuffleFlushManager) {
+    this.nettyServerEnabled = conf.get(ShuffleServerConf.NETTY_SERVER_PORT) >= 0;
    long heapSize = Runtime.getRuntime().maxMemory();


PlatformDependent.maxDirectMemory() This method contains the logic of Runtime.getRuntime().maxMemory(). Can it be merged?

We allow users to configure the maximum value of the direct memory through the variable MAX_DIRECT_MEMORY_SIZE in the rss-env.sh script, so the value of PlatformDependent.maxDirectMemory() may be different from the value of Runtime.getRuntime().maxMemory(), so we cannot merge this part of the code.

… ByteBuf sizes

rickyma · 2024-02-21T14:52:08Z

In the previous implementation, the pool cache was not used either.

Yeah, that seems right. Because the parameters smallCacheSize and normalCacheSize are set to 0 when initializing PooledByteBufAllocator.

zuston

Overall LGTM. Left some minor comments

zuston · 2024-02-22T04:40:22Z

server/src/main/java/org/apache/uniffle/server/ShuffleServerGrpcService.java

      final long start = System.currentTimeMillis();
      List<ShufflePartitionedData> shufflePartitionedData = toPartitionedData(req);
      long alreadyReleasedSize = 0;
+      boolean hasFailureOccurred = false;


This is not related with this PR. You can submit another PR to fix this.

This code logic of GRPC and Netty is basically the same. In order to solve the problem of usedMemory being inaccurate in the Netty scenario, I fixed the Netty scenario and also made the same changes to the GRPC side.

So, do we keep the changes for hasFailureOccurred in Netty? It is a bit weird to only modify the same logic code for the Netty part?

Got it. Let's reserve this.

zuston · 2024-02-22T04:45:53Z

server/src/main/java/org/apache/uniffle/server/buffer/ShuffleBufferManager.java

  protected Map<String, Map<Integer, AtomicLong>> shuffleSizeMap = JavaUtils.newConcurrentMap();

  public ShuffleBufferManager(ShuffleServerConf conf, ShuffleFlushManager shuffleFlushManager) {
+    this.nettyServerEnabled = conf.get(ShuffleServerConf.RPC_SERVER_TYPE) == ServerType.GRPC_NETTY;


This could be defined by the ShuffleServer

You wanna pass nettyServerEnabled from ShuffleServer into ShuffleBufferManager?

This should be fixed.

That would add an extra parameter in ShuffleBufferManager's constructor.

It is acceptable? Because the parameter ShuffleServerConf has already contained the value of nettyServerEnabled indirectly.

The single source principle should be ensured. And the checking logic is not same, you can see the current code in shuffleServer.java

nettyServerEnabled = shuffleServerConf.get(ShuffleServerConf.NETTY_SERVER_PORT) >= 0; if (nettyServerEnabled) { streamServer = new StreamServer(this); }

I know this. I'll fix this in the next PR which will fix #1008 together(mentioned in #1531). I've tried before. If I change this in the current PR, plenty of UTs will be failed. (really plenty of UTs, not kidding) @zuston

Ok. let me merge this.

The single source principle should be ensured. And the checking logic is not same, you can see the current code in shuffleServer.java

nettyServerEnabled = shuffleServerConf.get(ShuffleServerConf.NETTY_SERVER_PORT) >= 0; if (nettyServerEnabled) { streamServer = new StreamServer(this); }

Done in #1540.

zuston · 2024-02-22T04:50:09Z

server/src/main/java/org/apache/uniffle/server/buffer/ShuffleBufferManager.java


  public void releasePreAllocatedSize(long delta) {
-    preAllocatedSize.addAndGet(-delta);
+    if (preAllocatedSize.get() >= delta) {


it could be optimized by like this:

int allocated = preAllocatedSize.addAndGet(-delta); if (allocated < 0) { LOG.warn( "Current pre-allocated memory[" + preAllocatedSize.get() + "] is less than released[" + delta + "], set pre-allocated memory to 0"); preAllocatedSize.set(0L); }

I'm just following the same pattern in releaseMemory, releaseFlushMemory and releaseReadMemory.

Maybe we can refactor the pattern in all the above methods like you said later in another PR?

roryqi · 2024-02-22T07:46:37Z

There is a flaky test. You can see https://github.com/apache/incubator-uniffle/actions/runs/7983021033/job/21797479973

Job aborted due to stage failure: Task 0 in stage 5.0 failed 1 times, most recent failure: Lost task 0.0 in stage 5.0 (TID 14, localhost, executor driver): org.apache.uniffle.common.exception.RssFetchFailedException: Failed to read shuffle data from HOT handler
	at org.apache.uniffle.storage.handler.impl.ComposedClientReadHandler.readShuffleData(ComposedClientReadHandler.java:124)
	at org.apache.uniffle.client.impl.ShuffleReadClientImpl.read(ShuffleReadClientImpl.java:273)
	at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:185)
	at org.apache.spark.shuffle.reader.RssShuffleDataIterator.hasNext(RssShuffleDataIterator.java:115)
	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
	at org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:156)
	at org.apache.spark.Aggregator.combineCombinersByKey(Aggregator.scala:50)
	at org.apache.spark.shuffle.reader.RssShuffleReader.read(RssShuffleReader.java:155)
	at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:105)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:123)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83)
	at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:148)
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:101)
	at org.apache.uniffle.common.netty.buffer.NettyManagedBuffer.release(NettyManagedBuffer.java:59)
	at org.apache.uniffle.common.ShuffleIndexResult.release(ShuffleIndexResult.java:75)
	at org.apache.uniffle.storage.handler.impl.DataSkippableReadHandler.readShuffleData(DataSkippableReadHandler.java:82)
	at org.apache.uniffle.storage.handler.impl.ComposedClientReadHandler.readShuffleData(ComposedClientReadHandler.java:113)
	... 19 more

rickyma · 2024-02-22T08:02:47Z

There is a flaky test. You can see https://github.com/apache/incubator-uniffle/actions/runs/7983021033/job/21797479973

Job aborted due to stage failure: Task 0 in stage 5.0 failed 1 times, most recent failure: Lost task 0.0 in stage 5.0 (TID 14, localhost, executor driver): org.apache.uniffle.common.exception.RssFetchFailedException: Failed to read shuffle data from HOT handler
	at org.apache.uniffle.storage.handler.impl.ComposedClientReadHandler.readShuffleData(ComposedClientReadHandler.java:124)
	at org.apache.uniffle.client.impl.ShuffleReadClientImpl.read(ShuffleReadClientImpl.java:273)
	at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:185)
	at org.apache.spark.shuffle.reader.RssShuffleDataIterator.hasNext(RssShuffleDataIterator.java:115)
	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
	at org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:156)
	at org.apache.spark.Aggregator.combineCombinersByKey(Aggregator.scala:50)
	at org.apache.spark.shuffle.reader.RssShuffleReader.read(RssShuffleReader.java:155)
	at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:105)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:123)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83)
	at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:148)
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:101)
	at org.apache.uniffle.common.netty.buffer.NettyManagedBuffer.release(NettyManagedBuffer.java:59)
	at org.apache.uniffle.common.ShuffleIndexResult.release(ShuffleIndexResult.java:75)
	at org.apache.uniffle.storage.handler.impl.DataSkippableReadHandler.readShuffleData(DataSkippableReadHandler.java:82)
	at org.apache.uniffle.storage.handler.impl.ComposedClientReadHandler.readShuffleData(ComposedClientReadHandler.java:113)
	... 19 more

I don't think it's related to this PR. It does not happen everytime, I'll take a deep look in the next couple of days.
I think maybe #1522 does not solve the issue completely. Perhaps there are other places where potential leaks may exist.

rickyma · 2024-02-22T19:41:24Z

There is a flaky test. You can see https://github.com/apache/incubator-uniffle/actions/runs/7983021033/job/21797479973

Job aborted due to stage failure: Task 0 in stage 5.0 failed 1 times, most recent failure: Lost task 0.0 in stage 5.0 (TID 14, localhost, executor driver): org.apache.uniffle.common.exception.RssFetchFailedException: Failed to read shuffle data from HOT handler
	at org.apache.uniffle.storage.handler.impl.ComposedClientReadHandler.readShuffleData(ComposedClientReadHandler.java:124)
	at org.apache.uniffle.client.impl.ShuffleReadClientImpl.read(ShuffleReadClientImpl.java:273)
	at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:185)
	at org.apache.spark.shuffle.reader.RssShuffleDataIterator.hasNext(RssShuffleDataIterator.java:115)
	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
	at org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:156)
	at org.apache.spark.Aggregator.combineCombinersByKey(Aggregator.scala:50)
	at org.apache.spark.shuffle.reader.RssShuffleReader.read(RssShuffleReader.java:155)
	at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:105)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:123)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83)
	at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:148)
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:101)
	at org.apache.uniffle.common.netty.buffer.NettyManagedBuffer.release(NettyManagedBuffer.java:59)
	at org.apache.uniffle.common.ShuffleIndexResult.release(ShuffleIndexResult.java:75)
	at org.apache.uniffle.storage.handler.impl.DataSkippableReadHandler.readShuffleData(DataSkippableReadHandler.java:82)
	at org.apache.uniffle.storage.handler.impl.ComposedClientReadHandler.readShuffleData(ComposedClientReadHandler.java:113)
	... 19 more

Done by #1536. @jerqi

…le data requests (#1551) ### What changes were proposed in this pull request? Refresh `timestamp` when sending `SendShuffleDataRequest`. ### Why are the changes needed? A follow-up PR for: #1534 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing UTs.

… when failing to cache shuffle data (#1597) ### What changes were proposed in this pull request? Release memory more accurately when failing to cache shuffle data. ### Why are the changes needed? A follow-up PR for: #1534. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing UTs.

rickyma mentioned this pull request Feb 21, 2024

[#1472][part-5] Inaccurate flow control leads to Shuffle server OOM when enabling Netty #1531

Closed

XuQianJin-Stars reviewed Feb 21, 2024

View reviewed changes

rickyma force-pushed the issue-1472-part-5-new branch from 5da2661 to bcbd1be Compare February 21, 2024 14:28

[apache#1472][part-5] Use UnpooledByteBufAllocator to obtain accurate…

d159181

… ByteBuf sizes

rickyma force-pushed the issue-1472-part-5-new branch from bcbd1be to d159181 Compare February 21, 2024 14:31

rickyma requested a review from XuQianJin-Stars February 21, 2024 15:04

rickyma force-pushed the issue-1472-part-5-new branch 3 times, most recently from c53111c to 408ffcf Compare February 21, 2024 15:43

Use rss.rpc.server.type to judge whether Netty is enabled or not

d84cf09

rickyma force-pushed the issue-1472-part-5-new branch from 408ffcf to d84cf09 Compare February 21, 2024 16:16

zuston reviewed Feb 22, 2024

View reviewed changes

rickyma requested a review from zuston February 22, 2024 06:23

zuston approved these changes Feb 23, 2024

View reviewed changes

zuston merged commit 9142516 into apache:master Feb 23, 2024

rickyma mentioned this pull request Feb 28, 2024

[#1472][part-6][FOLLOWUP] Fix Netty transport time when sending shuffle data requests #1551

Merged

rickyma mentioned this pull request Mar 21, 2024

[#1472][FOLLOWUP] improvement(server): Release memory more accurately when failing to cache shuffle data #1597

Merged

rickyma deleted the issue-1472-part-5-new branch May 5, 2024 08:34

leixm mentioned this pull request Jul 22, 2024

[MINOR] fix(client): Fix allocate size by add 9 bytes #1940

Closed

rickyma mentioned this pull request Sep 25, 2024

[#2145] feat(server): The server-side supports choosing Netty's ByteBufAllocator. #2146

Closed

Conversation

rickyma commented Feb 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

rickyma commented Feb 21, 2024

Uh oh!

codecov-commenter commented Feb 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rickyma commented Feb 21, 2024

Uh oh!

github-actions bot commented Feb 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

zuston commented Feb 21, 2024

Uh oh!

XuQianJin-Stars commented Feb 21, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rickyma Feb 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rickyma commented Feb 21, 2024

Uh oh!

zuston left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rickyma Feb 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rickyma Feb 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rickyma Feb 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roryqi commented Feb 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rickyma commented Feb 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

rickyma commented Feb 21, 2024 •

edited

Loading

codecov-commenter commented Feb 21, 2024 •

edited

Loading

github-actions bot commented Feb 21, 2024 •

edited

Loading

rickyma Feb 21, 2024 •

edited

Loading

rickyma Feb 22, 2024 •

edited

Loading

rickyma Feb 23, 2024 •

edited

Loading

rickyma Feb 22, 2024 •

edited

Loading

roryqi commented Feb 22, 2024 •

edited

Loading

rickyma commented Feb 22, 2024 •

edited

Loading

rickyma commented Feb 22, 2024 •

edited

Loading