[Core] Likely regression in GCS memory usage from 2.2 to 2.3 releases #32090

cadedaniel · 2023-01-31T00:23:06Z

many_actors _peak_memory goes from 3.18 GB to 6.35 GB https://github.com/ray-project/ray/pull/32086/files#diff-fc74cbfe7c843f438a53b988a2c5c3d0f929b974282018139321a1dfb3cddc47R4
- gcs_server itself goes from 1.44GiB to 4.5GiB
many_nodes _peak_memory goes from 1.15 to 4.43 GB https://github.com/ray-project/ray/pull/32086/files#diff-cd08139101abdab2a959dac4641ebe110b5f3d9e54e67e6f69cdecffbf955df6R4
- gcs_server itself goes from 0.2GiB to 3.28GiB
many_pgs _peak_memory goes from 1.58 to 4.38 GB https://github.com/ray-project/ray/pull/32086/files#diff-04a50a53097a4f8df7259da0101429205591aa58085160392c1c87d3f8b77844R4
- gcs_server itself goes from 0.58GiB to 3.33GiB
many_tasks _peak_memory goes from 2.95 to 6.4 GB https://github.com/ray-project/ray/pull/32086/files#diff-d793fa081d168e96d23551cf49fe87abbbb9e6f1529af3f75271a8c19d67275bR4
- gcs_server itself goes from 1.11GiB to 4.35GiB.

The text was updated successfully, but these errors were encountered:

scv119 · 2023-02-01T17:09:10Z

hmm cc @rkooo567 @rickyyx likely due to the new task backend?

rickyyx · 2023-02-01T18:13:16Z

hmmm, we cap at 100k tasks. And the amount of data stored is only on MiB, while there are auxiliary data structures, but should be proportional to the number of tasks stored. (This is on the many_task)

But let me rerun those tests with test backend entirely disabled.

rickyyx · 2023-02-01T20:03:32Z

So I disabled task events (no task events will be reported). The memory usage is still high, and the interesting thing is it has pretty high memory usage when it just begins (look at the very first memory monitor printing):

rkooo567 · 2023-02-06T19:52:19Z

Looks like we need to bisect.

rkooo567 · 2023-02-06T19:52:27Z

Maybe some regression in GCS memory usage

fishbone · 2023-02-06T20:29:22Z

Hi @rkooo567 feel free to bisect it if you like

fishbone · 2023-02-07T20:49:53Z

File: gcs_server
Type: inuse_space
Showing nodes accounting for 990.37MB, 99.04% of 999.99MB total
Dropped 1174 nodes (cum <= 5MB)
      flat  flat%   sum%        cum   cum%
  683.41MB 68.34% 68.34%   971.24MB 97.12%  ray::rpc::ServerCallFactoryImpl::CreateCall
  133.72MB 13.37% 81.71%   200.49MB 20.05%  grpc::ServerInterface::RequestAsyncCall
   87.33MB  8.73% 90.45%    87.33MB  8.73%  google::protobuf::internal::AllocateMemory
   62.77MB  6.28% 96.72%    66.77MB  6.68%  grpc_core::Server::RequestRegisteredCall
   23.15MB  2.31% 99.04%    23.15MB  2.31%  gpr_malloc
         0     0% 99.04%    21.08MB  2.11%  (anonymous namespace)::ArenaStorage
         0     0% 99.04%    26.73MB  2.67%  clone
         0     0% 99.04%    27.14MB  2.71%  cq_next
         0     0% 99.04%    27.14MB  2.71%  end_worker
         0     0% 99.04%     5.96MB   0.6%  exec_ctx_run
         0     0% 99.04%    26.51MB  2.65%  execute_native_thread_routine
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::AllocateAlignedWithHook
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::AllocateInternal
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::CreateMaybeMessage
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::CreateMessage (inline)
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::CreateMessageInternal
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::DoCreateMessage
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::internal::ThreadSafeArena::AllocateAligned
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::internal::ThreadSafeArena::AllocateAlignedFallback
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::internal::ThreadSafeArena::GetSerialArenaFallback
         0     0% 99.04%    21.08MB  2.11%  gpr_malloc_aligned
         0     0% 99.04%    27.14MB  2.71%  grpc::CompletionQueue::AsyncNext
         0     0% 99.04%    27.14MB  2.71%  grpc::CompletionQueue::AsyncNextInternal
         0     0% 99.04%    66.77MB  6.68%  grpc::ServerInterface::PayloadAsyncRequest::PayloadAsyncRequest
         0     0% 99.04%    66.77MB  6.68%  grpc::ServerInterface::RegisteredAsyncRequest::IssueRequest
         0     0% 99.04%   200.49MB 20.05%  grpc::Service::RequestAsyncUnary
         0     0% 99.04%    21.09MB  2.11%  grpc_call_create
         0     0% 99.04%    21.09MB  2.11%  grpc_chttp2_parsing_accept_stream
         0     0% 99.04%    21.50MB  2.15%  grpc_chttp2_perform_read
         0     0% 99.04%    21.51MB  2.15%  grpc_combiner_continue_exec_ctx

zhe-thoughts · 2023-02-08T18:53:12Z

Fixing this makes me so happy :) Thanks @iycheng and folks!

rickyyx · 2023-02-08T19:52:57Z

File: gcs_server
Type: inuse_space
Showing nodes accounting for 990.37MB, 99.04% of 999.99MB total
Dropped 1174 nodes (cum <= 5MB)
      flat  flat%   sum%        cum   cum%
  683.41MB 68.34% 68.34%   971.24MB 97.12%  ray::rpc::ServerCallFactoryImpl::CreateCall
  133.72MB 13.37% 81.71%   200.49MB 20.05%  grpc::ServerInterface::RequestAsyncCall
   87.33MB  8.73% 90.45%    87.33MB  8.73%  google::protobuf::internal::AllocateMemory
   62.77MB  6.28% 96.72%    66.77MB  6.68%  grpc_core::Server::RequestRegisteredCall
   23.15MB  2.31% 99.04%    23.15MB  2.31%  gpr_malloc
         0     0% 99.04%    21.08MB  2.11%  (anonymous namespace)::ArenaStorage
         0     0% 99.04%    26.73MB  2.67%  clone
         0     0% 99.04%    27.14MB  2.71%  cq_next
         0     0% 99.04%    27.14MB  2.71%  end_worker
         0     0% 99.04%     5.96MB   0.6%  exec_ctx_run
         0     0% 99.04%    26.51MB  2.65%  execute_native_thread_routine
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::AllocateAlignedWithHook
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::AllocateInternal
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::CreateMaybeMessage
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::CreateMessage (inline)
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::CreateMessageInternal
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::DoCreateMessage
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::internal::ThreadSafeArena::AllocateAligned
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::internal::ThreadSafeArena::AllocateAlignedFallback
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::internal::ThreadSafeArena::GetSerialArenaFallback
         0     0% 99.04%    21.08MB  2.11%  gpr_malloc_aligned
         0     0% 99.04%    27.14MB  2.71%  grpc::CompletionQueue::AsyncNext
         0     0% 99.04%    27.14MB  2.71%  grpc::CompletionQueue::AsyncNextInternal
         0     0% 99.04%    66.77MB  6.68%  grpc::ServerInterface::PayloadAsyncRequest::PayloadAsyncRequest
         0     0% 99.04%    66.77MB  6.68%  grpc::ServerInterface::RegisteredAsyncRequest::IssueRequest
         0     0% 99.04%   200.49MB 20.05%  grpc::Service::RequestAsyncUnary
         0     0% 99.04%    21.09MB  2.11%  grpc_call_create
         0     0% 99.04%    21.09MB  2.11%  grpc_chttp2_parsing_accept_stream
         0     0% 99.04%    21.50MB  2.15%  grpc_chttp2_perform_read
         0     0% 99.04%    21.51MB  2.15%  grpc_combiner_continue_exec_ctx

Which tool generates this? pprof?

cadedaniel · 2023-02-08T22:36:19Z

I am reopening this until #32323 is cherry-picked onto 2.3.0

fishbone · 2023-02-08T23:59:56Z

File: gcs_server
Type: inuse_space
Showing nodes accounting for 990.37MB, 99.04% of 999.99MB total
Dropped 1174 nodes (cum <= 5MB)
      flat  flat%   sum%        cum   cum%
  683.41MB 68.34% 68.34%   971.24MB 97.12%  ray::rpc::ServerCallFactoryImpl::CreateCall
  133.72MB 13.37% 81.71%   200.49MB 20.05%  grpc::ServerInterface::RequestAsyncCall
   87.33MB  8.73% 90.45%    87.33MB  8.73%  google::protobuf::internal::AllocateMemory
   62.77MB  6.28% 96.72%    66.77MB  6.68%  grpc_core::Server::RequestRegisteredCall
   23.15MB  2.31% 99.04%    23.15MB  2.31%  gpr_malloc
         0     0% 99.04%    21.08MB  2.11%  (anonymous namespace)::ArenaStorage
         0     0% 99.04%    26.73MB  2.67%  clone
         0     0% 99.04%    27.14MB  2.71%  cq_next
         0     0% 99.04%    27.14MB  2.71%  end_worker
         0     0% 99.04%     5.96MB   0.6%  exec_ctx_run
         0     0% 99.04%    26.51MB  2.65%  execute_native_thread_routine
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::AllocateAlignedWithHook
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::AllocateInternal
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::CreateMaybeMessage
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::CreateMessage (inline)
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::CreateMessageInternal
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::Arena::DoCreateMessage
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::internal::ThreadSafeArena::AllocateAligned
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::internal::ThreadSafeArena::AllocateAlignedFallback
         0     0% 99.04%    87.33MB  8.73%  google::protobuf::internal::ThreadSafeArena::GetSerialArenaFallback
         0     0% 99.04%    21.08MB  2.11%  gpr_malloc_aligned
         0     0% 99.04%    27.14MB  2.71%  grpc::CompletionQueue::AsyncNext
         0     0% 99.04%    27.14MB  2.71%  grpc::CompletionQueue::AsyncNextInternal
         0     0% 99.04%    66.77MB  6.68%  grpc::ServerInterface::PayloadAsyncRequest::PayloadAsyncRequest
         0     0% 99.04%    66.77MB  6.68%  grpc::ServerInterface::RegisteredAsyncRequest::IssueRequest
         0     0% 99.04%   200.49MB 20.05%  grpc::Service::RequestAsyncUnary
         0     0% 99.04%    21.09MB  2.11%  grpc_call_create
         0     0% 99.04%    21.09MB  2.11%  grpc_chttp2_parsing_accept_stream
         0     0% 99.04%    21.50MB  2.15%  grpc_chttp2_perform_read
         0     0% 99.04%    21.51MB  2.15%  grpc_combiner_continue_exec_ctx

Which tool generates this? pprof?

Yes

cadedaniel added release-blocker P0 Issue that blocks the release P0 Issues that should be fixed in short order core Issues that should be addressed in Ray Core labels Jan 31, 2023

cadedaniel mentioned this issue Jan 31, 2023

[Do not merge] Compare 2.2.0 release benchmarks against 2.3 master branch #32086

Closed

cadedaniel assigned fishbone Jan 31, 2023

scv119 assigned rickyyx and unassigned fishbone Feb 1, 2023

rickyyx assigned fishbone and unassigned rickyyx Feb 1, 2023

fishbone mentioned this issue Feb 5, 2023

[core] long_running_node_failures (24d0376) weekly test failure #32180

Closed

This was referenced Feb 7, 2023

[Do not merge] Compare 2.2.0 release benchmarks against 2.3 release branch #32274

Closed

[Ray 2.3 Release] Check for performance regressions on release branch #32237

Closed

rkooo567 added the bug Something that is supposed to be working; but isn't label Feb 7, 2023

fishbone linked a pull request Feb 8, 2023 that will close this issue

[core] Fix the GCS memory usage high issue. #32302

Merged

7 tasks

fishbone closed this as completed in #32302 Feb 8, 2023

cadedaniel reopened this Feb 8, 2023

scv119 closed this as completed Feb 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core] Likely regression in GCS memory usage from 2.2 to 2.3 releases #32090

[Core] Likely regression in GCS memory usage from 2.2 to 2.3 releases #32090

cadedaniel commented Jan 31, 2023

scv119 commented Feb 1, 2023

rickyyx commented Feb 1, 2023 •

edited

Loading

rickyyx commented Feb 1, 2023

rkooo567 commented Feb 6, 2023

rkooo567 commented Feb 6, 2023

fishbone commented Feb 6, 2023

fishbone commented Feb 7, 2023

zhe-thoughts commented Feb 8, 2023

rickyyx commented Feb 8, 2023

cadedaniel commented Feb 8, 2023

fishbone commented Feb 8, 2023

[Core] Likely regression in GCS memory usage from 2.2 to 2.3 releases #32090

[Core] Likely regression in GCS memory usage from 2.2 to 2.3 releases #32090

Comments

cadedaniel commented Jan 31, 2023

scv119 commented Feb 1, 2023

rickyyx commented Feb 1, 2023 • edited Loading

rickyyx commented Feb 1, 2023

rkooo567 commented Feb 6, 2023

rkooo567 commented Feb 6, 2023

fishbone commented Feb 6, 2023

fishbone commented Feb 7, 2023

zhe-thoughts commented Feb 8, 2023

rickyyx commented Feb 8, 2023

cadedaniel commented Feb 8, 2023

fishbone commented Feb 8, 2023

rickyyx commented Feb 1, 2023 •

edited

Loading