calculate_logical_size: no longer use spawn_blocking #3664
Conversation
koivunej
requested review from
knizhnik and
hlinnaka
and removed request for
a team
koivunej
changed the title
There was a problem hiding this comment.
Yeah, this is clearly the right thing to do.
I do wonder about the issue mentioned in the comment that this removes, though:
// Run in a separate thread since this can do a lot of
// synchronous file IO without .await inbetween
// if there are no RemoteLayers that would require downloading.
That's still a valid concern, right? I guess it's a more general issue that we use blocking I/O on files, and we don't need to specifically care about it here.
This is an issue still yes, more like #2975, but we created a deadlock with the |
this fixes a bug which exhausted the spawn_blocking pool which hung up the blocking io operations needed by download remote layer, because these threads used block_on.
koivunej
force-pushed
the
not_blocking_anymore
branch
from
7105485
to
d0238e2
Compare
|
Yay, first non-transient pass of |
Calculation of logical size is now async because of layer downloads, so we shouldn't use spawn_blocking for it. Use of `spawn_blocking` exhausted resources which are needed by `tokio::io::copy` when copying from a stream to a file which lead to deadlock. Fixes: #3657
Calculation of logical size is now async because of layer downloads, so we shouldn't use spawn_blocking for it. Use of
spawn_blockingexhausted resources which are needed bytokio::io::copywhen copying from a stream to a file which lead to deadlock.Fixes: #3657