Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rpcserver subscriber can hang when backsource unknown content length file #3483

Open
NickYadance opened this issue Sep 4, 2024 · 0 comments
Assignees
Labels

Comments

@NickYadance
Copy link

Bug report:

When backsource file with unknown content length, the remote peer will hang if the local peer didn't sync content length and total pieces with remote peer before sending the last piece.

Suppose file size=128MB, pieceSize=4MB, totalPieces=32. the local peer will send 32 pieces to remote peer, and total pieces=-1 in the last piece.

{"level":"info","ts":"2024-09-03 07:11:41.710","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:41.732","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:41.756","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:41.779","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:41.801","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:41.822","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:41.845","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:41.869","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:41.893","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:41.916","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:41.943","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:41.965","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:41.986","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.008","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.032","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.056","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.080","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.102","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.124","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.147","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.169","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.195","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.219","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.240","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.261","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.282","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.304","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.326","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.352","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}

This is due to the last piece(31) is sent before the content length is generated, and the remote peer will wait infinitiy since it didn't get the content length info.

{"level":"debug","ts":"2024-09-03 07:11:42.351","caller":"peer/piece_manager.go:262","msg":"piece 32 calculate digest","peer":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","task":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","component":"PeerTask","trace":"347dce459fee75f6b3ff19b0c985a745"}
{"level":"info","ts":"2024-09-03 07:11:42.352","caller":"rpcserver/subscriber.go:95","msg":"send ready piece 31","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.352","caller":"rpcserver/subscriber.go:99","msg":"sent 1 pieces, total: -1","taskID":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","localPeerID":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","remotePeerID":"xx-7-fef8e5f1-6c81-47b0-8e94-b0e5f4198b45"}
{"level":"info","ts":"2024-09-03 07:11:42.351","caller":"rpcserver/seeder.go:195","msg":"receive piece info, num: 31, ordered num: 31, finish: false","peer":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","task":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","component":"seedService"}
{"level":"info","ts":"2024-09-03 07:11:42.352","caller":"storage/local_storage.go:216","msg":"generated digest: c74f6296bec5d1f484ee54f2297461aff8e80b7a9675db36ab1dd0ed6d85ed31, total pieces: 32, content length: 134217728","task":"c5fad0a51a440ded9d2f27036b8a9bb4768b1caec402b94202a5821d8a8fa0d7","peer":"xx-1-ab92e31b-543c-48c3-b2ef-807b99491bd9_Seed","component":"localTaskStore"}

Root cause in code:

The last piece(32) size can be zero when file size aligns with piece size, so it didn't count into total pieces which is fine.

https://github.com/dragonflyoss/Dragonfly2/blob/75d624260f55578fc6abbd1a7643d02fe20a13c3/client/daemon/peer/piece_manager.go#L556-L563

But when content length is updated, the real last piece(31) can already be sent, causing content length info missed.

https://github.com/dragonflyoss/Dragonfly2/blob/75d624260f55578fc6abbd1a7643d02fe20a13c3/client/daemon/peer/piece_manager.go#L597-L602

Expected behavior:

How to reproduce it:

Add a time.Sleep before updating content length, and download a 128MB file.

https://github.com/dragonflyoss/Dragonfly2/blob/75d624260f55578fc6abbd1a7643d02fe20a13c3/client/daemon/peer/piece_manager.go#L597-L602

Environment:

  • Dragonfly version: v2.1.33
  • OS: ubuntu
  • Kernel (e.g. uname -a):
  • Others:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants