Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lotus Assigns SnapDealsWaitDeals Status to Sectors with +90 Days Duration Causing Snap Failure #10914

Closed
5 of 11 tasks
TippyFlitsUK opened this issue May 23, 2023 · 3 comments · Fixed by #11002
Closed
5 of 11 tasks
Assignees
Labels
area/sealing area/ux Area: UX kind/bug Kind: Bug need/team-input Hint: Needs Team Input P1 P1: Must be resolved

Comments

@TippyFlitsUK
Copy link
Contributor

Checklist

  • This is not a security-related bug/issue. If it is, please follow please follow the security policy.
  • I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
  • I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
  • I did not make any code changes to lotus.

Lotus component

  • lotus daemon - chain sync
  • lotus fvm/fevm - Lotus FVM and FEVM interactions
  • lotus miner/worker - sealing
  • lotus miner - proving(WindowPoSt/WinningPoSt)
  • lotus JSON-RPC API
  • lotus message management (mpool)
  • Other

Lotus Version

Daemon:  1.23.1-rc2+mainnet+git.5a1738c9b+api1.5.0
Local: lotus-miner version 1.23.1-rc2+mainnet+git.5a1738c9b

Repro Steps

  1. Snap deal is assigned to Available CC sector
  2. Sector has a duration in excess of 90 days of the deal term
  3. Sector progresses to SnapDea;lsWaitDeals
  4. Snap fails with term end beyond sector expiration

Describe the Bug

Following the activation of FIP-0045, an Available CC snap sector must have a duration that is not in excess of 90 days of the underlying deal term.

Lotus is currently allowing deals to be assigned to sectors with a duration in excess of 90 days of the underlying deal term.

The aforementioned CC sector is then progressing to a SnapDealsWaitDeals status only to fail snapping with the following message:

2023-05-22T07:07:25.905-0400	DEBUG	sectors	pipeline/sealing.go:179	deal not assignable to sector	{"sector": "185", "deal": 37635978, "dealEnd": "4036481", "dealStart": "2884481", "dealClaimEnd": "4295681", "lastAssignedDealEnd": "0", "update": true, "sectorExpiration": "4436749", "reason": "term end beyond sector expiration"}

Lotus should check the Available CC snap sector duration before assigning the deal and select a more suitable sector if the first sector's duration is in excess of 90 days of the proposed deal term. If no suitable sectors are available, the deal should either be assigned to a newly created deal sector (depending on user config settings) or remain in a pending assignment state.

This issue has been classified as P1. SPs are already seeing snap failures as a result of this and the number of reported failures is likely to grow quickly!

Slack conversation: https://filecoinproject.slack.com/archives/C04DW2KPR97/p1684752175351219
CC. @nonsense, @LexLuthr

Logging Information

2023-05-22T07:07:25.905-0400	DEBUG	sectors	pipeline/sealing.go:179	deal not assignable to sector	{"sector": "185", "deal": 37635978, "dealEnd": "4036481", "dealStart": "2884481", "dealClaimEnd": "4295681", "lastAssignedDealEnd": "0", "update": true, "sectorExpiration": "4436749", "reason": "term end beyond sector expiration"}
@TippyFlitsUK TippyFlitsUK added P1 P1: Must be resolved kind/bug Kind: Bug need/team-input Hint: Needs Team Input area/ux Area: UX area/sealing labels May 23, 2023
@TippyFlitsUK TippyFlitsUK changed the title Lotsus Assigns SnapDealsWaitDeals Status to Sectors with +90 Days Duration Causing Snap Failure Lotus Assigns SnapDealsWaitDeals Status to Sectors with +90 Days Duration Causing Snap Failure May 23, 2023
@magik6k
Copy link
Contributor

magik6k commented May 26, 2023

#10915 adds a whole bunch of improved logging, it would be great to try reproduce this issue with that PR, and get logs grepped for input.go

(That PR also cleans up the upgrade sector selection logic a bit, and removes the TargetExpiration knob, but I really doubt it was the cause of this issue)

@TippyFlitsUK
Copy link
Contributor Author

Thanks @magik6k!! I will get this installed now!!
@LexLuthr would you also be able to give this a try on FILCollins with your deal proposals that were causing issues?

@LexLuthr
Copy link
Contributor

Sure. Let me find a gap in sealing pipeline to rebuild and restart. It might take me some time to test this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/sealing area/ux Area: UX kind/bug Kind: Bug need/team-input Hint: Needs Team Input P1 P1: Must be resolved
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants