Mine is taking removed/faulty/inactive sectors for computing winning post proof

[jennijuju]

Seems like since our sector remove is not declaring the sector has been removed on chain(partially related #3841), mine is still picking those sectors up for computing winning post proof which triggers the following error

2020-09-25T03:00:59.170+0800    INFO    storageminer    storage/miner.go:237    Computing WinningPoSt ;[{SealProof:3 SectorNumber:541 SealedCID:bagboea4b5abcauhi3mvvjexxy4scjinnvc2ucxh5m7bowdlchlddrptb3obwhzbm}]; [35 159 97 13 183 242 203 117 11 115 187 167 156 35 180 209 153 89 61 124 229 48 106 3 205 135 244 88 52 119 202 47]
2020-09-25T03:00:59.170 INFO filcrypto::proofs::api > generate_winning_post: start
2020-09-25T03:00:59.170 INFO filcrypto::proofs::api > generate_winning_post: finish
2020-09-25T03:00:59.170+0800    ERROR   miner   miner/miner.go:226      mining block failed: failed to compute winning post proof:
    github.com/filecoin-project/lotus/miner.(*Miner).mineOne
        /root/lotus/miner/miner.go:418
  - could not read from path="p_aux"
    Caused by:
        No such file or directory (os error 2)
    github.com/filecoin-project/filecoin-ffi.GenerateWinningPoSt
        /root/lotus/extern/filecoin-ffi/proofs.go:545
    github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).GenerateWinningPoSt
        /root/lotus/extern/sector-storage/ffiwrapper/verifier_cgo.go:32
    github.com/filecoin-project/lotus/storage.(*StorageWpp).ComputeProof
        /root/lotus/storage/miner.go:240
    github.com/filecoin-project/lotus/miner.(*Miner).mineOne
        /root/lotus/miner/miner.go:416
    github.com/filecoin-project/lotus/miner.(*Miner).mine
        /root/lotus/miner/miner.go:224
    runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373

[jennijuju]

Seems like it is also picking up terminated sectors, which is unexpected from our discussion(we said that if we add termination option to remove, then this should be resolved naturally).

For example, miner f010399 has removed a lot of sectors due to sanity check error back in SR1 and they have been removed more than 1 month. Status suggests the same.

lotus-miner sectors status --on-chain-info 1446
SectorID:       1446
Status:         Removed
Expiration Info
OnTime:         0
Early:          0

However, mine is still trying to computing this winning post base on this sector

2020-10-17T18:56:36.380+0800    INFO    storageminer    storage/miner.go:246    Computing WinningPoSt ;[{SealProof:3 SectorNumber:1446 SealedCID:bagboea4b5abcbu25klcolqa7sfyvjiwr6zlcamny3kwlrhzgyyjb7qd2tdfnyhsk}]; [205 64 221 44 130 50 8 38 154 117 253 225 141 24 238 51 173 38 4 81 129 46 52 111 64 90 94 213 106 92 98 188]
2020-10-17T18:56:36.380 INFO filcrypto::proofs::api > generate_winning_post: start
2020-10-17T18:56:36.380 INFO filcrypto::proofs::api > generate_winning_post: finish
2020-10-17T18:56:36.380+0800    ERROR   miner   miner/miner.go:228      mining block failed: failed to compute winning post proof:
    github.com/filecoin-project/lotus/miner.(*Miner).mineOne
        /root/lotus/miner/miner.go:421
  - could not read from path="p_aux"
    Caused by:
        No such file or directory (os error 2)
    github.com/filecoin-project/filecoin-ffi.GenerateWinningPoSt
--
        /root/lotus/extern/sector-storage/ffiwrapper/verifier_cgo.go:32
    github.com/filecoin-project/lotus/storage.(*StorageWpp).ComputeProof
        /root/lotus/storage/miner.go:249
    github.com/filecoin-project/lotus/miner.(*Miner).mineOne
        /root/lotus/miner/miner.go:419
    github.com/filecoin-project/lotus/miner.(*Miner).mine
        /root/lotus/miner/miner.go:226
    runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373 (edited) 

For a smaller miner that has a lot of removed sectors, this is effect their block winning chance severely.

The miner mentioned that almost everyday, such a sector will be picked for winning post.

Total: 6354
        Proving: 4697
        Empty: 4
        WaitDeals: 2
        Packing: 2
        PreCommit1: 1
        Committing: 6
        FinalizeSector: 7
        Removed: 1631
        FailedUnrecoverable: 3
        RecoverDealIDs: 1

[axzmin]

my miner Computing WinningPoSt selected these removed sectors everyday,please help me! thanks!

[arajasek]

Low effort, but maybe needs a FIP.

[jennijuju]

Team decided no FIP required

[arajasek]

Closed by #4770

[axzmin]

lotus version
Daemon: 1.2.0+git.dc3d17c9e+api1.0.0
Local: lotus version 1.2.0+git.dc3d17c9e

This problem not resolved on v1.2.0,

2020-11-20T05:48:28.696+0800 INFO storageminer storage/miner.go:249 Computing WinningPoSt ;[{SealProof:3 SectorNumber:476 SealedCID:bagboea4b5abcbyfnohy5ezjqb7khrqnjzgzoj3elpzn4gbxwq4q3oczgr2cx5hbb}]; [181 49 175 84 243 229 101 58 59 234 54 117 230 117 218 132 253 206 12 132 148 234 46 246 5 149 162 145 107 217 73 248]
2020-11-20T05:48:29.711+0800 INFO storageminer storage/miner.go:249 Computing WinningPoSt ;[{SealProof:3 SectorNumber:476 SealedCID:bagboea4b5abcbyfnohy5ezjqb7khrqnjzgzoj3elpzn4gbxwq4q3oczgr2cx5hbb}]; [181 49 175 84 243 229 101 58 59 234 54 117 230 117 218 132 253 206 12 132 148 234 46 246 5 149 162 145 107 217 73 248]

SectorID: 476
Status: Removed
CIDcommD: baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
CIDcommR: bagboea4b5abcbyfnohy5ezjqb7khrqnjzgzoj3elpzn4gbxwq4q3oczgr2cx5hbb
Ticket: cac14146297b522f8275133e20664f2c23ec49e0f007444e77f1529436ace8f9
TicketH: 19855