Daemon stopped syncing with state root mismatch, restart fixed it

[karalabe]

My miner lost all its sectors today. Turned out that the lotus node stopped syncing with:

	Base:	[bafy2bzacebeusmfol3fl6324ov5ok4vdeboo32frb4kfwuws5mj6x6funlnqe bafy2bzaceanyw3ekqztuuhqnvahuk6m7yoqah2td25iv5mmct6yqy7oxpffpk bafy2bzacedbjsfo3wknn6m7cpft6llioye5qajb3j5im2pg3ra4ecxgqird5c bafy2bzacea7opy5on2zi767t3thukzbhd73c3ilh4jhzixiy3a6flk2dkrmfq bafy2bzaceakndvpdnneq6pfqielt22jstq7lqbgkiav7esc7mt7qew5sbjjd2 bafy2bzacedo3a3hjcsr4vwhcs4fp2t6avvu2ucvwcvic44yypi6lkhntsjqjo]
	Target:	[bafy2bzacebwvdyojebt7y547ovanr2bjpzg7mtqqqjerebegp2fbwgd7jouqk] (195545)
	Height diff:	111
	Stage: error
	Height: 0
	Elapsed: 32.025µs
	Error: chain linked to block marked previously as bad ([bafy2bzacebwvdyojebt7y547ovanr2bjpzg7mtqqqjerebegp2fbwgd7jouqk], bafy2bzaceb66ybljgiuyxeepndxxbd7u4jyxnrxcmcjdzlvrhahr4xoson742) (reason: linked to bafy2bzacecfyaq7tmjspkbh6q7cszuklg4j3aptvgia4tbtv6wlholw2rw5dk caused by: [bafy2bzacea4xiay3opdar5tzl3b3e4fiw3pokix3nxhpeciaosy2ux64ocuxq] 1 error occurred:
	* parent state root did not match computed state (bafy2bzacebyxlrlc7dqlpzy2eq7dhzr5vcpnbusvba42thfuj4eikbr6rahe4 != bafy2bzaceazymr3nhutjouahhtcphlq3yd7kygaljxpfgqiufofnhmmpymzxc):
    github.com/filecoin-project/lotus/chain.(*Syncer).ValidateBlock.func6
        /work/lotus/chain/sync.go:810

Looking at it closer, it just hit some weird path:

Oct 31 20:37:05 filecoin lotus[281]: thread 'thread 'thread '<unnamed><unnamed>' panicked at '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: >
Oct 31 20:37:05 filecoin lotus[281]: :/root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-cpupool-0.1.8/src/lib.rs
Oct 31 20:37:05 filecoin lotus[281]: /root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-cpupool-0.1.8/src/lib.rs', 13note: run with `RUST_BACKTRACE=1` environment variable t>
Oct 31 20:37:05 filecoin lotus[281]: ::/root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-cpupool-0.1.8/src/lib.rs
Oct 31 20:37:05 filecoin lotus[281]: 427427:thread '::4271313<unnamed>:
Oct 31 20:37:05 filecoin lotus[281]: thread '
Oct 31 20:37:05 filecoin lotus[281]: ' panicked at 'thread '13thread '<unnamed>called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporar>
Oct 31 20:37:05 filecoin lotus[281]: ' panicked at '<unnamed>', ' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporar>
Oct 31 20:37:05 filecoin lotus[281]: ' panicked at '
Oct 31 20:37:05 filecoin lotus[281]: ' panicked at 'thread '
Oct 31 20:37:05 filecoin lotus[281]: 13called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }called `Result::unwrap>
Oct 31 20:37:05 filecoin lotus[281]: ', thread '' panicked at '', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-cpupool-0.1.8/src/lib.rs<unnamed>called `Result::unwrap(>
Oct 31 20:37:05 filecoin lotus[281]: ::thread '
Oct 31 20:37:05 filecoin lotus[281]: thread '13427<unnamed><unnamed><unnamed>
Oct 31 20:37:05 filecoin lotus[281]: :' panicked at '' panicked at '13called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unava>
Oct 31 20:37:05 filecoin lotus[281]: <unnamed>called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }', /root/.cargo>
Oct 31 20:37:05 filecoin lotus[281]: ' panicked at '427427
Oct 31 20:37:05 filecoin lotus[281]: ', :called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }:/root/.cargo/regist>
Oct 31 20:37:05 filecoin lotus[281]: /root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-cpupool-0.1.8/src/lib.rs
Oct 31 20:37:05 filecoin lotus[281]: 427::42713:
Oct 31 20:37:05 filecoin lotus[281]: 13thread '
Oct 31 20:37:05 filecoin lotus[281]: <unnamed>thread '' panicked at '<unnamed>called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporari>
Oct 31 20:37:05 filecoin lotus[281]: :called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }called `Result::unwrap(>
Oct 31 20:37:05 filecoin lotus[281]: /root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-cpupool-0.1.8/src/lib.rs/root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures>
Oct 31 20:37:05 filecoin lotus[281]: 427' panicked at ':427', called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" >
Oct 31 20:37:05 filecoin lotus[281]: ::
Oct 31 20:37:05 filecoin lotus[281]: 427427::1313
Oct 31 20:37:05 filecoin lotus[281]: thread '<unnamed>thread '<unnamed>' panicked at 'thread '' panicked at '<unnamed>called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: >
Oct 31 20:37:05 filecoin lotus[281]: <unnamed>:', ' panicked at '427/root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-cpupool-0.1.8/src/lib.rs::13427thread 'thread ':<unnam>
Oct 31 20:37:05 filecoin lotus[281]: called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }called `Result::unwrap()>
Oct 31 20:37:05 filecoin lotus[281]: :13<unnamed>13
Oct 31 20:37:05 filecoin lotus[281]: ' panicked at '
Oct 31 20:37:05 filecoin lotus[281]: called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }', /root/.cargo/registry>
Oct 31 20:37:05 filecoin lotus[281]: thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavai>
Oct 31 20:37:05 filecoin lotus[281]: thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavai>
Oct 31 20:37:05 filecoin lotus[281]: thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavai>
Oct 31 20:37:05 filecoin lotus[281]: thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavai>
Oct 31 20:37:05 filecoin lotus[281]: thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavai>
Oct 31 20:37:05 filecoin lotus[281]: called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }', /root/.cargo/registry>
Oct 31 20:37:05 filecoin lotus[281]: thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavai>
Oct 31 20:37:05 filecoin lotus[281]: 427:13
Oct 31 20:37:05 filecoin lotus[281]: thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavai>
Oct 31 20:37:05 filecoin lotus[281]: thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavai>
Oct 31 20:37:05 filecoin lotus[281]: thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavai>
Oct 31 20:37:05 filecoin lotus[281]: <unnamed>thread '' panicked at '<unnamed>called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporari>
Oct 31 20:37:05 filecoin lotus[281]: <unnamed>', <unnamed>' panicked at '' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource>
Oct 31 20:37:05 filecoin lotus[281]: :13', thread '13
Oct 31 20:37:05 filecoin lotus[281]: /root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-cpupool-0.1.8/src/lib.rs
Oct 31 20:37:05 filecoin lotus[281]: <unnamed>:' panicked at '427called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailabl>
Oct 31 20:37:05 filecoin lotus[281]: :thread 'thread '427<unnamed><unnamed>:' panicked at '' panicked at '13called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock>
Oct 31 20:37:05 filecoin lotus[281]: ', ', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-cpupool-0.1.8/src/lib.rs/root/.cargo/registry/src/github.com-1ecc6299db9ec823/f>

A restart fixed it. Apart from all my sectors going down the drain.

[arajasek]

Thanks for the issue! This should get fixed by the proofs update being prepped in #4630.