[venus-market] Deal status stuck in StorageDealStaged / 订单状态卡在 StorageDealStaged

[simlecode]

链服务模块 / Chain Service Components

• venus
• venus-auth
• venus-gateway
• venus-messager
• venus-miner
• 文档 / docs

订单服务模块 / Deal Service Components

• venus-market
• 文档 / docs

算力服务模块 / Storage Power Service Components

• venus-sector-manager
• venus-worker
• 文档 / docs

版本 / Version

v2.7.0

描述 / Describe the Bug

大量订单卡在 StorageDealStaged ，重启后能够恢复，可能是生成索引的时候卡住了

日志 / Logging Information

无

重现步骤 / Repo Steps

No response

[simlecode]

原因

从日志上来看，生成一个索引要 10 分钟左右，比较耗时，当导入大量离线订单时，部分订单会卡在 StorageDealStaged 状态，也就是等索引生成完成。目前可以同时生成 5 个索引，但生成索引的速度远小于订单生成的速度。

2023-05-03T21:23:15.774+0800  DEBUG dagstore  dagstore@v0.6.0/dagstore_async.go:117 initialize: successfully fetched from mount upgrader  {"shard": "baga6ea4seaqn7hfmmtvkej3pq5lfya4ahzovem43ioek4zt2ggmiwlec5bllcda"}
2023-05-03T21:40:54.816+0800  DEBUG dagstore  dagstore@v0.6.0/dagstore_async.go:125 initialize: finished generating index for shard {"shard": "baga6ea4seaqn7hfmmtvkej3pq5lfya4ahzovem43ioek4zt2ggmiwlec5bllcda"}


2023-05-03T21:23:15.710+0800  DEBUG dagstore  dagstore@v0.6.0/dagstore_async.go:117 initialize: successfully fetched from mount upgrader  {"shard": "baga6ea4seaqmw4j772g7fuhqsi4figdvegun346zgrgowwzlcj2hcparniogeaq"}
2023-05-03T21:31:32.302+0800  DEBUG dagstore  dagstore@v0.6.0/dagstore_async.go:125 initialize: finished generating index for shard {"shard": "baga6ea4seaqmw4j772g7fuhqsi4figdvegun346zgrgowwzlcj2hcparniogeaq"}

重启能恢复

重启的时候dagstore会自动遍历索引状态，若有索引还未生成成功，则会重新生成。当重启后 venus-market 尝试给订单生成索引，但前面已经在生成索引了，就会报错，然后订单就到 StorageDealAwaitingPreCommit 状态。

2023-05-03T21:23:44.966+0800  ERROR storageadapter  storageprovider/deal_handler.go:536 failed to activate shard: failed to schedule register shard for piece CID baga6ea4seaqmtrecgtquw35nmdhj5fahie5n3x3lvqjlne4xuo3hzickltzwqky: baga6ea4seaqmtrecgtquw35nmdhj5fahie5n3x3lvqjlne4xuo3hzickltzwqky: shard already exists
2023-05-03T21:23:44.966+0800  ERROR storageadapter  storageprovider/deal_handler.go:536 failed to activate shard: failed to schedule register shard for piece CID baga6ea4seaqmuh7f5pejqcciayuzno6kn5thjauo5ekpo4d7bw23jo6lgeoeqbi: baga6ea4seaqmuh7f5pejqcciayuzno6kn5thjauo5ekpo4d7bw23jo6lgeoeqbi: shard already exists

解决

目前索引生成失败也只是打印一行日志，没有其它额外处理，可以考虑单独的协程来等待索引生成完成，这样订单可以快速进入 StorageDealAwaitingPreCommit 状态。