ipfs daemon hangs on startup after crash

[andrew]

Version information:

go-ipfs version: 0.9.0
Repo version: 11
System version: 386/linux
Golang version: go1.16.5

Description:

My ipfs node crashed last night (last few hundred lines of output I could pull from screen) and I'm unable to get it to fully start again, it hangs half way through startup:

deploy@sd-117322:~$ ipfs daemon
Initializing daemon...
go-ipfs version: 0.9.0
Repo version: 11
System version: 386/linux
Golang version: go1.16.5

It stayed like this for over an hour this morning.

It appears to be doing things whilst it appears to hang, here's the log from briefly running IPFS_LOGGING=DEBUG ipfs daemon: https://gist.github.com/andrew/67334c2be4709f1c629036b7c3a62c9c

~/.ipfs/config: https://gist.github.com/andrew/e4675f8f92222aa73b51afb7ee4d6859

I've tried removing the api and repo.lock files from ~/.ipfs, no difference.

I'm using the urlstore and had added ~200k tarballs (around 1mb each) to the repo when it crashed, my .ipfs directory is approx 1.2GB in size.

ipfs stats repo also hangs, will try leaving that for a while, see if it returns eventually.

[aschmahmann]

Did some debugging on the node, closing as duplicate of #8149