block count freezes after IBD

[r1ky]

Newb question. I have just started running btcsuite/btcd v0.24.0. After spending a couple days initializing the blockchain it no longer appears to be writing new blocks. btcctl getblockcount continually reports 829813 which is currently about 2 hours behind the block count reported on Blockexplorer.

The data files reflect this issue with nothing being written for the past ~2h:

root@meta-01:~# ls -lt /data/btcd-data/data/mainnet/blocks_ffldb/ | head
total 535368128
drwxr-xr-x 2 root root   1859584 Feb 10 16:58 metadata
-rw-r--r-- 1 root root 162222618 Feb 10 14:53 000001022.fdb
-rw-r--r-- 1 root root 536286030 Feb 10 14:51 000001021.fdb
-rw-r--r-- 1 root root 535253677 Feb 10 14:47 000001020.fdb

I've included a snippet from the logs below that appears to be the start of the problem. After that there are just repeated occurrences of the last reported Rejected block reported at the end of the log excerpt.

Does anyone have an idea what may be wrong?

2024-02-10 14:53:50.935 [INF] SYNC: Rejected block 00000000000000000001a9931d3181dcad10319187e0b668fc2db0f8dff656c9 from 158.69.116.1
69:8333 (outbound): output 314f9abdebc73c4c2062b2416278f4956b399f4f2ae1250aa065e23fab85e9fd:0 referenced from transaction 1dc8c6b6063
3eb711818a95c23034faf0fffdb278ac29476b99bfd47c029a209:49 either does not exist or has already been spent
2024-02-10 14:53:50.968 [INF] SYNC: Rejected block 00000000000000000003b3480ed47cd0d435069efcc1eccdbcd84629a9e912ae from 158.69.116.1
69:8333 (outbound): previous block 00000000000000000001a9931d3181dcad10319187e0b668fc2db0f8dff656c9 is known to be invalid
2024-02-10 14:53:51.019 [INF] CHAN: Adding orphan block 00000000000000000001be54eb192b683a066b035ec59547de17eaaded2cfa73 with parent 
00000000000000000003b3480ed47cd0d435069efcc1eccdbcd84629a9e912ae
2024-02-10 14:53:51.044 [INF] CHAN: Adding orphan block 00000000000000000000444b5117d4dd006490de38ce067443ac1c9a1ccc0f08 with parent 
00000000000000000001be54eb192b683a066b035ec59547de17eaaded2cfa73
2024-02-10 14:53:51.109 [INF] CHAN: Adding orphan block 00000000000000000001a05e045444fb4227933d5be5f0efc54ca926963d0c43 with parent 
00000000000000000000444b5117d4dd006490de38ce067443ac1c9a1ccc0f08
2024-02-10 14:53:51.206 [INF] CHAN: Adding orphan block 000000000000000000019e6204beb96b122036247a8664bf4d66a8b811f769db with parent 
00000000000000000001a05e045444fb4227933d5be5f0efc54ca926963d0c43
2024-02-10 14:53:51.253 [INF] CHAN: Adding orphan block 00000000000000000000b7f267dd8f7a867287fb0c07e164187cf585c5639d02 with parent 
000000000000000000019e6204beb96b122036247a8664bf4d66a8b811f769db
2024-02-10 14:53:51.305 [INF] CHAN: Adding orphan block 0000000000000000000227e6c2da74b6ef93fc7743a5878fc6e07c8b7f887ee4 with parent 00000000000000000000b7f267dd8f7a867287fb0c07e164187cf585c5639d02
2024-02-10 14:53:51.377 [INF] CHAN: Adding orphan block 00000000000000000003237d04ad2415292d0fe61e5d367b72fd07b3172e1199 with parent 0000000000000000000227e6c2da74b6ef93fc7743a5878fc6e07c8b7f887ee4
2024-02-10 14:53:51.436 [INF] CHAN: Adding orphan block 00000000000000000000c6cd552536b518f42a959a0576c19fad4421ae858ed2 with parent 00000000000000000003237d04ad2415292d0fe61e5d367b72fd07b3172e1199
2024-02-10 14:53:51.493 [INF] CHAN: Adding orphan block 00000000000000000001c720d7375d41a0839da2e8209d270e68794df61b65a1 with parent 00000000000000000000c6cd552536b518f42a959a0576c19fad4421ae858ed2
2024-02-10 14:53:51.544 [INF] CHAN: Adding orphan block 00000000000000000002db1605171cc371564792013fcb1898bbce8a4fa5d5e2 with parent 00000000000000000001c720d7375d41a0839da2e8209d270e68794df61b65a1
2024-02-10 14:53:51.585 [INF] CHAN: Adding orphan block 0000000000000000000380f62b7ee0b43ebd0e2e3389fdcd782ebd5b3fcb0513 with parent 00000000000000000002db1605171cc371564792013fcb1898bbce8a4fa5d5e2
2024-02-10 14:53:51.636 [INF] CHAN: Adding orphan block 00000000000000000003775d97ec610e54b8b201b5feff6db72ff9d7754beac2 with parent 0000000000000000000380f62b7ee0b43ebd0e2e3389fdcd782ebd5b3fcb0513
2024-02-10 14:53:51.676 [INF] CHAN: Adding orphan block 000000000000000000001b770eb2d1a54be1b5a1b5270921b361bc72b59591e2 with parent 00000000000000000003775d97ec610e54b8b201b5feff6db72ff9d7754beac2
2024-02-10 14:53:51.757 [INF] CHAN: Adding orphan block 00000000000000000003a3b72ea88be42356514ebd74a7c2a80a539a71befd42 with parent 000000000000000000001b770eb2d1a54be1b5a1b5270921b361bc72b59591e2
2024-02-10 14:53:51.805 [INF] CHAN: Adding orphan block 000000000000000000001d009527deada865fbf9751609d7419aa8564f0b1eb9 with parent 00000000000000000003a3b72ea88be42356514ebd74a7c2a80a539a71befd42
2024-02-10 14:53:51.867 [INF] CHAN: Adding orphan block 000000000000000000031895d8101046803363852d84c9fcf3c10c18baa06978 with parent 000000000000000000001d009527deada865fbf9751609d7419aa8564f0b1eb9
2024-02-10 14:53:51.933 [INF] CHAN: Adding orphan block 00000000000000000000956e8b14383d75dc47aa86b7c745e8074ffd801f88af with parent 000000000000000000031895d8101046803363852d84c9fcf3c10c18baa06978
2024-02-10 14:53:52.005 [INF] CHAN: Adding orphan block 00000000000000000002237a1aa31b928a7f89211c9e82c9e539405eb7f6d224 with parent 00000000000000000000956e8b14383d75dc47aa86b7c745e8074ffd801f88af
2024-02-10 14:53:52.575 [INF] SYNC: Rejected block 00000000000000000003b3480ed47cd0d435069efcc1eccdbcd84629a9e912ae from 158.69.116.169:8333 (outbound): previous block 00000000000000000001a9931d3181dcad10319187e0b668fc2db0f8dff656c9 is known to be invalid