downloader: stoped at 100%

[AskAlexSharov]

sync from scratch. today main. eth-mainnet. restart - helped.
FYI: after start on empty datadir - i did 1 restart at ~10% progress.

goroutine.txt

[INFO] [11-16|03:49:39.268] [1/6 OtterSync] Downloading              progress="(1282/1282 files) 100.00% - 1.7TB/1.7TB" rate=0B/s time-left=999hrs:99m total-time=18h24m20s download-rate=0B/s completion-rate=0B/s alloc=5.2GB sys=19.4GB
[DBUG] [11-16|03:49:39.275] [snapshots] progress                     file=v1-001500-002000-transactions.seg progress=99.49% peers=1 webseeds=5
[DBUG] [11-16|03:49:39.276] [snapshots] webseed peers                file=v1-001500-002000-transactions.seg erigon3-v3-snapshots-mainnet.erigon.network=228.5KB/s erigon3-v3-snapshots-mainnet.erigon.network/v2=105.8KB/s erigon2-v1-snapshots-mainnet.erigon.network=227.9KB/s erigon2-v3-snapshots-mainnet.erigon.network=523.3KB/s erigon3-v1-snapshots-mainnet.erigon.network=457.6KB/s
[DBUG] [11-16|03:49:39.276] [snapshots] bittorrent peers             file=v1-001500-002000-transactions.seg libtorrent/2.0.10.0=0B/s
[DBUG] [11-16|03:49:39.279] [snapshots] progress                     file=v1-018200-018300-transactions.seg progress=99.98% peers=2 webseeds=5
[DBUG] [11-16|03:49:39.279] [snapshots] webseed peers                file=v1-018200-018300-transactions.seg erigon3-v1-snapshots-mainnet.erigon.network=1.1MB/s erigon3-v3-snapshots-mainnet.erigon.network=1.2MB/s erigon3-v3-snapshots-mainnet.erigon.network/v2=1.6MB/s erigon2-v1-snapshots-mainnet.erigon.network=1.6MB/s erigon2-v3-snapshots-mainnet.erigon.network=1.6MB/s
[DBUG] [11-16|03:49:39.279] [snapshots] bittorrent peers             file=v1-018200-018300-transactions.seg libtorrent/2.0.10.0=0B/s libtorrent/2.0.10.0=0B/s
[DBUG] [11-16|03:49:39.290] [snapshots] progress                     file=v1-019100-019200-transactions.seg progress=99.98% peers=3 webseeds=5
[DBUG] [11-16|03:49:39.290] [snapshots] webseed peers                file=v1-019100-019200-transactions.seg erigon2-v1-snapshots-mainnet.erigon.network=1.2MB/s erigon2-v3-snapshots-mainnet.erigon.network=1.2MB/s erigon3-v1-snapshots-mainnet.erigon.network=1.1MB/s erigon3-v3-snapshots-mainnet.erigon.network=1.6MB/s erigon3-v3-snapshots-mainnet.erigon.network/v2=1012.2KB/s
[DBUG] [11-16|03:49:39.290] [snapshots] bittorrent peers             file=v1-019100-019200-transactions.seg libtorrent/2.0.10.0=0B/s libtorrent/2.0.10.0=0B/s libtorrent/2.0.10.0=0B/s
[DBUG] [11-16|03:49:39.294] [snapshots] progress                     file=v1-003000-003500-transactions.seg progress=99.65% peers=3 webseeds=5
[DBUG] [11-16|03:49:39.294] [snapshots] webseed peers                file=v1-003000-003500-transactions.seg erigon2-v1-snapshots-mainnet.erigon.network=683.8KB/s erigon2-v3-snapshots-mainnet.erigon.network=681.5KB/s erigon3-v1-snapshots-mainnet.erigon.network=464.6KB/s erigon3-v3-snapshots-mainnet.erigon.network=220.9KB/s erigon3-v3-snapshots-mainnet.erigon.network/v2=2.4MB/s
[DBUG] [11-16|03:49:39.294] [snapshots] bittorrent peers             file=v1-003000-003500-transactions.seg libtorrent/2.0.10.0=0B/s libtorrent/2.0.10.0=0B/s libtorrent/2.0.10.0=0B/s
[DBUG] [11-16|03:49:39.298] [snapshots] progress                     file=v1-015500-016000-transactions.seg progress=99.99% peers=2 webseeds=5
[DBUG] [11-16|03:49:39.298] [snapshots] webseed peers                file=v1-015500-016000-transactions.seg erigon2-v3-snapshots-mainnet.erigon.network=1.8MB/s erigon3-v1-snapshots-mainnet.erigon.network=2.2MB/s erigon3-v3-snapshots-mainnet.erigon.network=1.7MB/s erigon3-v3-snapshots-mainnet.erigon.network/v2=2.0MB/s erigon2-v1-snapshots-mainnet.erigon.network=2.3MB/s
[DBUG] [11-16|03:49:39.298] [snapshots] bittorrent peers             file=v1-015500-016000-transactions.seg libtorrent/2.0.10.0=0B/s libtorrent/2.0.10.0=0B/s
[DBUG] [11-16|03:49:39.302] [snapshots] files progress               files=5 list="v1-001500-002000-transactions.seg (99%), v1-003000-003500-transactions.seg (100%), v1-015500-016000-transactions.seg (100%), v1-018200-018300-transactions.seg (100%), v1-019100-019200-transactions.seg (100%)"
[DBUG] [11-16|03:49:39.302] [snapshots] downloading                  len=1282 webTransfers=0 torrent=1282 db=5 t-complete=1277 hashed=1.9TB hash-rate=0B/s completed=1.7TB completion-rate=0B/s flushed=1.8TB flush-rate=0B/s webseed-trips=989756 webseed-active=0 webseed-max-active=1869 webseed-discards=0 webseed-fails=43 webseed-bytes=1.9TB localHashes=0 localHashTime=0s

[RedstonedLife]

I'll recount my experience with the downloader slowing down regular speeds (56+ MB/s for me) all the way down to 0 B/s.

1. This may happen at random intervals.
   From what I observed there isn't a clear stage or point at which it slows down, but just that it randomly slows down, as if its impeded by something else on the network (network checks showed that the only running task that uses majority bandwidth was Erigon). It would slow at 11% sometimes at 20% sometimes after downloading 0.2%.
2. Restarts help (sort of)
   From observations, restarts help. Sort of.
   A restart may cause it to download more of the data yes, but in my case when encountering this, a restart would give it a temporary boost enough to download between 0.2%-4%. Multiple restarts seem to "unclog" it, and make it download at normal speeds until it again halts to 0 B/s.
3. In some cases leaving it alone helps.
   From testing it seems that sometimes, It regains speed and composure after 3-5 minutes of 0 B/s speeds and regains speed. But in the majority of times this happened to me, waiting for it to speed up resulted in nothing.

These are the keynotes I have from my encounter with this issue.

[RedstonedLife]

Encountered this issue again at donwloader

Nov 20 22:43:00 enode-01 erigon[12466]: [INFO] [11-20|22:43:00.391] [1/6 OtterSync] Downloading              progress="(284/284 files) 99.96% - 10.4GB/10.4GB" rate=572.3KB/s time-left=0hrs:0m total-time=6m20s download-rate=572.3KB/s completion-rate=572.3KB/s alloc=2.3GB sys=6.3GB
Nov 20 22:43:20 enode-01 erigon[12466]: [INFO] [11-20|22:43:20.392] [1/6 OtterSync] Downloading              progress="(284/284 files) 99.96% - 10.4GB/10.4GB" rate=143.1KB/s time-left=0hrs:0m total-time=6m40s download-rate=143.1KB/s completion-rate=143.1KB/s alloc=1.8GB sys=6.3GB
Nov 20 22:43:40 enode-01 erigon[12466]: [INFO] [11-20|22:43:40.393] [1/6 OtterSync] Downloading              progress="(284/284 files) 99.96% - 10.4GB/10.4GB" rate=17.9KB/s time-left=0hrs:3m total-time=7m0s download-rate=17.9KB/s completion-rate=17.9KB/s alloc=3.0GB sys=6.3GB
Nov 20 22:43:52 enode-01 erigon[12466]: [INFO] [11-20|22:43:52.758] P2P                                      app=caplin peers=10
Nov 20 22:44:00 enode-01 erigon[12466]: [INFO] [11-20|22:44:00.391] [1/6 OtterSync] Downloading              progress="(284/284 files) 99.96% - 10.4GB/10.4GB" rate=2.2KB/s time-left=0hrs:30m total-time=7m20s download-rate=2.2KB/s completion-rate=2.2KB/s alloc=2.1GB sys=6.9GB
Nov 20 22:44:20 enode-01 erigon[12466]: [INFO] [11-20|22:44:20.391] [1/6 OtterSync] Downloading              progress="(284/284 files) 99.96% - 10.4GB/10.4GB" rate=286B/s time-left=4hrs:4m total-time=7m40s download-rate=286B/s completion-rate=286B/s alloc=3.0GB sys=6.9GB
Nov 20 22:44:40 enode-01 erigon[12466]: [INFO] [11-20|22:44:40.392] [1/6 OtterSync] Downloading              progress="(284/284 files) 99.96% - 10.4GB/10.4GB" rate=17B/s time-left=68hrs:32m total-time=8m0s download-rate=17B/s completion-rate=17B/s alloc=2.2GB sys=6.9GB
Nov 20 22:44:52 enode-01 erigon[12466]: [INFO] [11-20|22:44:52.758] P2P                                      app=caplin peers=10
Nov 20 22:45:00 enode-01 erigon[12466]: [INFO] [11-20|22:45:00.391] [1/6 OtterSync] Downloading              progress="(284/284 files) 99.96% - 10.4GB/10.4GB" rate=1B/s time-left=1165hrs:5m total-time=8m20s download-rate=1B/s completion-rate=1B/s alloc=2.9GB sys=6.9GB
Nov 20 22:45:20 enode-01 erigon[12466]: [INFO] [11-20|22:45:20.391] [1/6 OtterSync] Downloading              progress="(284/284 files) 99.96% - 10.4GB/10.4GB" rate=0B/s time-left=999hrs:99m total-time=8m40s download-rate=0B/s completion-rate=0B/s alloc=2.8GB sys=7.7GB
Nov 20 22:45:40 enode-01 erigon[12466]: [INFO] [11-20|22:45:40.391] [1/6 OtterSync] Downloading              progress="(284/284 files) 99.96% - 10.4GB/10.4GB" rate=0B/s time-left=999hrs:99m total-time=9m0s download-rate=0B/s completion-rate=0B/s alloc=2.2GB sys=7.7GB
Nov 20 22:45:40 enode-01 erigon[12466]: [INFO] [11-20|22:45:40.391] [p2p] GoodPeers                          eth68=13 eth67=5
Nov 20 22:45:40 enode-01 erigon[12466]: [INFO] [11-20|22:45:40.621] [mem] memory stats                       Rss=15.0GB Size=0B Pss=15.0GB SharedClean=2.3MB SharedDirty=0B PrivateClean=10.8GB PrivateDirty=4.2GB Referenced=15.0GB Anonymous=4.2GB Swap=0B alloc=2.2GB sys=7.7GB
Nov 20 22:45:52 enode-01 erigon[12466]: [INFO] [11-20|22:45:52.759] P2P                                      app=caplin peers=14
Nov 20 22:46:00 enode-01 erigon[12466]: [INFO] [11-20|22:46:00.391] [1/6 OtterSync] Downloading              progress="(284/284 files) 99.96% - 10.4GB/10.4GB" rate=0B/s time-left=999hrs:99m total-time=9m20s download-rate=0B/s completion-rate=0B/s alloc=2.6GB sys=7.7GB
Nov 20 22:46:20 enode-01 erigon[12466]: [INFO] [11-20|22:46:20.391] [1/6 OtterSync] Downloading              progress="(284/284 files) 99.96% - 10.4GB/10.4GB" rate=0B/s time-left=999hrs:99m total-time=9m40s download-rate=0B/s completion-rate=0B/s alloc=3.3GB sys=7.7GB

But, this resolved within a few minutes after OtterSync requested downloads again.
But it ran for approx. ~5 minutes at 0B/s at 99.96% for the 10.4 GB package