Synchronization around 3 minutes behind head for two days

[manelio]

After some problems I decided going with a new fast sync from scratch.

For two days the block's age was continuously reduced until reaching a value of around 3 minutes. But after two more days this value is stuck at around same value.

My hardware is a Raspberry Pi 4 (8 GB) with an USB3 SSD (write speed > 300 MB/s). CPU load remains below 1 and there's no heavy IO activity.

This is an extract of the log (filtered to show only Pivot became stale, moving and lines with age information):

ubuntu@bscnode:~/node$ cat node/bsc.log.* | grep "age=\|stale"

t=2021-04-15T20:48:10+0000 lvl=warn msg="Pivot became stale, moving"          old=6597024 new=6597089
t=2021-04-15T20:48:10+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=483.840ms number=6597088 hash=0x6a892b89412b6156a9276ce5f2e1da27380a295b4ec629df1e486e4f20f942db age=3m24s     size="8.14 MiB"
t=2021-04-15T20:51:24+0000 lvl=warn msg="Pivot became stale, moving"          old=6597089 new=6597154
t=2021-04-15T20:51:25+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=449.962ms number=6597153 hash=0x782dc6a926e3f99d7dd6f1e480f5273ae40113f7fff5ddfdabcc3a11216eea71 age=3m24s     size="7.67 MiB"
t=2021-04-15T20:54:39+0000 lvl=warn msg="Pivot became stale, moving"          old=6597154 new=6597219
t=2021-04-15T20:54:39+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=383.839ms number=6597218 hash=0x8b98fd916f774ef0702563220d8aa2e9ac0992d5226b25e45fad24b158c3231a age=3m23s     size="6.70 MiB"
t=2021-04-15T20:57:56+0000 lvl=warn msg="Pivot became stale, moving"          old=6597219 new=6597284
t=2021-04-15T20:57:57+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=451.079ms number=6597283 hash=0xc668862ecfe538e47a1300e6ea93cb6b4ed27fcb0668afe198e8d60823a69a8b age=3m25s     size="7.58 MiB"
t=2021-04-15T21:01:11+0000 lvl=warn msg="Pivot became stale, moving"          old=6597284 new=6597349
t=2021-04-15T21:01:11+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=417.851ms number=6597348 hash=0xd0f1909af2448b683cdb7a9c2293094b2efc7c186fd839e87a2643bf5d5ba345 age=3m24s     size="7.30 MiB"
t=2021-04-15T21:04:25+0000 lvl=warn msg="Pivot became stale, moving"          old=6597349 new=6597414
t=2021-04-15T21:04:26+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=507.054ms number=6597413 hash=0x1e5028ffddd807de6cb5caf923bd8f65ddf99c9776addc2243fcdcf775fe6a57 age=3m24s     size="8.35 MiB"
t=2021-04-15T21:07:40+0000 lvl=warn msg="Pivot became stale, moving"          old=6597414 new=6597479
t=2021-04-15T21:07:41+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=540.557ms number=6597478 hash=0xdd5bde6dc1fcaa0ce502a418fc777bf2643701655f4aa20cdbdffbad8e17b25b age=3m24s     size="9.25 MiB"
t=2021-04-15T21:10:57+0000 lvl=warn msg="Pivot became stale, moving"          old=6597479 new=6597544
t=2021-04-15T21:10:57+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=760.996ms number=6597543 hash=0x3814bf912ecb85b114ec718f6b3b0b496691a57dbdc575732fca80e011c7ed3b age=3m25s     size="12.40 MiB"
t=2021-04-15T21:14:11+0000 lvl=warn msg="Pivot became stale, moving"          old=6597544 new=6597609
t=2021-04-15T21:14:11+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=480.340ms number=6597608 hash=0x808047c3d214c9e4b2a8e632d10e34e9eabd56c3d75f059653f9aabb21caa465 age=3m24s     size="8.43 MiB"
t=2021-04-15T21:17:25+0000 lvl=warn msg="Pivot became stale, moving"          old=6597609 new=6597674
t=2021-04-15T21:17:26+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=484.089ms number=6597673 hash=0xef939da2af5e343deea9ae696a91da5d8ac0a057706d3179e33741bcb2bced34 age=3m24s     size="8.50 MiB"
t=2021-04-15T21:20:40+0000 lvl=warn msg="Pivot became stale, moving"          old=6597674 new=6597739
t=2021-04-15T21:20:40+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=567.109ms number=6597738 hash=0x2d99d7de6f72ee482a9318a19b2279a1f474a3b992ed286e792b0cc4fc8a041d age=3m23s     size="9.65 MiB"
t=2021-04-15T21:23:56+0000 lvl=warn msg="Pivot became stale, moving"          old=6597739 new=6597804
t=2021-04-15T21:23:57+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=514.676ms number=6597803 hash=0x29eff223169a98c9db630a96dd9747781cbbb47d2fdb7e314d845a2bfd76d91b age=3m25s     size="8.92 MiB"
t=2021-04-15T21:27:10+0000 lvl=warn msg="Pivot became stale, moving"          old=6597804 new=6597869
t=2021-04-15T21:27:11+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=598.589ms number=6597868 hash=0x208cca9f77e0e796eff8eb8aeb17250377bf9b4eae3123ec5b2f600c865db0c3 age=3m24s     size="9.89 MiB"
t=2021-04-15T21:30:28+0000 lvl=warn msg="Pivot became stale, moving"          old=6597869 new=6597935
t=2021-04-15T21:30:28+0000 lvl=info msg="Imported new block receipts"         count=66   elapsed=625.484ms number=6597934 hash=0xb448b679fca13b4ad890a511d02d4030b95299720a908f679bb75b16316988c2 age=3m23s     size="10.32 MiB"
t=2021-04-15T21:33:42+0000 lvl=warn msg="Pivot became stale, moving"          old=6597935 new=6598000
t=2021-04-15T21:33:43+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=550.659ms number=6597999 hash=0x46bb81669f7d4dbe397a1562fd68c0726fbf5f30563a3a3ecbdb0ef9dc41bfe7 age=3m23s     size="9.13 MiB"
t=2021-04-15T21:36:59+0000 lvl=warn msg="Pivot became stale, moving"          old=6598000 new=6598065
t=2021-04-15T21:37:00+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=766.419ms number=6598064 hash=0xd6e300417bd8543bb991ae99a1de3a1ecf829bcc44aea41bfa1488ac8f5f0497 age=3m25s     size="12.09 MiB"
t=2021-04-15T21:40:13+0000 lvl=warn msg="Pivot became stale, moving"          old=6598065 new=6598130
t=2021-04-15T21:40:13+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=538.969ms number=6598129 hash=0xc8b313e65b7d8c62e4096b7ebb70d43ec1708c9e8f37226ec35086088a52c9d1 age=3m23s     size="8.93 MiB"
t=2021-04-15T21:43:28+0000 lvl=warn msg="Pivot became stale, moving"          old=6598130 new=6598195
t=2021-04-15T21:43:29+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=1.020s    number=6598194 hash=0x6d2f6019050423a1fe57270bef1e2543c850ec18a42b04b2086a73fa21cde0ae age=3m24s     size="12.55 MiB"
t=2021-04-15T21:46:44+0000 lvl=warn msg="Pivot became stale, moving"          old=6598195 new=6598260
t=2021-04-15T21:46:44+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=505.952ms number=6598259 hash=0xfc669cb715b18d685fa4a4a83db6ee5b544c2db45edaa3e08dec9cd0694a5bfd age=3m24s     size="9.70 MiB"
t=2021-04-15T21:49:57+0000 lvl=warn msg="Pivot became stale, moving"          old=6598260 new=6598325
t=2021-04-15T21:49:58+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=616.449ms number=6598324 hash=0x12eb26a4b22d42b52209ca76384cba8d5f4f1796d3373eacb7d74c31187c0cc7 age=3m23s     size="11.33 MiB"
t=2021-04-15T21:53:14+0000 lvl=warn msg="Pivot became stale, moving"          old=6598325 new=6598390
t=2021-04-15T21:53:14+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=440.568ms number=6598389 hash=0x1e59574408c38e39811458316b33a0a1cb89660696bd483f1690c36d34567a96 age=3m24s     size="8.39 MiB"
t=2021-04-15T21:56:29+0000 lvl=warn msg="Pivot became stale, moving"          old=6598390 new=6598455
t=2021-04-15T21:56:30+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=472.210ms number=6598454 hash=0x71feeb7be15a8d5a9d22438abc1a086036242fec96ca62d14f1b7c377b532b32 age=3m25s     size="8.71 MiB"
t=2021-04-15T21:59:44+0000 lvl=warn msg="Pivot became stale, moving"          old=6598455 new=6598520
t=2021-04-15T21:59:45+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=574.843ms number=6598519 hash=0x51040427981cde12bf28dd9b4d6031dd16eff218e09a784908b027e9f34662a8 age=3m25s     size="10.05 MiB"
t=2021-04-15T22:02:58+0000 lvl=warn msg="Pivot became stale, moving"          old=6598520 new=6598585
t=2021-04-15T22:02:58+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=557.179ms number=6598584 hash=0x5738f2c9e8b08268609b5ed357c36aaa021c5c95eefb21ef89eef568c6f8664a age=3m23s     size="9.70 MiB"
t=2021-04-15T22:06:12+0000 lvl=warn msg="Pivot became stale, moving"          old=6598585 new=6598650
t=2021-04-15T22:06:13+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=510.883ms number=6598649 hash=0xb034015844e74c8e5948cb72ed3280ea7a37c94da6ec0937a2e100c627621c6c age=3m23s     size="9.10 MiB"
t=2021-04-15T22:09:29+0000 lvl=warn msg="Pivot became stale, moving"          old=6598650 new=6598715
t=2021-04-15T22:09:29+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=473.118ms number=6598714 hash=0x8a524a26195677040b2d2ea92b46ffdaf0798adfae91a533d15b60a74a63d95f age=3m24s     size="8.30 MiB"
t=2021-04-15T22:12:42+0000 lvl=warn msg="Pivot became stale, moving"          old=6598715 new=6598780
t=2021-04-15T22:12:43+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=428.575ms number=6598779 hash=0x93ff521d0f8705b04ead8fa77bffe7837391e41aba5d87c483f4af263b56edb9 age=3m23s     size="7.63 MiB"
t=2021-04-15T22:15:57+0000 lvl=warn msg="Pivot became stale, moving"          old=6598780 new=6598845
t=2021-04-15T22:15:57+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=395.780ms number=6598844 hash=0x06f6fa1cd819319cb6bb908a3b94f77c78b7d53c7bfde94bd6cdb2cfc00959fc age=3m22s     size="7.11 MiB"
t=2021-04-15T22:19:14+0000 lvl=warn msg="Pivot became stale, moving"          old=6598845 new=6598910
t=2021-04-15T22:19:15+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=440.296ms number=6598909 hash=0x7992a0b29ca95872fee4cf943787ee0fa9b2fe6ecd5dcd3325cc46c468e3d5db age=3m25s     size="8.00 MiB"
t=2021-04-15T22:22:28+0000 lvl=warn msg="Pivot became stale, moving"          old=6598910 new=6598975
t=2021-04-15T22:22:28+0000 lvl=info msg="Imported new block receipts"         count=65   elapsed=515.666ms number=6598974 hash=0xffa029d44723ecaed75981c188eb4f652ad4956d7a54c98ab696015704b4de70 age=3m23s     size="9.11 MiB"
t=2021-04-15T22:25:49+0000 lvl=warn msg="Pivot became stale, moving"          old=6598975 new=6599042
t=2021-04-15T22:25:49+0000 lvl=info msg="Imported new block receipts"         count=67   elapsed=414.338ms number=6599041 hash=0x2d8f35210642f0ba74e65c820fb47566ead6276b733f69d39b69437752628423 age=3m23s     size="7.57 MiB"

My only concern is if this will eventually get synced or if this will be forever behind the head.

I think there's a lack of metrics tools or diagnostics information to get an idea of the ETA, global status and even the viability of the process.

[unclezoro]

https://s3.ap-northeast-1.amazonaws.com/dex-bin.bnbstatic.com/s3-witness-data-download/chaindata-2021-04-15.zip?AWSAccessKeyId=AKIAYINE6SBQPUZDDRRO&Expires=1644459350&Signature=a4vfmga8%2BRwNZl3boisIMEdSWbA%3D

Tyr the latest snapshot!

[manelio]

Thanks!

But after a few hours, I get a previous error again:

INFO [04-17|22:19:14.525] Imported new chain segment               blocks=4  txs=414  mgas=54.624  elapsed=8.562s    mgasps=6.380  number=6601178 hash=dd3cb0…0e0b87 age=1d22h9m  dirty=649.90MiB                                                                                         
INFO [04-17|22:19:51.970] Imported new chain segment               blocks=5  txs=389  mgas=62.637  elapsed=37.444s   mgasps=1.673  number=6601183 hash=d31b72…e1602b age=1d22h9m  dirty=649.90MiB                                                                                         
INFO [04-17|22:20:01.055] Imported new chain segment               blocks=6  txs=667  mgas=90.295  elapsed=9.085s    mgasps=9.939  number=6601189 hash=e47c51…069143 age=1d22h9m  dirty=649.90MiB                                                                                         
INFO [04-17|22:20:12.472] Imported new chain segment               blocks=6  txs=490  mgas=81.134  elapsed=11.416s   mgasps=7.106  number=6601195 hash=9f43bc…653827 age=1d22h9m  dirty=649.90MiB                                                                                         
INFO [04-17|22:20:14.574] Deep froze chain segment                 blocks=20 elapsed=3.528s    number=6511193 hash=5059c4…d75fce                                                                                                                                                          
INFO [04-17|22:20:20.923] Imported new chain segment               blocks=4  txs=427  mgas=52.893  elapsed=8.450s    mgasps=6.259  number=6601199 hash=d7e403…29e75b age=1d22h9m  dirty=649.90MiB                                                                                         
panic: expected tx hash 0x102a66ae23a4b0e17dfc25eb6e2ddbc83163bc67840290e807681bd26e07699f, get 0x87deef7e22fc594f6bae3640995a98fd2f2be2d30ccf1b763667cda940173d43                                                                                                                        
                                                                                                                                                                                                                                                                                          
goroutine 818 [running]:                                                                                                                                                                                                                                                                  
github.com/ethereum/go-ethereum/consensus/parlia.(*Parlia).Finalize(0x4000cd0380, 0x18fa820, 0x400566e000, 0x40897fe900, 0x40fdae0160, 0x40de7b94e0, 0x258d258, 0x0, 0x0, 0x40de7b93e0, ...)                                                                                              
        /go/src/github.com/binance-chain/bsc/consensus/parlia/parlia.go:707 +0xd7c                                                                                                                                                                                                        
github.com/ethereum/go-ethereum/core.(*StateProcessor).Process(0x400016d160, 0x4089ce2510, 0x40fdae0160, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)                                                                                                                                          

Too many problems. I'll just look for alternative clients.

Thanks.

[gauss345]

I have the same issue( as per the initial post in this thread), and node not sync'd now

[soltrac]

Some time ago was possible to sync with these specs, but right now, BSC has many txs, so it is not possible. You need to upgrade your hardware.

[ChristBKK]

Some time ago was possible to sync with these specs, but right now, BSC has many txs, so it is not possible. You need to upgrade your hardware.

what hardware can you suggest as minimum?

[manelio]

I'm about to order the following:

• MiniPC barebones, i3 processor, with 16G RAM
• 1TB SDD NVMe

My estimation is an expense of around $500. But that's just a try, not based on any metric. I have no enough information.

[gauss345]

Our specs meet the minimum stated requirements for hardware and we restarted the node with --cache , but still it takes a while to sync up....

Is there any information I can provide, that might help diagnose the problem better ?

currently I see WARN messages in the logs,

WARN [04-21|04:59:43.934] Pivot became stale, moving old=6750213 new=6750278
WARN [04-21|05:03:01.493] Pivot became stale, moving old=6750278 new=6750344
WARN [04-21|05:06:21.020] Pivot became stale, moving old=6750344 new=6750410
WARN [04-21|05:09:34.568] Pivot became stale, moving old=6750410 new=6750475
WARN [04-21|05:13:02.045] Pivot became stale, moving old=6750475 new=6750542
WARN [04-21|05:13:08.542] Stalling state sync, dropping peer peer=cb63bf89de39efea
WARN [04-21|05:16:13.623] Pivot became stale, moving old=6750542 new=6750608
WARN [04-21|05:19:33.157] Pivot became stale, moving old=6750608 new=6750674
WARN [04-21|05:22:48.617] Pivot became stale, moving old=6750674 new=6750739
WARN [04-21|05:26:07.062] Pivot became stale, moving old=6750739 new=6750805

eth.syncing
{
currentBlock: 6750804,
highestBlock: 6750903,
knownStates: 169369973,
pulledStates: 169369972,
startingBlock: 6731430
}

and the state of sync, has been 100 blocks or so, behind for almost a week now.

The size on disk is 102GB, and it seems to sync, about 1 or 2 GB ( if that ) per day.
Using the latest config (1.0.7)

[AlexWD]

Seeing this issue as well.

[chainwhisper]

@AlexWD please download the latest config.toml with new bootstrap nodes and restart

[gauss345]

@AlexWD please download the latest config.toml with new bootstrap nodes and restart

which is the latest 'config.toml' ? is it not 1.0.7 ?

[havsar]

Tried with newest config still the issue occurs.

[muyinliu]

Please do use native NVMe SSD insteads of SATA SSD(or EBS on AWS), or it will never complete the sync.

[j75689]

Hi all.
Thank you for your report.
We have received many report of a sync issue.
You can try the latest version. If you have any probleums, please feedback #338.
We will pay attention to the issue #338 for a long time, and if there are any updates, we will explain it on this issue.

Thanks.