Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

If all mining stop for few hours, network peers will colapse? #1890

Closed
rrguardo opened this issue Feb 1, 2018 · 18 comments
Closed

If all mining stop for few hours, network peers will colapse? #1890

rrguardo opened this issue Feb 1, 2018 · 18 comments

Comments

@rrguardo
Copy link

rrguardo commented Feb 1, 2018

I was experiment with an independent private DASH copy network(fork), and note that after ~12 hour not mining, all the nodes get stuck synchronizing.

Is DASH daemon not prepare to work under that non realistic conditions?

Maybe I miss something, I am new with the related tech.

@codablock
Copy link

Can you show some logs of the stuck nodes?

@rrguardo
Copy link
Author

rrguardo commented Feb 1, 2018

I remember that qt wallet show all the time stuck, that was in sync process since the last generated block.
An for that reason, I am unable to continue mining.

Give me some hours to provide logs.

@codablock
Copy link

Did it show that block sync was stuck or MN sync? Was the wallet freshly started or did it run for some time and then suddenly showed a stuck sync? Did you continue mining when you've seen that and if yes, did the Qt wallet get synced?

@rrguardo
Copy link
Author

rrguardo commented Feb 1, 2018

I was mining fine for ~2 days, all the network was fine until I stop mining for ~12 hours or a little more.
After 12 hours without mining network looks stuck, daemons get forever in sync status like assuming that should exist blocks from past hours not synchronized.

In my setup network don't have yet masternodes online, but it worked in that way until mining stop.
I tried using no qt daemon and deleting all file minus wallet and new nodes are stuck specting sync blocks that don't exist.

In some hours will provide logs and more info.

@rrguardo
Copy link
Author

rrguardo commented Feb 1, 2018

Logs before the problem, at one node

2018-01-30 12:27:47 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0
2018-01-30 12:28:47 CMasternodeMan::CheckAndRemove
2018-01-30 12:28:47 CMasternodeMan::CheckAndRemove -- Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-01-30 12:28:47 CMasternodePayments::CheckAndRemove -- Votes: 0, Blocks: 0
2018-01-30 12:28:47 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0
2018-01-30 12:29:19 UpdateTip: new best=00000018303613506594fd2e2e5997f821eb392c05f4400d756be8fc920a03d0 height=1383 log2_work=34.030093 tx=1388 date=2018-01-30 12:29:18 progress=0.999992 cache=0.1MiB(341txo)
2018-01-30 12:29:19 ProcessNewBlock : ACCEPTED
2018-01-30 12:29:47 CMasternodeMan::CheckAndRemove
2018-01-30 12:29:47 CMasternodeMan::CheckAndRemove -- Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-01-30 12:29:47 CMasternodePayments::CheckAndRemove -- Votes: 0, Blocks: 0
2018-01-30 12:29:47 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0
2018-01-30 12:29:47 CGovernanceManager::UpdateCachesAndClean -- Governance Objects: 0 (Proposals: 0, Triggers: 0, Watchdogs: 0/0, Other: 0; Erased: 0), Votes: 0
2018-01-30 12:30:47 CMasternodeMan::CheckAndRemove
2018-01-30 12:30:47 CMasternodeMan::CheckAndRemove -- Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-01-30 12:30:47 CMasternodePayments::CheckAndRemove -- Votes: 0, Blocks: 0
2018-01-30 12:30:47 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0
2018-01-30 12:31:47 CMasternodeMan::CheckAndRemove

@rrguardo
Copy link
Author

rrguardo commented Feb 1, 2018

In the hours with zero mining we see this logs at one node:

2018-01-31 18:36:08 CMasternodeMan::CheckAndRemove -- Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-01-31 18:36:08 CMasternodePayments::CheckAndRemove -- Votes: 0, Blocks: 0
2018-01-31 18:36:08 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0
2018-01-31 18:37:08 CMasternodeMan::CheckAndRemove
2018-01-31 18:37:08 CMasternodeMan::CheckAndRemove -- Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-01-31 18:37:08 CMasternodePayments::CheckAndRemove -- Votes: 0, Blocks: 0
2018-01-31 18:37:08 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0
2018-01-31 18:38:08 CMasternodeMan::CheckAndRemove
2018-01-31 18:38:08 CMasternodeMan::CheckAndRemove -- Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-01-31 18:38:08 CMasternodePayments::CheckAndRemove -- Votes: 0, Blocks: 0
2018-01-31 18:38:08 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0
2018-01-31 18:39:08 CMasternodeMan::CheckAndRemove
2018-01-31 18:39:08 CMasternodeMan::CheckAndRemove -- Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-01-31 18:39:08 CMasternodePayments::CheckAndRemove -- Votes: 0, Blocks: 0
2018-01-31 18:39:08 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0

@rrguardo
Copy link
Author

rrguardo commented Feb 1, 2018

After some hours without mining the logs change to this at nodes

2018-01-31 21:54:20 ThreadSocketHandler -- removing node: peer=428 addr=ip2:34706 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:20 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=429
2018-01-31 21:54:20 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:20 ThreadSocketHandler -- removing node: peer=429 addr=ip2:34708 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:21 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=430
2018-01-31 21:54:21 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:21 ThreadSocketHandler -- removing node: peer=430 addr=ip2:34710 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:22 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=431
2018-01-31 21:54:22 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:22 ThreadSocketHandler -- removing node: peer=431 addr=ip2:34712 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:22 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=432
2018-01-31 21:54:22 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:22 ThreadSocketHandler -- removing node: peer=432 addr=ip2:34714 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:23 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=433
2018-01-31 21:54:23 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:23 ThreadSocketHandler -- removing node: peer=433 addr=ip2:34716 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:23 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=434
2018-01-31 21:54:23 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:23 ThreadSocketHandler -- removing node: peer=434 addr=ip2:34718 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:24 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=435
2018-01-31 21:54:24 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:24 ThreadSocketHandler -- removing node: peer=435 addr=ip2:34720 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:24 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=436
2018-01-31 21:54:25 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:25 ThreadSocketHandler -- removing node: peer=436 addr=ip2:34722 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:25 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=437
2018-01-31 21:54:25 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:25 ThreadSocketHandler -- removing node: peer=437 addr=ip2:34724 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:26 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=438
2018-01-31 21:54:26 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:26 ThreadSocketHandler -- removing node: peer=438 addr=ip2:34726 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:26 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=439
2018-01-31 21:54:26 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:26 ThreadSocketHandler -- removing node: peer=439 addr=ip2:34728 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:27 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=440
2018-01-31 21:54:27 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:27 ThreadSocketHandler -- removing node: peer=440 addr=ip2:34730 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:27 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=441
2018-01-31 21:54:27 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:28 ThreadSocketHandler -- removing node: peer=441 addr=ip2:34732 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:28 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=442

real ip port replaced

@rrguardo
Copy link
Author

rrguardo commented Feb 1, 2018

Network looks stuck at block=1401
Maybe was something that only affect my custom network, I change some parameters at chainparams file.

@rrguardo
Copy link
Author

rrguardo commented Feb 2, 2018

New nodes in the network get stuck in a loop with this log repeated >>

2018-02-02 01:15:23 trying connection 219.25.22.23:9999 lastseen=0.1hrs
2018-02-02 01:15:24 Added connection peer=461
2018-02-02 01:15:24 sending version (107 bytes) peer=461
2018-02-02 01:15:24 send version message: version 70208, blocks=1, us=[::]:0, peer=461
2018-02-02 01:15:24 received: version (107 bytes) peer=461
2018-02-02 01:15:24 sending verack (0 bytes) peer=461
2018-02-02 01:15:24 sending getaddr (0 bytes) peer=461
2018-02-02 01:15:24 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=145.102.197.218:34030, peer=461
2018-02-02 01:15:24 received: verack (0 bytes) peer=461
2018-02-02 01:15:24 sending sendheaders (0 bytes) peer=461
2018-02-02 01:15:24 sending ping (8 bytes) peer=461
2018-02-02 01:15:24 initial getheaders (1400) to peer=461 (startheight:1401)
2018-02-02 01:15:24 sending getheaders (741 bytes) peer=461
2018-02-02 01:15:24 received: sendheaders (0 bytes) peer=461
2018-02-02 01:15:24 received: ping (8 bytes) peer=461
2018-02-02 01:15:24 sending pong (8 bytes) peer=461
2018-02-02 01:15:24 received: addr (31 bytes) peer=461
2018-02-02 01:15:24 received: pong (8 bytes) peer=461
2018-02-02 01:15:24 received: headers (82 bytes) peer=461
2018-02-02 01:15:24 ERROR: detected bad peer for initial headers sync, disconnecting 461
2018-02-02 01:15:24 ProcessMessages(headers, 82 bytes) FAILED peer=461
2018-02-02 01:15:24 ThreadSocketHandler -- removing node: peer=461 addr=219.25.22.23:9999 nRefCount=1 fNetworkNode=1 fInbound=0 fMasternode=0
2018-02-02 01:15:24 disconnecting peer=461
2018-02-02 01:15:24 trying connection 219.25.22.23:9999 lastseen=0.1hrs

@rrguardo
Copy link
Author

rrguardo commented Feb 2, 2018

Main reason looks >>
ERROR: detected bad peer for initial headers sync, disconnecting

@rrguardo
Copy link
Author

rrguardo commented Feb 2, 2018

header is detected as invalid from all peers, will be reviewing the related code

@rrguardo
Copy link
Author

rrguardo commented Feb 2, 2018

Will try now, set one node without internet and put date before the problem(~block 1401 date) at the related computer.
If all work fine again, just will reset all other nodes to sync from the one fixed.
Additionally will try to never stop mining, previous steps are possible only because my fork network is not alive yet.

@codablock
Copy link

Hmm not sure which behavior you expected, but what you observe seems to be normal. The "ERROR: detected bad peer for initial headers sync, disconnecting" you see means that it connected to a peer which did not provide recent enough HEADERs to be considered as up-to-date, which is to be expected after some time if there is no mining peer in the network.

It should however heal itself when a peer starts to do mining again. Were you able to restart mining after the errors appeared? Or is it exactly the mining node that struggles now?

@rrguardo
Copy link
Author

rrguardo commented Feb 2, 2018

I am not able to continue mining, mining node get stuck connecting and disconnecting from all accessible peers.

I was able to start mining in one node after setup old date on node server.
Is possible that mining node have some error or problem, will be reviewing, thanks

@rrguardo
Copy link
Author

rrguardo commented Feb 5, 2018

My network is working again following this steps:

  • setup mining node time to 5 hours before last block time
  • after generate a new block set time +5 hours until reach current date time
  • reset all main nodes to get synchronized from zero using only the updated miner node or updated nodes in the synchronization.
  • setup some main nodes as miner nodes to avoid repeat the problem.

I imagine that should exist other more easy way to do the same thing, unknown from me.

@charlesrocket
Copy link

@rrguardo i guess you could change the time network can be abandoned. but this might break it in a long run

@nmarley
Copy link

nmarley commented Mar 25, 2018

Appears to be issue with local date/time settings but difficult to re-produce without more information. Closing as OP was able to find a fix.

@osnwt
Copy link

osnwt commented May 22, 2018

It's old issue, but just for reference. As far as I see, if current time is more than 6 hours more than last block mined, just started node will never sync. It is by design (or, say, by code). If node is running and no new blocks mined for a long time, this node will continue running. But if you restart it, it will always wait for more recent block (not older than 6 hours).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants