Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warning Msgs 'Large Hash rate changes' in concert with sending uptime proofs not being received can cause node to deregister #822

Closed
anon1972 opened this issue Sep 4, 2019 · 8 comments

Comments

@anon1972
Copy link

anon1972 commented Sep 4, 2019

Statement:

This Node had been running 8 days + without any problem. Just as a FYI... I have been running loki nodes since Loki launch and follow detailed instructions per guidance at loki.network. Last night I received a warning from LokiSNbot stating 'WARNING: Service node 3fe211…ef4 last uptime proof is 1h06m08s ago'. I quickly went to review the status of the node and noticed numerous logs indicating 'WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 0 blocks in the last 20 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.'

These logs were also intermixed showing that I had successfully submitted uptime proofs w/o reciept. Here is the example. '019-09-04 03:48:47.138 [P2P9] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours)'.

None the less, LokiSNbot continued to complain until '1h36m20s'. Below within the logs you will see a capture showing successful uptime proofs being sent, and then the abnormality starting at '2019-09-04 03:44:49.066'. You will also see that I rebooted at '2019-09-04 03:55:19.240'. The problem still continued to occur, but did eventually resolve at '2019-09-04 04:20:31.895'. You will notice that the logs are vague but do show that the node was running at optimum before the abnormality occurred.

A few things... I noticed in the Telegram chat that another individual reported this same problem on 4.0.3, but the issue seemed to have been downplayed due to the version. You will also see that that I am running at current over the spec guide lines for the node. Lastly, as of this morning, I noticed chatter to problems last night with nodes having problems.

Speculation... with the Node numbers increasing, could there be impact as certain checks are now being implemented?

Note: This should be considered at 'Medium Severity due the the potential impact of being de-registered if the dependency of meeting the uptime-proof criteria can not occur. I don't see this as an operator issue but rather a bug. In general, Loki logs also are not descriptive enough.

Thank you Loki Team!

//Software/Hardware version Info

//Loki-Launcher version 1.0.0

//Loki version 'Hefty Heimdall' (v4.0.5-e6520522b)

//OS Version

enoch@lokisnode3:~$ uname -r
5.0.0-25-generic

enoch@lokisnode3:$ cat /etc/os-release
NAME="Ubuntu"
VERSION="19.04 (Disco Dingo)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 19.04"
VERSION_ID="19.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=disco
UBUNTU_CODENAME=disco
enoch@lokisnode3:
$

//Kernel Version

enoch@lokisnode3:$ uname -a
Linux lokisnode3 5.0.0-25-generic #26-Ubuntu SMP Thu Aug 1 12:04:58 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
enoch@lokisnode3:
$

//VPS Hardware Information:

enoch@lokisnode3:~$ sudo lshw -short
[sudo] password for snode3:
H/W path Device Class Description

                        system     Standard PC (i440FX + PIIX, 1996)

/0 bus Motherboard
/0/0 memory 96KiB BIOS
/0/400 processor Virtual CPU 82d9ed4018dd
/0/1000 memory 4GiB System Memory
/0/1000/0 memory 4GiB DIMM RAM
/0/100 bridge 440FX - 82441FX PMC [Natoma]
/0/100/1 bridge 82371SB PIIX3 ISA [Natoma/Triton II]
/0/100/1.1 storage 82371SB PIIX3 IDE [Natoma/Triton II]
/0/100/1.2 bus 82371SB PIIX3 USB [Natoma/Triton II]
/0/100/1.2/1 usb1 bus UHCI Host Controller
/0/100/1.2/1/1 input QEMU USB Tablet
/0/100/1.3 bridge 82371AB/EB/MB PIIX4 ACPI
/0/100/2 display GD 5446
/0/100/3 network Virtio network device
/0/100/3/0 ens3 network Ethernet interface
/0/100/4 storage Virtio block device
/0/100/4/0 /dev/vda disk 85GB Virtual I/O device
/0/100/4/0/1 /dev/vda1 volume 79GiB EXT4 volume
/0/100/5 generic Virtio memory balloon
/0/100/5/0 generic Virtual I/O device
/0/100/6 generic Virtio RNG
/0/100/6/0 generic Virtual I/O device
/0/1 scsi1 storage
/0/1/0.0.0 /dev/cdrom disk QEMU DVD-ROM
enoch@lokisnode3:~$

//CPU Info

enoch@lokisnode3:~$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 40 bits physical, 48 bits virtual
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Virtual CPU 82d9ed4018dd
Stepping: 4
CPU MHz: 2593.906
BogoMIPS: 5187.81
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
L3 cache: 16384K
NUMA node0 CPU(s): 0,1
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq clwb avx512cd avx512bw avx512vl xsaveopt

//loki logs

//All is well...piece of cake

2019-09-03 21:38:39.712 [P2P4] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-03 22:38:41.195 [P2P0] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-03 23:38:42.629 [P2P7] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 00:38:43.728 [P2P6] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 01:38:44.777 [P2P5] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 02:38:45.848 [P2P5] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 03:38:46.975 [P2P5] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 03:43:47.050 [P2P7] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>

//Here we go!

2019-09-04 03:44:49.066 [P2P8] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 0 blocks in the last 20 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 03:46:19.102 [P2P2] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 0 blocks in the last 20 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 03:47:49.126 [P2P5] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 0 blocks in the last 20 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 03:48:47.138 [P2P9] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 03:49:19.146 [P2P6] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 0 blocks in the last 20 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 03:50:49.172 [P2P8] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 0 blocks in the last 20 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 03:52:19.192 [P2P9] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 0 blocks in the last 20 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 03:53:47.212 [P2P3] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 03:53:49.212 [P2P5] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 2 blocks in the last 30 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 03:55:19.240 [P2P9] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 0 blocks in the last 30 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.

//Note I rebooted here

2019-09-04 03:56:36.062 7fcd8b49c700 WARNING global src/cryptonote_protocol/cryptonote_protocol_handler.inl:2430 lokid is now disconnected from the network
2019-09-04 03:56:36.091 [SRV_MAIN] INFO global src/daemon/p2p.h:81 p2p net loop stopped
2019-09-04 03:56:36.894 [SRV_MAIN] INFO global src/daemon/rpc.h:84 Stopping core RPC server...
2019-09-04 03:56:36.895 [SRV_MAIN] INFO global src/daemon/daemon.cpp:210 Node stopped.
2019-09-04 03:56:36.897 [SRV_MAIN] INFO global src/daemon/rpc.h:96 Deinitializing core RPC server...
2019-09-04 03:56:36.902 [SRV_MAIN] INFO global src/daemon/p2p.h:91 Deinitializing p2p...
2019-09-04 03:56:36.905 [SRV_MAIN] INFO global src/daemon/core.h:95 Deinitializing core...
2019-09-04 03:56:37.096 [SRV_MAIN] INFO global src/daemon/protocol.h:76 Stopping cryptonote protocol...
2019-09-04 03:56:37.097 [SRV_MAIN] INFO global src/daemon/protocol.h:80 Cryptonote protocol stopped successfully
2019-09-04 03:58:28.966 7f768ee46b80 INFO logging contrib/epee/src/mlog.cpp:273 New log categories: *:WARNING,net:FATAL,net.http:FATAL,net.ssl:FATAL,net.p2p:FATAL,net.cn:FATAL,global:INFO,verify:FATAL,serialization:FATAL,logging:INFO,msgwriter:INFO
2019-09-04 03:58:28.983 7f768ee46b80 INFO global src/daemon/main.cpp:282 Loki 'Hefty Heimdall' (v4.0.5-e6520522b)
2019-09-04 03:58:28.987 7f768ee46b80 INFO global src/daemon/protocol.h:54 Initializing cryptonote protocol...
2019-09-04 03:58:28.987 7f768ee46b80 INFO global src/daemon/protocol.h:59 Cryptonote protocol initialized OK
2019-09-04 03:58:28.989 7f768ee46b80 INFO global src/daemon/p2p.h:64 Initializing p2p server...
2019-09-04 03:58:28.993 7f768ee46b80 INFO global src/daemon/p2p.h:69 p2p server initialized OK
2019-09-04 03:58:28.993 7f768ee46b80 INFO global src/daemon/rpc.h:63 Initializing core RPC server...
2019-09-04 03:58:28.994 7f768ee46b80 INFO global contrib/epee/include/net/http_server_impl_base.h:79 Binding on 127.0.0.1 (IPv4):22023
2019-09-04 03:58:28.995 7f768ee46b80 INFO global contrib/epee/src/net_ssl.cpp:124 Generating SSL certificate
2019-09-04 03:58:30.696 7f768ee46b80 INFO global src/daemon/rpc.h:69 core RPC server initialized OK on port: 22023
2019-09-04 03:58:30.696 7f768ee46b80 INFO global src/daemon/core.h:74 Initializing core...
2019-09-04 03:58:30.697 7f768ee46b80 INFO global src/cryptonote_core/cryptonote_core.cpp:398 Storage server endpoint is set to: 95.179.163.78:23023
2019-09-04 03:58:30.697 7f768ee46b80 INFO global src/cryptonote_core/cryptonote_core.cpp:833 Service node pubkey is 3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4
2019-09-04 03:58:30.698 7f768ee46b80 INFO global src/cryptonote_core/cryptonote_core.cpp:565 Loading blockchain from folder /home/snode3/.loki/lmdb ...
2019-09-04 03:58:30.698 7f768ee46b80 WARNING global src/blockchain_db/lmdb/db_lmdb.cpp:1323 The blockchain is on a rotating drive: this will be very slow, use an SSD if possible
2019-09-04 03:58:31.969 7f768ee46b80 INFO global src/cryptonote_core/service_node_list.cpp:1893 Service node data loaded successfully, height: 350898
2019-09-04 03:58:31.969 7f768ee46b80 INFO global src/cryptonote_core/service_node_list.cpp:1896 734 nodes and 1715 historical states loaded (7.45 MB)
2019-09-04 03:58:31.970 7f768ee46b80 INFO global src/cryptonote_core/service_node_list.cpp:78 Recalculating service nodes list, scanning blockchain from height: 350898 to: 351193
2019-09-04 03:58:32.519 7f768ee46b80 INFO global src/cryptonote_core/service_node_list.cpp:121 Done recalculating service nodes list (0.548s)
2019-09-04 03:58:32.521 7f768ee46b80 INFO global src/cryptonote_core/cryptonote_core.cpp:764 Loading checkpoints
2019-09-04 03:58:32.523 7f768ee46b80 INFO global src/daemon/core.h:84 Core initialized OK
2019-09-04 03:58:32.523 7f768ee46b80 INFO global src/daemon/rpc.h:74 Starting core RPC server...
2019-09-04 03:58:32.523 [SRV_MAIN] INFO global src/daemon/rpc.h:79 core RPC server started ok
2019-09-04 03:58:32.526 [SRV_MAIN] INFO global src/daemon/p2p.h:79 Starting p2p net loop...
2019-09-04 03:58:33.526 [P2P1] INFO global src/cryptonote_core/cryptonote_core.cpp:1885


The daemon will start synchronizing with the network. This may take a long time to complete.

You can set the level of process detailization through "set_log <level|categories>" command,
where is between 0 (no details) and 4 (very verbose), or custom category based levels (eg, *:WARNING).

Use the "help" command to see the list of available commands.
Use "help " to see a command's documentation.


2019-09-04 03:59:59.543 [P2P7] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 0 blocks in the last 30 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 04:00:31.553 [P2P3] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 04:01:29.564 [P2P3] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 0 blocks in the last 30 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 04:01:46.632 [P2P0] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:2094


You are now synchronized with the network. You may now start loki-wallet-cli.

Use the "help" command to see the list of available commands.


2019-09-04 04:02:59.581 [P2P4] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 11 blocks in the last 60 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 04:04:29.608 [P2P5] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 10 blocks in the last 60 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 04:05:31.623 [P2P1] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 04:05:59.637 [P2P9] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 10 blocks in the last 60 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 04:07:29.656 [P2P9] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 8 blocks in the last 60 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 04:08:59.673 [P2P9] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 8 blocks in the last 60 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 04:10:29.693 [P2P1] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 7 blocks in the last 60 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 04:10:31.693 [P2P3] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 04:11:59.710 [P2P7] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 7 blocks in the last 60 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 04:12:35.971 [P2P9] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:374 [104.248.189.245:22022 OUT] Sync data returned a new top block candidate: 351193 -> 351218 [Your node is 25 blocks (0 days) behind]
SYNCHRONIZATION started
2019-09-04 04:12:40.899 [P2P9] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:1415 Synced 351218/351218
2019-09-04 04:12:40.899 [P2P9] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:2072 SYNCHRONIZED OK
2019-09-04 04:15:31.808 [P2P8] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 04:20:31.895 [P2P7] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
enoch@lokisnode3:~/.loki$

@jagerman
Copy link
Member

jagerman commented Sep 4, 2019

Can you check lokid status and confirm that your height agrees with lokiblocks.com?

@jagerman
Copy link
Member

jagerman commented Sep 4, 2019

... the next time this occurs. Also lokid sync_info would be helpful to see how many peers you have.

@anon1972
Copy link
Author

anon1972 commented Sep 5, 2019

This appears to be a very serious problem. This has occurred again on the same node. From my observation, this will not resolve on its own once the condition is triggered. The node will need to be rebooted. Something of value is that you will see:

  1. Checkpoint warning:

2019-09-04 23:02:57.139 [P2P3] WARNING checkpoints src/checkpoints/checkpoints.cpp:58 CHECKPOINT FAILED FOR HEIGHT 349136. EXPECTED HASH <0e34e9d393e1ecf1faa977066d139e65fc7dc6a0e4f5de2474f55976e372e009>GIVEN HASH: <52a02df6bd1c0c58e46e09aecd907019b790cb0a40a4764b9a346aff235e01f7>

  1. Then sync data returned a new top block candidate (117 days ahead)

2019-09-04 23:43:10.517 [P2P9] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:374 [139.99.122.160:50754 INC] Sync data returned a new top block candidate: 351798 -> 267280 [Your node is 84518 blocks (117 days) ahead]
SYNCHRONIZATION started

Then you will see Warnings Msgs 'Large Hash rate changes'

As mentioned above and yesterday, a reboot seems to resolve.

Logs today detail:

2019-09-04 21:15:24.375 [P2P3] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:2072 SYNCHRONIZED OK
2019-09-04 21:45:53.084 [P2P6] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 22:45:54.739 [P2P6] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 23:02:57.138 [P2P3] INFO global src/cryptonote_core/blockchain.cpp:1942 HF12 Checkpointing Pre-Soft Fork: ###### Would have REORGANIZED on height: 349136 of 351779, checkpoint is found in alternative chain on height 349136
2019-09-04 23:02:57.139 [P2P3] WARNING checkpoints src/checkpoints/checkpoints.cpp:58 CHECKPOINT FAILED FOR HEIGHT 349136. EXPECTED HASH <0e34e9d393e1ecf1faa977066d139e65fc7dc6a0e4f5de2474f55976e372e009>GIVEN HASH: <52a02df6bd1c0c58e46e09aecd907019b790cb0a40a4764b9a346aff235e01f7>
2019-09-04 23:02:57.181 [P2P3] INFO global src/cryptonote_core/blockchain.cpp:1959 ----- BLOCK ADDED AS ALTERNATIVE ON HEIGHT 349137
id:
PoW: <53df94402fe895bc8daaa14fb675e13e1e7a1857a43c5e411f15f2ff01000000>
difficulty: 1255308067
2019-09-04 23:02:57.231 [P2P3] INFO global src/cryptonote_core/blockchain.cpp:1959 ----- BLOCK ADDED AS ALTERNATIVE ON HEIGHT 349138
id: <20c97a695abec356a1f39742efdadfc1d9e6039445b77008c00e36bf6cff4564>
PoW: <00b85f156264ffa7eaec6b8878b9ba19c5beebcd9e0103851927d63600000000>
difficulty: 1192493388
2019-09-04 23:02:57.276 [P2P3] INFO global src/cryptonote_core/blockchain.cpp:1959 ----- BLOCK ADDED AS ALTERNATIVE ON HEIGHT 349139
id: <8bfcfb2de9910021c78e51005aa065fa147ebac8303765e69e472602de4c2431>
PoW: <7cebc299fecec4eb2b5808f0445e3cb0868fa02b9621aea6069955e500000000>
difficulty: 1198586790
2019-09-04 23:12:49.122 [P2P0] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:1415 Synced 351784/351784
2019-09-04 23:12:49.131 [P2P0] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:2072 SYNCHRONIZED OK
2019-09-04 23:43:10.517 [P2P9] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:374 [139.99.122.160:50754 INC] Sync data returned a new top block candidate: 351798 -> 267280 [Your node is 84518 blocks (117 days) ahead]
SYNCHRONIZATION started
2019-09-04 23:45:56.401 [P2P5] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 23:50:56.498 [P2P8] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 23:54:24.581 [P2P7] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 0 blocks in the last 20 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 23:55:02.074 [P2P3] WARNING global src/cryptonote_protocol/cryptonote_protocol_handler.inl:2430 lokid is now disconnected from the network
2019-09-04 23:55:54.606 [P2P4] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 0 blocks in the last 20 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 23:55:56.606 [P2P9] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-04 23:57:24.647 [P2P3] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 1 blocks in the last 30 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-04 23:58:54.674 [P2P6] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 1 blocks in the last 30 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-05 00:00:24.704 [P2P1] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 1 blocks in the last 30 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-05 00:00:56.738 [P2P3] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>

Instead of pasting all of the logs, you will see that the reboot nullified the situation as shown below.

2019-09-05 00:40:32.038 [P2P6] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 15 blocks in the last 90 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-05 00:41:33.060 [P2P3] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-05 00:42:02.067 [P2P8] WARNING cn src/cryptonote_core/cryptonote_core.cpp:2159 There were 15 blocks in the last 90 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.
2019-09-05 00:42:23.228 [P2P4] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:374 [194.126.181.117:57412 INC] Sync data returned a new top block candidate: 351798 -> 351826 [Your node is 28 blocks (0 days) behind]
SYNCHRONIZATION started
2019-09-05 00:42:28.169 [P2P1] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:1415 Synced 351826/351826
2019-09-05 00:42:28.169 [P2P1] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:2072 SYNCHRONIZED OK
2019-09-05 00:42:28.169 [P2P1] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:2094


You are now synchronized with the network. You may now start loki-wallet-cli.

Use the "help" command to see the list of available commands.


2019-09-05 00:46:33.205 [P2P9] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-05 00:51:33.311 [P2P9] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-05 00:56:33.434 [P2P2] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
2019-09-05 01:01:33.538 [P2P1] INFO global src/cryptonote_core/cryptonote_core.cpp:1418 Submitted uptime-proof for service node (yours): <3fe211fb719274279794f2cfbc0d9abf3f531f30567de7a3dc355f57133d8ef4>
snode3@lokisnode3:~/.loki$

@anon1972
Copy link
Author

anon1972 commented Sep 5, 2019

... the next time this occurs. Also lokid sync_info would be helpful to see how many peers you have.

Sorry... I rushed home after getting these LokiSNBot alerts and did not see your messages.

I'm ok now, but I will capture this info the next time. Here is what I see @current:

status
FROM SOCKET: Height: 351842/351842 (100.0%) on mainnet, not mining, net hash 14.35 MH/s, v12, up to date, 2(out)+7(in) connections, uptime 0d 0h 35m 23s

sync_info
FROM SOCKET: Height: 351842, target: 351842 (100%)
Downloading at 0 kB/s
6 peers
116.203.196.12:36694 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
51.38.133.145:49136 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
116.203.196.12:36272 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
51.38.133.145:47002 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
51.38.133.145:46670 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
94.130.176.250:22022 f52325ed1da21c85 normal 0 351842 0 kB/s, 0 blocks / 0 MB queued
0 spans, 0 MB

Stay tuned :)

@jagerman
Copy link
Member

jagerman commented Sep 5, 2019

This peer list doesn't look healthy, suggesting your lokid is dropping peers for some reason. Can you check it again, and also cross-check the "Height" value with the current height at lokiblocks.com? One other thing to check is the output of diff: it should give you something like:

BH: 352186, TH: 387320d2f343ca1dc6675af43a2464bf39cff01ccc495079c431fc19129e1401, DIFF: 2240139797, HR: 18667831 H/s

and the DIFF value there shouold be identical to the value near the top of lokiblocks:
Network difficulty: 2,240,139,797.

If it isn't (but the block height matches) then you can try recalculating the difficulty by restarting with --recalculate-difficulty 1. It certainly won't hurt, and might help if the difficulty calculation broke at some point.

@anon1972
Copy link
Author

anon1972 commented Sep 5, 2019

Here you go my friend....

//status
FROM SOCKET: Height: 352466/352466 (100.0%) on mainnet, not mining, net hash 32.98 MH/s, v12, up to date, 8(out)+9(in) connections, uptime 0d 20h 40m 6s

//sync_info
FROM SOCKET: Height: 352466, target: 352466 (100%)
Downloading at 4 kB/s
15 peers
51.38.133.145:40560 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
51.38.133.145:39414 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
116.203.196.12:55022 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
51.38.133.145:38956 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
116.203.196.12:54248 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
51.38.133.145:37268 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
51.38.133.145:38706 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
95.216.146.127:22022 33e792f1ed05886c normal 0 352466 0 kB/s, 0 blocks / 0 MB queued
66.42.33.130:22022 73e46e147273a1e8 normal 0 352466 0 kB/s, 0 blocks / 0 MB queued
144.134.204.230:22022 a9aa9607bfdf0a7e normal 0 352466 0 kB/s, 0 blocks / 0 MB queued
157.245.79.93:22022 5dab7e99a2f64d94 normal 0 352466 0 kB/s, 0 blocks / 0 MB queued
116.203.196.12:56410 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
51.79.55.159:22022 da19a7f8c373940d normal 0 352466 0 kB/s, 0 blocks / 0 MB queued
13.113.158.162:22022 3a8b1e4b69b4d49f normal 0 352466 4 kB/s, 0 blocks / 0 MB queued
209.141.33.176:22022 c5bb9cad88c0b3c0 normal 0 196033 0 kB/s, 0 blocks / 0 MB queued
0 spans, 0 MB

//diff
FROM SOCKET: BH: 352469, TH: 5f263da177c1945616f38bb4b61b5cda84ba19dca8e4d63bd622c30970fa2cb6, DIFF: 3420383698, HR: 28503197 H/s

//Compare against lokiblocks

Network difficulty: 3,420,383,698 | Hard fork: v12 | Hash rate: 28.503 MH/s | Staking Requirement: 21524.384822930 | Fee per byte: 0.000016162 | Median block size limit: 292.97 kB | Total blockchain size: 15.00 GB

@anon1972
Copy link
Author

anon1972 commented Sep 5, 2019

Something that may be of interest. I noticed that memory use appears to be a bit high. (74.9)

779 snode3 20 0 34.5g 2.9g 1.9g S 2.0 74.9 33:28.03 lokid

@Doy-lee
Copy link
Collaborator

Doy-lee commented Nov 25, 2019

This should be resolved after some of the checkpointing hot fixes went into master. Including failure to reorg or handle mismatched checkpoints properly #814

There's also #845 that notifies you when uptime proof is received back from the network for further confirmation.

@Doy-lee Doy-lee closed this as completed Nov 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants