The Round Change continues indefinitely, BUT no new block is being generated. #207
Comments
|
Hello @Mameta29 To confirm the above:
Could you put some logs of the time this happened? Also what changed in your infrastructure/versions etc around the time that the blocks stopped? Typically that would indicate one of the following:
Steps to resolve the above are:
|
|
@joshuafernandes
-> YES
-> I see round changes, when I send a tx. Nomally there is nothing change like below. About steps:
|
|
Information added;
|
|
Hello @Mameta29 Yep that makes sense if the validators were uninstalled. To check on the validator keys you can check the secrets - each secret should have nodekey(or privatekey), public key and address and they get mounted in each container as files. The keys and such for the validators get created as part of the genesis chart. You then need to check the genesis file itself and see what addresses the genesis is expecting in the It is possible depending on the values.yml overrides the keys were deleted and new keys would have been created when your validators were re installed - those keys will not match whats in the genesis extra data and hence you get no blocks created. This part we are careful off and from memory we don't delete the keys because once the nodeprivate key is delete you can't create it again. To override this behavior the values.yml change this flag https://github.com/Consensys/quorum-kubernetes/blob/master/helm/values/validator.yml#L5 |
|
Thank you for your response. However, when the block generation stopped, there were no specific changes made to the validator, and it suddenly stopped. The attempt to restart the validator with 'helm uninstall' was done after the block generation had already stopped. Also, when reviewing the 'values.yml' file, it was configured as follows:
To confirm whether the keys necessary for block generation are mounted correctly within the Kubernetes pods for both validators and the RPC node, you can check the 'keys' directory within each pod. If the keys are correctly mounted, you should be able to see them in the respective 'keys' directories. Regarding the absence of the genesis file as a pod, it's not typically a pod itself. The genesis file is usually used during the initialization of the network and should be accessible on the file system of the nodes, often outside of the pods. I appreciate your patience, and please feel free to ask any further questions. |
|
I add some information. |
|
Hi @Mameta29
Thankyou for clearing that up. @antonydenyer any idea what would cause validators to suddenly stop? @Mameta29 do you have logs of the validators at that period we can look at? I'd suggest raising this as an issue on the goquorum repo and closing this - the maintainers there will be able to answer why it suddenly stopped creating blocks.
When you do an un install of the validators, what you have in the values.yml will determine wheter the existing keys are deleted or not. We default this to keeping the keys from memory as below
Not quite sure I follow you here,. The genesis file is created in a pod (as a job) which will exit when done - typically takes less than a couple of mins. It creates the genesis file as a configmap which is used by all nodes Cheers |
|
@joshuafernandes
The logs are below. It seems to be stuck in a loop of "round change." I wonder that something related that different validator lock the different block?
Thank you for this. I checked the genesis.json file. There are correct validator's addresses in the field of "alloc". But at "extraData" It seems to be nothing match of validator's. Please let me make sure that what I should check on the "extraData". After 64 hex characters, There should be account address?
Signer address mean same as account address like this? Thank you. |
Yep, this is correct and expected - in goq you can enable discovery and provide bootnodes or a list of static-nodes to connect to |
|
I just add an information.
I found an issue that validator are stopping. Is that something related to this issue? |
|
Hello @Mameta29 Consensys/quorum#1630 is different in that its potentially a hard fork, and logs clearly have them creating blocks. Your logs show nothing of the sort so blocks aren't being minted. The genesis file has:
More info on the genesis file here https://consensys.io/blog/hyperledger-besu-how-to-create-an-ethereum-genesis-file. So I'd suggest you do the following in this order:
|
|
@joshuafernandes Could you check if the fork happened with the logs below. The validator know the same number of block heigh but different block hash. |
|
Regarding the method to verify the extraData and the pub key as mentioned above, could you please confirm if my approach is correct? Also, if my verification method seems accurate, what should be the next steps based on these results? Does a discrepancy in extraData imply that the validator addresses have changed, making it impossible to restore this chain?
import RLP, { Input } from 'rlp';
export type Address = Buffer;
const validatorAddresses = [
"0x111f0f5e627b332ef26a2e7c507204a85ac52bbb",
"0xa874f9df79de99f3e558f1cdba7f7c6af33a5434",
"0x0c25b9b6e66caa1a43a803cb3d5cce4088f07422",
"0x09d0f31105244da2a79af132d32b8dcf11c2e9b8"
];
const validatorAddressBuffers: Address[] = validatorAddresses.map(address => Buffer.from(address.slice(2), 'hex'));
function generateExtraDataString(validatorAddressList: Address[]): string {
let extraData: Input;
const extraDataContent: Input = [new Uint8Array(32), validatorAddressList, [], null, []];
const extraDataCoded: Uint8Array = RLP.encode(extraDataContent);
extraData = '0x' + Buffer.from(extraDataCoded).toString('hex');
return extraData;
}
const extraData = generateExtraDataString(validatorAddressBuffers);
console.log("Generated extraData:", extraData);
The other three validators also had the same values for enode and nodekey.pub. While the extraData mentioned in point 1 is different, am I correct in understanding that the Pub key mentioned in point 2 is accurate? |
|
I've just updated to have found the "error logs" below. |
|
Hello @Mameta29 Right I'll give some more context: GoQ uses 2 types of addresses:
For what you are doing with verifying which validators are in the extradata we only care about item 1. We don;t need the alloc at all. So getting back to the nodekey:
You can also see all the files (for 1 & 2 that we use in the quickstart https://github.com/Consensys/quorum-dev-quickstart/tree/master/files/common/config/nodes/validator1) In the genesis file we then have the extra data made up of 4 address and padding (1 of those address is the one above) |
|
The goq team should be able to help figure out what happened with the nodes, looks like their either not peered and each creating their own block proposal or something like that. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I built a GoQuorum chain on AWS's EKS service following this tutorial:
https://docs.goquorum.consensys.net/tutorials/kubernetes/deploy-charts
Overview:
I was able to operate it for about a month, but suddenly blocks could no longer be produced. I was using EC2 instances for the EKS nodes, and their memory and CPU limits had been exceeded. I adjusted these and prepared sufficient EC2 instances. However, the same issue persists.
Regarding the event, a Round Change is taking place and an attempt is being made to produce blocks. There are no errors in each of the steps, but block production is not being reached. (I've posted the logs from the validators below.)
Validators: 4
RPC: 1
I am currently stuck as I cannot determine the cause and solution. Any assistance would be greatly appreciated.
The text was updated successfully, but these errors were encountered: