Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The node stopped producing blocks at the beginning of the epoch #2852

Closed
narniec opened this issue Jun 15, 2020 · 15 comments
Closed

The node stopped producing blocks at the beginning of the epoch #2852

narniec opened this issue Jun 15, 2020 · 15 comments

Comments

@narniec
Copy link

narniec commented Jun 15, 2020

This morning I noticed that my node stopped producing blocks. I immediately opened the logs and saw the following:
photo_2020-06-15_16-54-43
I rebooted the node, but it didn't solve the problem. After deleted folder data /root/.near/betanet/data and it helped
Blocks were no longer produced at the beginning of the epoch (only 1, an hour later, another 1 block)
333
at the same time my cpu grew sharply
cpu

Version (please complete the following information):

  • nearcore commit/branch: b30864b
  • betanet
@bowenwang1996
Copy link
Collaborator

Two questions:

  • Did it only happen at the beginning of an epoch?
  • How were you sure that your node was not producing blocks?

@narniec
Copy link
Author

narniec commented Jun 15, 2020

Two questions:

  • Did it only happen at the beginning of an epoch?
  • How were you sure that your node was not producing blocks?
  • Yes, at the beginning of the epoch. The previous 2 epochs fully received 100% blocks
  • I received a notification in my email, I checked "near validators current" and saw my Blocks produced = 0 and i checked logs. I had this problem yesterday, but I thought it wouldn't happen again

@bowenwang1996
Copy link
Collaborator

I checked "near validators current" and saw my Blocks produced = 0 and i checked logs

How many blocks were you supposed to produce when you saw there are zero blocks produced?

@narniec
Copy link
Author

narniec commented Jun 15, 2020

I checked "near validators current" and saw my Blocks produced = 0 and i checked logs

How many blocks were you supposed to produce when you saw there are zero blocks produced?
I think about 30, before that i was on the road

@bowenwang1996
Copy link
Collaborator

Looks like it is caused by #2806, but it is not clear to me why this only happens to your node. What kind of CPU do you use?

@narniec
Copy link
Author

narniec commented Jun 15, 2020

Looks like it is caused by #2806, but it is not clear to me why this only happens to your node. What kind of CPU do you use?

My CPU:
AMD EPYC Processor (with IBPB)
Снимок

There are 2 more people who have the same problem the last 2 days. They have the same high CPU load. I can find out what processor they , if necessary
1.
photo_2020-06-15_20-29-08
2. Intel(R) Xeon(R) CPU @ 2.30GHz
photo_2020-06-15_16-28-42

@bowenwang1996
Copy link
Collaborator

2 more people who have the same problem the last 2 days

I see. Did they report it anywhere?

@narniec
Copy link
Author

narniec commented Jun 15, 2020

2 more people who have the same problem the last 2 days

I see. Did they report it anywhere?

No.
They wrote only in the telegram chat RU

@bowenwang1996
Copy link
Collaborator

@narniec how long does the high cpu usage last?

@narniec
Copy link
Author

narniec commented Jun 16, 2020

@narniec how long does the high cpu usage last?

Until I stop the node and delete the data folder. My node works for 2 epochs, after which it does not produce blocks.
I set up another server.
Public_key and account ID a good
ll

@bowenwang1996
Copy link
Collaborator

Until I stop the node and delete the data folder.

Can you give an estimate in terms of the amount of time or number of blocks?

@narniec
Copy link
Author

narniec commented Jun 16, 2020

Exactly two epochs passed, and the node worked well. Now the beginning of the new epoch and in the logs immediately appeared this:
1
Cpu UP:
image

@bowenwang1996
Copy link
Collaborator

How long does this last?

@narniec
Copy link
Author

narniec commented Jun 17, 2020

How long does this last?
As i said earlier, until i delete the data folder.
Today i moved to a new server, made everything new, changed the validator key, and check how everything will work. The 3rd epoch is already underway and everything is fine!

@narniec narniec closed this as completed Jul 21, 2020
@narniec
Copy link
Author

narniec commented Jul 21, 2020

I have not encountered this problem again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants