Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AttesterSlashings causes gc spiked on mainnet nodes #6112

Closed
twoeths opened this issue Nov 17, 2023 · 7 comments
Closed

AttesterSlashings causes gc spiked on mainnet nodes #6112

twoeths opened this issue Nov 17, 2023 · 7 comments
Assignees
@twoeths
Labels
meta-bug Issues that identify a bug and require a fix. prio-high Resolve issues as soon as possible.

Comments

@twoeths
Copy link
Contributor

twoeths commented Nov 17, 2023 •
edited
Loading

Describe the bug

This is on the stable and unstable mainnet node, since v1.12.0 Nov 14 gc rate increased significantly

Screenshot 2023-11-17 at 09 31 31

Epoch transition increased at the same time

Screenshot 2023-11-19 at 05 59 40

However it's significantly better on the 1k and md16 md64 goerli node

Screenshot 2023-11-17 at 09 32 21

Expected behavior

gc rate is the same or less compared to v1.11.x, or give an explanation on the difference

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.12.0
@twoeths twoeths added the meta-bug Issues that identify a bug and require a fix. label Nov 17, 2023
@twoeths
Copy link
Contributor Author

twoeths commented Nov 17, 2023

Update: need to ignore 1k, md16, md64 nodes due to the spiked epoch transition time issue #6063

@twoeths
Copy link
Contributor Author

twoeths commented Nov 17, 2023

This may not link to v1.12.0, could be because node processes more AtterterSlashing messages recently on mainnet. Found this on a node of 64 validators:

Screenshot 2023-11-17 at 16 49 28

This links to the recent high AttesterSlashing messages

Screenshot 2023-11-17 at 16 49 58

note that AtterterSlashing uses BigInt which may cause memory and performance issue to our node, see #5892

@twoeths
Copy link
Contributor Author

twoeths commented Nov 17, 2023

also the gc increased at the same time

Screenshot 2023-11-17 at 17 21 48

@twoeths twoeths changed the title High gc rate since v1.12.0 High gc rate since Nov 14 Nov 18, 2023
@twoeths
Copy link
Contributor Author

twoeths commented Nov 18, 2023

Confirm that there are a lot of validators get slashed on Nov 14 https://beaconcha.in/validators/slashings

@twoeths twoeths changed the title High gc rate since Nov 14 High gc rate since Nov 14 on mainnet nodes Nov 18, 2023
@twoeths twoeths self-assigned this Nov 20, 2023
@twoeths
Copy link
Contributor Author

twoeths commented Nov 21, 2023

node wasn't able to recover since Nov 14

Screenshot 2023-11-21 at 13 01 43 Screenshot 2023-11-21 at 13 01 57

@twoeths twoeths added the prio-high Resolve issues as soon as possible. label Nov 21, 2023
@twoeths twoeths changed the title High gc rate since Nov 14 on mainnet nodes AttesterSlashings causes gc spiked on mainnet nodes Nov 21, 2023
This was referenced Nov 21, 2023
Closed
Merged
@twoeths twoeths mentioned this issue Nov 30, 2023
Merged
@wemeetagain
Copy link
Member

@twoeths can this be closed?

@twoeths
Copy link
Contributor Author

twoeths commented Oct 23, 2024

yes we upgraded NodeJS so not sure BigInt is an issue, and haven't seen this for a while

@twoeths twoeths closed this as completed Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@twoeths twoeths

Labels
meta-bug Issues that identify a bug and require a fix. prio-high Resolve issues as soon as possible.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wemeetagain @twoeths