Prune too many checkpoint states at restart

[twoeths]

Describe the bug

@nflaig noticed this when he restarted his mainnet node

info: Removed persisted checkpoint states from the last run count=178, maxEpochsInMemory=3

also @philknows noticed in another mainnet node

Feb-09 03:32:48.623[chain]            info: Removed persisted checkpoint states from the last run count=261, maxEpochsInMemory=3

Expected behavior

with mainnet condition, it should be <= 3

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.27.0

[philknows]

Hmm... one of our mainnet unstable servers shows some occurrences of >3 in the last 30d, though happening a lot less now:

  |   | Jan-20 18:18:51.906[chain]            info: Removed persisted checkpoint states from the last run count=2, maxEpochsInMemory=3 |
  |   | Jan-20 21:50:54.716[chain]            info: Removed persisted checkpoint states from the last run count=1, maxEpochsInMemory=3 |  
  |   | Jan-20 22:10:47.512[chain]            info: Removed persisted checkpoint states from the last run count=1, maxEpochsInMemory=3 |  
  |   | Jan-23 10:59:04.986[chain]            info: Removed persisted checkpoint states from the last run count=58, maxEpochsInMemory=3 |  
  |   | Jan-23 19:10:54.152[chain]            info: Removed persisted checkpoint states from the last run count=11, maxEpochsInMemory=3 |  
  |   | Jan-23 19:50:52.223[chain]            info: Removed persisted checkpoint states from the last run count=1, maxEpochsInMemory=3 |  
  |   | Jan-23 21:18:58.629[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Jan-23 22:02:05.620[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Jan-23 22:50:09.566[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Jan-24 15:50:08.072[chain]            info: Removed persisted checkpoint states from the last run count=3, maxEpochsInMemory=3 |  
  |   | Jan-27 15:34:21.458[chain]            info: Removed persisted checkpoint states from the last run count=29, maxEpochsInMemory=3 |  
  |   | Jan-27 15:50:15.307[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Jan-27 23:26:11.967[chain]            info: Removed persisted checkpoint states from the last run count=5, maxEpochsInMemory=3 |  
  |   | Jan-29 09:42:14.771[chain]            info: Removed persisted checkpoint states from the last run count=25, maxEpochsInMemory=3 |  
  |   | Jan-29 10:58:14.907[chain]            info: Removed persisted checkpoint states from the last run count=2, maxEpochsInMemory=3 |  
  |   | Jan-30 01:02:11.091[chain]            info: Removed persisted checkpoint states from the last run count=14, maxEpochsInMemory=3 |  
  |   | Jan-30 13:58:13.610[chain]            info: Removed persisted checkpoint states from the last run count=21, maxEpochsInMemory=3 |  
  |   | Jan-30 21:58:11.574[chain]            info: Removed persisted checkpoint states from the last run count=6, maxEpochsInMemory=3 |  
  |   | Jan-30 22:18:08.369[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Jan-31 07:18:12.443[chain]            info: Removed persisted checkpoint states from the last run count=20, maxEpochsInMemory=3 |  
  |   | Jan-31 15:10:11.120[chain]            info: Removed persisted checkpoint states from the last run count=14, maxEpochsInMemory=3 |  
  |   | Jan-31 15:42:12.224[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Jan-31 16:10:12.348[chain]            info: Removed persisted checkpoint states from the last run count=1, maxEpochsInMemory=3 |  
  |   | Jan-31 16:26:14.238[chain]            info: Removed persisted checkpoint states from the last run count=1, maxEpochsInMemory=3 |  
  |   | Feb-03 15:34:12.169[chain]            info: Removed persisted checkpoint states from the last run count=19, maxEpochsInMemory=3 |  
  |   | Feb-03 15:50:13.542[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Feb-04 19:10:11.108[chain]            info: Removed persisted checkpoint states from the last run count=4, maxEpochsInMemory=3 |  
  |   | Feb-06 19:34:11.313[chain]            info: Removed persisted checkpoint states from the last run count=8, maxEpochsInMemory=3 |  
  |   | Feb-06 21:58:09.352[chain]            info: Removed persisted checkpoint states from the last run count=1, maxEpochsInMemory=3 |  
  |   | Feb-07 02:18:37.496[chain]            info: Removed persisted checkpoint states from the last run count=1, maxEpochsInMemory=3 |  
  |   | Feb-07 20:10:07.039[chain]            info: Removed persisted checkpoint states from the last run count=4, maxEpochsInMemory=3 |  
  |   | Feb-07 22:18:13.576[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Feb-10 13:18:07.975[chain]            info: Removed persisted checkpoint states from the last run count=3, maxEpochsInMemory=3 |  
  |   | Feb-11 18:18:32.234[chain]            info: Removed persisted checkpoint states from the last run count=3, maxEpochsInMemory=3 |  
  |   | Feb-11 18:34:07.925[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Feb-11 21:58:05.768[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Feb-11 22:18:05.568[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Feb-11 22:42:11.467[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Feb-12 12:34:03.512[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Feb-12 13:02:06.949[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Feb-12 21:10:03.901[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Feb-13 13:26:09.776[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Feb-13 13:42:07.726[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Feb-13 16:02:36.122[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Feb-14 22:34:07.242[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3 |  
  |   | Feb-17 02:18:06.634[chain]            info: Removed persisted checkpoint states from the last run count=0, maxEpochsInMemory=3