Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flush commit metadata even when punning fails #115

Merged
merged 2 commits into from
Feb 19, 2022

Conversation

p0mvn
Copy link
Member

@p0mvn p0mvn commented Feb 19, 2022

Description

Closes: #XXXX


Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

  • included the correct type prefix in the PR title
  • added ! to the type prefix if API or client breaking change
  • targeted the correct branch (see PR Targeting)
  • provided a link to the relevant issue or specification
  • followed the guidelines for building modules
  • included the necessary unit and integration tests
  • added a changelog entry to CHANGELOG.md
  • included comments for documenting Go code
  • updated the relevant documentation or specification
  • reviewed "Files changed" and left comments if necessary
  • confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

  • confirmed the correct type prefix in the PR title
  • confirmed ! in the type prefix if API or client breaking change
  • confirmed all author checklist items have been addressed
  • reviewed state machine logic
  • reviewed API design and naming
  • reviewed documentation is accurate
  • reviewed tests and test coverage
  • manually tested (if applicable)

@p0mvn p0mvn requested a review from UnityChaos February 19, 2022 02:08
@UnityChaos
Copy link
Member

Looks reasonable to me, but @ValarDragon may object because this might cause other issues that I'm not aware of. Will rerun the "crash when trying to prune snapshot height" test with the branch tho.

Copy link
Member

@UnityChaos UnityChaos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor rename based on suggestion from @ValarDragon.
Other than that this LGTM 🎉


Tested this on mainnet with settings:

pruning = "custom"

# These are applied if and only if the pruning strategy is custom.
# pruning-keep-recent = N means keep all of the last N states
pruning-keep-recent = "100"
# pruning-keep-every = N means keep every Nth state, in addition to keep-recent
pruning-keep-every = "0"
# pruning-interval = N means we delete old states from disk every Nth block.
pruning-interval = "10"

and

###############################################################################
###                        State Sync Configuration                         ###
###############################################################################

# State sync snapshots allow other nodes to rapidly join the network without replaying historical
# blocks, instead downloading and applying a snapshot of the application state at a given height.
[state-sync]

# snapshot-interval specifies the block interval at which local state sync snapshots are
# taken (0 to disable). Must be a multiple of pruning-keep-every.
snapshot-interval = 30

# snapshot-keep-recent specifies the number of recent snapshots to keep and serve (0 to keep all).
snapshot-keep-recent = 2

This consistently causes a failure on pruning, and I was able to confirm that flushing the metadata with this patch is successfully fixing the halt on replay we were finding earlier.

The settings I'm using here (keep recent = 100, snapshot interval 30, keep 2 snapshots) shouldn't be causing crashes actually, so that looks to be another bug that I will investigate separately.

store/rootmulti/store.go Outdated Show resolved Hide resolved
store/rootmulti/store.go Outdated Show resolved Hide resolved
store/rootmulti/store.go Outdated Show resolved Hide resolved
@p0mvn
Copy link
Member Author

p0mvn commented Feb 19, 2022

That's great to hear. Thanks for reviewing and testing.

Since we know that this change fixes the earlier problem related to 6.3.0 only, now we can try addressing the "active reader problem" related to both 6.2.0 and 6.3.0. I will draft it up tomorrow morning. I think it's relate to not having synchronization in IAVL's nodedb

Copy link
Member

@ValarDragon ValarDragon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Nice catch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants