Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add REST endpoint to retrieve historical_summaries #6675

Open
wants to merge 3 commits into
base: unstable
Choose a base branch
from

Conversation

kdeme
Copy link
Contributor

@kdeme kdeme commented Oct 23, 2024

In draft currently as it is not in spec. Needs to be accompanied still with a proposal for beacon REST API specifications.

The idea is that historical_summaries is very useful for verifying if beacon blocks (with their accompanied proofs) are part of the canonical chain, however there is no way to actually retrieve the historical_summaries except by using the /eth/v2/debug/beacon/states/{state_id} endpoint, which is quite the download in terms of size (We are currently using this endpoint in Portal/fluffy for our portal_bridge, but it takes long).

This PR adds an endpoint that provides the historical_summaries with its proof against the state root. This is done so that the historical_summaries can then be verified for example with the finalized state root that a (light) node might hold from its finality update.
Again, this is specifically useful in Portal (as a portal_bridge can inject it in the p2p network as is, and receiving nodes can verify it if they are light client synced) . But you could make an argument that it could also be useful on a consensus light client (assuming blocks with proofs would be somehow available).

edit: moved to /nimbus/v1/debug/ namespace and removed draft

RestApiResponse.jsonResponseFinalized(
response, node.getStateOptimistic(state), node.dag.isFinalized(bslot.bid)
)
elif contentType == sszMediaType:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opted for JSON and SSZ, as that is what the debug state endpoint does and also because I was using the SSZ version already in our portal_bridge.

I'm not sure what the convention is, or if there is a convention, of when to opt also for SSZ. I've seen quite some big responses that do not seem to offer SSZ as an option in the beacon API.

Copy link
Contributor

@cheatfate cheatfate Oct 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

preferredContentType function provides interface to choose what client want to get. Client could use different versions of this construction:

  • Accept: application/json
  • Accept: application/octet-stream (SSZ)
  • Accept: application/json;q=0.5, application/octet-stream;q=0.9
  • Accept: application/json;q=0.5, application/octet-stream

and so on...

when using preferredContentStream server side portion could which order is preferred for the server. And preferredContentStream() establish consensus and return you agreed content type.

)
elif contentType == sszMediaType:
let
headers = [("eth-consensus-version", consensusFork.toString())]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this as it seems to be added always (?), even though I can't think of any use for it in the specific case of historical_summaries. (Perhaps if the BeaconState amount of fields grows > 32 it could be useful for the proof)


router.metricsApi2(
MethodGet,
"/eth/v1/debug/beacon/states/{state_id}/historical_summaries",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added it under the debug namespace for now.

Although I realize that debug is not actually purely a list of debug endpoints as the state one is used by users for trusted node sync.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

debug section is part of official specification, you should not add endpoints to this section if its not yet part of official specification, we have nimbus section for such calls.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is why the PR is in draft for now. I want to hopefully get it added in official specification.

I was not aware about our specific nimbus prefixed API. I'd be happy to move it there already if that would mean earlier merging in nimbus :)

Copy link

github-actions bot commented Oct 23, 2024

Unit Test Results

       12 files  ±0    1 814 suites  ±0   54m 41s ⏱️ - 1m 1s
  5 232 tests ±0    4 884 ✔️ ±0  348 💤 ±0  0 ±0 
29 077 runs  ±0  28 625 ✔️ ±0  452 💤 ±0  0 ±0 

Results for commit 566e69b. ± Comparison against base commit e06af18.

♻️ This comment has been updated with latest results.


proc getHistoricalSummariesV1*(
client: RestClientRef, state_id: StateIdent, cfg: RuntimeConfig, restAccept = ""
): Future[Option[GetHistoricalSummariesV1Response]] {.async.} =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are not using simple {.async.} anymore, it should use new syntax with asyncraises.
By default PRESTO annotates client functions which returns RestPlainResponse with
[CancelledError, RestEncodingError, RestDnsResolveError, RestCommunicationError] as you can see here
https://github.com/status-im/nim-presto/blob/master/presto/client.nim#L489-L509

Of course you can use [RestError, CancelledError] because RestError is common ancestor for all exceptions generated by PRESTO, but its not recommended.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In your case it should also include RestResponseError which you generating manually.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I noticed the {.async.} still being used in the debug calls and wasn't sure if everything was ready to use asyncraises. But looking at the other APIs now I see it has been using asyncraises everywhere.
I willl adjust to asyncraises.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the calls which are using rest pragma annotation already include asyncraises (but its hidden). All other wrappers will be converted soon.

@tersec
Copy link
Contributor

tersec commented Oct 24, 2024

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

@kdeme
Copy link
Contributor Author

kdeme commented Oct 24, 2024

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

@cheatfate
Copy link
Contributor

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

Why you need to remove it? You can establish redirect() call from one endpoint to another when it will be part of specification, we was doing this many times.

@tersec
Copy link
Contributor

tersec commented Oct 25, 2024

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

Why you need to remove it? You can establish redirect() call from one endpoint to another when it will be part of specification, we was doing this many times.

Well, part of the point is not to end up with extra, redundant endpoints. The point of this subject to removal without notice would be to exactly allow experimentation without obligation such directions later.

@cheatfate
Copy link
Contributor

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

Why you need to remove it? You can establish redirect() call from one endpoint to another when it will be part of specification, we was doing this many times.

Well, part of the point is not to end up with extra, redundant endpoints. The point of this subject to removal without notice would be to exactly allow experimentation without obligation such directions later.

There could be exactly 2 situations:

  1. Change will be proposed, but declined.
  2. Change will be proposed and accepted.

In both this cases it could be better to start with /nimbus/ endpoint. Because /nimbus/ endpoint could not be regulated by EF, so we could add/remove any functions. Otherwise in case 1 you will be forced to remove it from /debug/.

return withState(state):
when consensusFork >= ConsensusFork.Capella:
# Build the proof for historical_summaries field (28th field in BeaconState)
let gIndex = GeneralizedIndex(59) # 31 + 28 = 59
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this index is heavily depends on current version of BeaconState object, which means that if BeaconState object will change in next fork this index should be modified or at least visible for new fork maintainers.
So we should add something like

static: doAssert high(ConsensusFork) == ConsensusFork.Electra

here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, and also, >= Capella is wrong because Electra has different gindex than Deneb.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I see in Electra the amount of fields grows > 32. Basically what I write here (#6675 (comment)) is already happening.

I will adjust index for different forks and assert for unsupported/unknown forks.

@@ -337,6 +337,11 @@ type
RestEpochRandao* = object
randao*: Eth2Digest

RestHistoricalSummaries* = object
historical_summaries*: seq[HistoricalSummary]
proof*: array[5, Eth2Digest]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment #6675 (comment) means also that this will become array[6, Eth2Digest] for Electra. For the JSON part that is not really an issue. In Nim this means I will need to either create different versions of RestHistoricalSummaries depending on fork or perhap use seq instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants