Add REST endpoint to retrieve historical_summaries

[kdeme]

In draft currently as it is not in spec. Needs to be accompanied still with a proposal for beacon REST API specifications.

The idea is that historical_summaries is very useful for verifying if beacon blocks (with their accompanied proofs) are part of the canonical chain, however there is no way to actually retrieve the historical_summaries except by using the /eth/v2/debug/beacon/states/{state_id} endpoint, which is quite the download in terms of size (We are currently using this endpoint in Portal/fluffy for our portal_bridge, but it takes long).

This PR adds an endpoint that provides the historical_summaries with its proof against the state root. This is done so that the historical_summaries can then be verified for example with the finalized state root that a (light) node might hold from its finality update.
Again, this is specifically useful in Portal (as a portal_bridge can inject it in the p2p network as is, and receiving nodes can verify it if they are light client synced) . But you could make an argument that it could also be useful on a consensus light client (assuming blocks with proofs would be somehow available).

edit: moved to /nimbus/v1/debug/ namespace and removed draft

[github-actions]

Unit Test Results

       15 files  ±0    2 614 suites  ±0   1h 13m 12s ⏱️ +40s
  6 412 tests ±0    5 891 ✔️ ±0  521 💤 ±0  0 ❌ ±0 
44 616 runs  ±0  43 898 ✔️ ±0  718 💤 ±0  0 ❌ ±0 

Results for commit 62293d6. ± Comparison against base commit 0e3ba6f.

♻️ This comment has been updated with latest results.

[tersec]

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

[kdeme]

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

[cheatfate]

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

Why you need to remove it? You can establish redirect() call from one endpoint to another when it will be part of specification, we was doing this many times.

[tersec]

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

Why you need to remove it? You can establish redirect() call from one endpoint to another when it will be part of specification, we was doing this many times.

Well, part of the point is not to end up with extra, redundant endpoints. The point of this subject to removal without notice would be to exactly allow experimentation without obligation such directions later.

[cheatfate]

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

Why you need to remove it? You can establish redirect() call from one endpoint to another when it will be part of specification, we was doing this many times.

Well, part of the point is not to end up with extra, redundant endpoints. The point of this subject to removal without notice would be to exactly allow experimentation without obligation such directions later.

There could be exactly 2 situations:

1. Change will be proposed, but declined.
2. Change will be proposed and accepted.

In both this cases it could be better to start with /nimbus/ endpoint. Because /nimbus/ endpoint could not be regulated by EF, so we could add/remove any functions. Otherwise in case 1 you will be forced to remove it from /debug/.

[cheatfate]

I think this index is heavily depends on current version of BeaconState object, which means that if BeaconState object will change in next fork this index should be modified or at least visible for new fork maintainers.
So we should add something like

static: doAssert high(ConsensusFork) == ConsensusFork.Electra

here.

[etan-status]

Yep, and also, >= Capella is wrong because Electra has different gindex than Deneb.

[kdeme]

Right, I see in Electra the amount of fields grows > 32. Basically what I write here (#6675 (comment)) is already happening.

I will adjust index for different forks and assert for unsupported/unknown forks.

[kdeme]

added support for forks in 631a155

[kdeme]

Went with different arrays for the different proofs as this is for example also what is done for FinalityBranch, CurrentSyncCommitteeBranch and NextSyncCommitteeBranch.

But could in theory also just use a seq[Eth2Digest] which I think would make the whole ForkedHistoricalSummariesWithProof helpers not required.

[etan-status]

I think having it consistent with other endpoints is better than seq, even though it adds some boilerplate here. There is for every fork only a single correct proof length, and using array guarantees that incorrect proof lengths are rejected early at deserialization time.

The real solution would be to adopt EIP-7688... then there are no more random proof length changes for different consensusFork. It's a design flaw that every client has to keep maintaining gindices whenever ethereum decides to release some random functionality unrelated to the client's interests.

[kdeme]

The real solution would be to adopt EIP-7688

Yes, I agree. But that is not in my sole hands here :)

[kdeme]

@etan-status and/or @cheatfate Would you mind having a look at this again? It is something crucial for our development on Portal network.

As mentioned in the comments, fork management is added, which I think was the only issue (at least for getting it under nimbus/debug namespace)?

Some remarks:

• Those HistoricalSummariesProof and ForkedHistoricalSummariesProof are added in the rest types code as I did not want to pollute the specs and forks code with types that are not actually part of the consensus specifications. But if you believe they now pollute too much the rest types, I can also move them into another new file.
• The Forked type is required as we use arrays for the proofs (which differ in size between forks). We could also opt for going with a sequence instead, but that is not consistent with what is done for example for FinalityBranch, CurrentSyncCommitteeBranch and NextSyncCommitteeBranch.

[etan-status]

There's also an effort from @Inspector-Butters and @wemeetagain to look into a general proof API. But I think as part of Nimbus namespace it's alright to have specific endpoints to unblock portal progress.

Code looks correct to me, I have put some remarks to make it less maintenance-heavy and reduce duplication. Once comments are addressed, good to go.

[etan-status]

You could reduce code duplication by extending forks.nim with

template historical_summaries_gindex*(
    kind: static ConsensusFork): GeneralizedIndex =
  when kind >= ConsensusFork.Electra:
    HISTORICAL_SUMMARIES_GINDEX_ELECTRA
  elif kind >= ConsensusFork.Capella:
    HISTORICAL_SUMMARIES_GINDEX
  else:
    {.error: "historical_summaries_gindex does not support " & $kind.}

and eth2_rest_serialization.nim with

template GetHistoricalSummariesResponse(
    kind: static ConsensusFork): auto =
  when kind >= ConsensusFork.Electra:
    GetHistoricalSummariesV1ResponseElectra
  elif kind >= ConsensusFork.Capella:
    GetHistoricalSummariesV1Response
  else:
    {.error: "GetHistoricalSummariesResponse does not support " & $kind.}

then can collapse the duplicated logic here:

        when consensusFork >= ConsensusFork.Capella:
          let response = consensusFork.GetHistoricalSummariesResponse(
            historical_summaries: forkyState.data.historical_summaries,
            proof: forkyState.data.build_proof(
              consensusFork.HISTORICAL_SUMMARIES_GINDEX).expect("Valid gindex"),
            slot: bslot.slot)

          if contentType == jsonMediaType:
            RestApiResponse.jsonResponseFinalizedWVersion(
              response,
              node.getStateOptimistic(state),
              node.dag.isFinalized(bslot.bid),
              consensusFork)
          elif contentType == sszMediaType:
            let headers = [("eth-consensus-version", consensusFork.toString())]
            RestApiResponse.sszResponse(response, headers)
          else:
            RestApiResponse.jsonError(Http500, InvalidAcceptError)

[etan-status]

Hmm. Do we really need to retain the exact consensusFork?

Or would a HistoricalSummariesFork make sense, similar to LightClientDataFork, EpochInfoFork or (experimental BlobFork in #6451)? That way, this code no longer has to be maintained with every fork, by simply having a capellaData and electraData member, and no denebData / fuluData etc

case x.kind
of HistoricalSummariesFork.Electra:
  const historicalFork {.inject, used.} = HistoricalSummariesFork.Electra
  template forkySummaries: untyped {.inject, used.} = x.electraData
  body
of ConsensusFork.Capella:
  const historicalFork {.inject, used.} = HistoricalSummariesFork.Capella
  template forkySummaries: untyped {.inject, used.} = x.capellaData
  body

One can still recover exact consensusFork if needed from Eth-Consensus-Version HTTP header, if needed.

[kdeme]

No, I don't think we need it.

In the past, I mostly found it confusing to work with these different fork types and always had to look back at which one contained which forks, hence why I did not go for that.

But I think I can understand how it makes maintenance probably quite a bit easier when you are dealing with that all the time. Though for an external/new reader of the code, having all these different fork types without a reference in the spec, might be confusing.

[etan-status]

Or just don't define forkySummaries instead of dummy value. accessing it when consensusFork < ConsensusFork.Capella then no longer compiles and makes misuse easier to detect at compiletime.

See template withForkyStore*( in forks_light_client.nim for comparison.

[etan-status]

I think having it consistent with other endpoints is better than seq, even though it adds some boilerplate here. There is for every fork only a single correct proof length, and using array guarantees that incorrect proof lengths are rejected early at deserialization time.

The real solution would be to adopt EIP-7688... then there are no more random proof length changes for different consensusFork. It's a design flaw that every client has to keep maintaining gindices whenever ethereum decides to release some random functionality unrelated to the client's interests.

[etan-status]

editor should be configured to add trailing newline

[kdeme]

editor is actually configured to use nph, but I have to disable this for some repos :)