feat: implement expected withdrawals API #6514

HiroyukiNaito · 2024-03-06T10:46:30Z

Motivation

Latest beacon API spec requires expected withdrawals API

Description

Implement the following route:

https://ethereum.github.io/beacon-APIs/#/Builder/getNextWithdrawals

Steps to test or reproduce

curl -X 'GET' \
  'http://localhost:9596/eth/v1/builder/states/head/expected_withdrawals?proposal_slot=1' \
  -H 'accept: application/json'

Progress

Creating routes packages/api/src/beacon/routes/builder.ts
Defining types packages/api/src/beacon/routes/builder.ts
Implementing route and namaspace
packages/api/src/beacon/client/builder.ts
packages/api/src/beacon/client/index.ts
packages/api/src/beacon/index.ts
packages/api/src/beacon/routes/index.ts
packages/api/src/beacon/server/builder.ts
packages/api/src/beacon/server/index.ts
packages/beacon-node/src/api/impl/api.ts
Implementing function
packages/beacon-node/src/api/impl/builder/index.ts
Implementing unit test

Components

get state from state_id

lodestar/packages/beacon-node/src/api/impl/debug/index.ts

Line 39 in c94ecd2

async getState(stateId: string | number, format?: ResponseFormat) {
get expectedWithdrawals array from getExpectedWithdrawls(state from 1.)
filter proposal_slot form the 2. data

data:

lodestar/packages/state-transition/src/block/processWithdrawals.ts

Line 65 in 918924e

export function getExpectedWithdrawals(state: CachedBeaconStateCapella): {
finalized: N/A

Currently Lodestar doesn't support finalized in the apis. You can discard this field for now. See #5693 for more.

execution_optimistic:

lodestar/packages/beacon-node/src/util/forkChoice.ts

Line 3 in 1d3bf04

export function isOptimisticBlock(block: ProtoBlock): boolean {

HiroyukiNaito · 2024-03-06T11:03:44Z

This is a tentative implement.
Please, do not hesitate to point out if I am in a wrong direction because I'm a new born baby in this field.

ensi321 · 2024-03-07T04:23:08Z

Thanks for your contribution🙂

This is a tentative implement.
Please, do not hesitate to point out if I am in a wrong direction because I'm a new born baby in this field.

Just trying to understand the scope of this PR. Is it only adding the route definition of getNextWithdrawals, or is it going to implement the its functionality as well but you are still working on it?

If latter, it would be great if you can mark it as PR draft and reach out to the maintainers on Discord if you need any assistance. If former, I am happy to review it once the CI checks are passed😀

HiroyukiNaito · 2024-03-07T15:49:40Z

Thanks for your contribution🙂

This is a tentative implement.
Please, do not hesitate to point out if I am in a wrong direction because I'm a new born baby in this field.

Just trying to understand the scope of this PR. Is it only adding the route definition of getNextWithdrawals, or is it going to implement the its functionality as well but you are still working on it?

If latter, it would be great if you can mark it as PR draft and reach out to the maintainers on Discord if you need any assistance. If former, I am happy to review it once the CI checks are passed😀

That would be appreciated! So far, surveying how to implement the API. If both adding Routes and Types are completed, I would like to ask you to review before the implementation of the function and test.

HiroyukiNaito · 2024-03-08T10:38:56Z

@ensi321
Could you please review my code both routes and types when you have a moment before implementing the function?
I'm wondering it is right implement.

codecov · 2024-03-08T10:39:12Z

Codecov Report

Attention: Patch coverage is 70.23810% with 25 lines in your changes missing coverage. Please review.

Project coverage is 61.47%. Comparing base (918924e) to head (7359cad).
Report is 427 commits behind head on unstable.

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #6514      +/-   ##
============================================
- Coverage     61.55%   61.47%   -0.08%     
============================================
  Files           556      558       +2     
  Lines         58586    58736     +150     
  Branches       1847     1848       +1     
============================================
+ Hits          36060    36110      +50     
- Misses        22486    22586     +100     
  Partials         40       40

HiroyukiNaito · 2024-03-08T14:33:45Z

I misunderstood the concept. I will recreate it.

ensi321

Good work! Left some comments regarding the route definition

packages/api/src/builder/routes.ts

packages/api/test/unit/builder/testData.ts

packages/api/src/builder/routes.ts

ensi321 · 2024-03-11T07:28:20Z

Please convert this PR into PR draft while working on the implementation and convert it back when it's ready for another round of review. You can read more about it here

Maintainers will get notifications every time you commit unless you make this a draft 😅

HiroyukiNaito · 2024-03-11T07:35:25Z

@ensi321 Great! Thank you for the tips. According to these hints, I will program it again.

HiroyukiNaito · 2024-03-13T11:49:22Z

@ensi321

I implemented the interface for getting the withdrawal data.
However, curl -X 'GET' 'http://localhost:9596/eth/v1/builder/states/head/expected_withdrawals' -H 'accept: application/json' still returns {"message":"Route GET:/eth/v1/builder/states/head/expected_withdrawals not found","error":"Not Found","statusCode":404}.
It looks failed to route the api. Anything needs to add for routing the API?

ensi321 · 2024-03-14T07:36:32Z

@ensi321

I implemented the interface for getting the withdrawal data. However, curl -X 'GET' 'http://localhost:9596/eth/v1/builder/states/head/expected_withdrawals' -H 'accept: application/json' still returns {"message":"Route GET:/eth/v1/builder/states/head/expected_withdrawals not found","error":"Not Found","statusCode":404}. It looks failed to route the api. Anything needs to add for routing the API?

You will also need to add a http server for your newly created builder api.
See

lodestar/packages/api/src/beacon/server/debug.ts

Line 1 in 87f07fe

import {ChainForkConfig} from "@lodestar/config";

lodestar/packages/api/src/beacon/server/index.ts

Line 39 in 87f07fe

debug: () => debug.getRoutes(config, api.debug),

lodestar/packages/api/src/beacon/index.ts

Line 14 in 87f07fe

debug: true,

Lodestar should start serving this route afterwards. Also you will need to define a http client too which is similar to http server. See packages/api/src/beacon/client/debug.ts

HiroyukiNaito · 2024-03-14T10:24:31Z

@ensi321 Thank you so much for taking the time to review the code, especially with your busy schedule.
I could get fake response now before implementating the function.
If you have any correction before that, please do not hesitate to point out.

 curl -X 'GET'   'http://localhost:9596/eth/v1/builder/states/head/expected_withdrawals'   -H 'accept: application/json'| python3 -m json.tool
{
    "execution_optimistic": false,
    "data": [
        {
            "index": 1,
            "validator_index": 1,
            "address": "0xAbcF8e0d4e9587369b2301D0790347320302cc09",
            "amount": 1
        }
    ]
}

HiroyukiNaito · 2024-03-15T15:07:04Z

Hi, I almost done (exception handlings are not yet) , but, Development network ./lodestar dev --genesisValidators 8 --reset --rest.namespace "*" always returns http://localhost:9596/eth/v1/builder/states/head/expected_withdrawals empty array. How do I get getExpectedWithdrawals method value on this environment?
Do I need to implement on my mainnet environment for conforming the logic?

ensi321 · 2024-03-18T12:26:36Z

Do I need to implement on my mainnet environment for conforming the logic?

I recommend testing this kind of features by running it as a beacon node on mainnet or other public testnet like holesky. I admit it might be hard for contributors who have limited hardwares.

Hi, I almost done (exception handlings are not yet) , but, Development network ./lodestar dev --genesisValidators 8 --reset --rest.namespace "*" always returns http://localhost:9596/eth/v1/builder/states/head/expected_withdrawals empty array. How do I get getExpectedWithdrawals method value on this environment?

This is expected. The reason why getExpectedWithdrawals() is returning empty list is because

The validators don't have eth1 withdrawal credentials so this condition fails

lodestar/packages/state-transition/src/block/processWithdrawals.ts

Line 85 in f672c10

if (!hasEth1WithdrawalCredential(validator.withdrawalCredentials)) {
The validators are not qualified for full or partial withdrawals. The latter is because their balance is 32ETH. See this and this

The current dev command does a poor job to initiate validators that meet these two requirements.

If running a beacon node on mainnet/testnet is too much for you, and wish to test this locally with dev command, I recommend to modify the codebase as follow such that validators can meet the two requirements above:

Change this line

lodestar/packages/cli/src/cmds/dev/handler.ts

Line 63 in f672c10

const {state} = nodeUtils.initDevState(config, validatorCount, {genesisTime, eth1BlockHash});

to const {state} = nodeUtils.initDevState(config, validatorCount, {genesisTime, eth1BlockHash, withEth1Credentials: true});
Change this line

lodestar/packages/beacon-node/src/node/utils/interop/deposits.ts

Line 37 in f672c10

amount: MAX_EFFECTIVE_BALANCE,

to amount: MAX_EFFECTIVE_BALANCE + 1,

Lastly run ./lodestar dev --genesisValidators 8 --reset --rest.namespace "*" --params.ALTAIR_FORK_EPOCH 0 --params.BELLATRIX_FORK_EPOCH 0 --params.CAPELLA_FORK_EPOCH 0 --terminal-total-difficulty-override 0 to activate cappella fork.

I ran this locally and the api returns getExpectedWithdrawals error - Do not know how to serialize a BigInt TypeError: Do not know how to serialize a BigInt.

HiroyukiNaito · 2024-03-21T16:22:57Z

@ensi321 Thanks! I will resume this soon.

nflaig · 2024-06-13T12:40:07Z

@HiroyukiNaito Are you planning to continue on this? Since we recently refactored our api framework, most of the changes in this PR are outdated, I suggest to just open a new PR and continue discussion there.

I recommend testing this kind of features by running it as a beacon node on mainnet or other public testnet like holesky. I admit it might be hard for contributors who have limited hardwares.

Just a side note on this, it's pretty easy to spin up a mainnet node on any device, it just takes 1-2 min on my laptop to startup and be synced to head, you can just run

./lodestar beacon --checkpointSyncUrl https://beaconstate-mainnet.chainsafe.io --forceCheckpointSync --execution.engineMock

This requires ~5-6GB RAM but other than that it's pretty lightweight.

nflaig · 2024-10-10T21:01:19Z

this PR has too many conflicts since we did a major refactoring of the api layer, it makes sense to tackle this in a new PR

HiroyukiNaito requested a review from a team as a code owner March 6, 2024 10:46

HiroyukiNaito changed the title ~~feat: Implement expected withdrawals API~~ feat: implement expected withdrawals API Mar 6, 2024

ensi321 requested changes Mar 11, 2024

View reviewed changes

HiroyukiNaito marked this pull request as draft March 11, 2024 07:33

Added a expected withdrawals API interface

06f0039

HiroyukiNaito force-pushed the hnaito_ew_03062024 branch from 2207ec4 to 06f0039 Compare March 13, 2024 10:05

HiroyukiNaito added 2 commits March 14, 2024 18:57

Added router definition

8cfb479

Changed the reponse as snake case

cfb3704

Added logic

7359cad

nflaig closed this Oct 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement expected withdrawals API #6514

feat: implement expected withdrawals API #6514

HiroyukiNaito commented Mar 6, 2024 •

edited

Loading

HiroyukiNaito commented Mar 6, 2024 •

edited

Loading

ensi321 commented Mar 7, 2024

HiroyukiNaito commented Mar 7, 2024 •

edited

Loading

HiroyukiNaito commented Mar 8, 2024

codecov bot commented Mar 8, 2024 •

edited

Loading

HiroyukiNaito commented Mar 8, 2024

ensi321 left a comment

ensi321 commented Mar 11, 2024

HiroyukiNaito commented Mar 11, 2024

HiroyukiNaito commented Mar 13, 2024 •

edited

Loading

ensi321 commented Mar 14, 2024

HiroyukiNaito commented Mar 14, 2024

HiroyukiNaito commented Mar 15, 2024

ensi321 commented Mar 18, 2024 •

edited

Loading

HiroyukiNaito commented Mar 21, 2024

nflaig commented Jun 13, 2024

nflaig commented Oct 10, 2024

feat: implement expected withdrawals API #6514

feat: implement expected withdrawals API #6514

Conversation

HiroyukiNaito commented Mar 6, 2024 • edited Loading

HiroyukiNaito commented Mar 6, 2024 • edited Loading

ensi321 commented Mar 7, 2024

HiroyukiNaito commented Mar 7, 2024 • edited Loading

HiroyukiNaito commented Mar 8, 2024

codecov bot commented Mar 8, 2024 • edited Loading

Codecov Report

HiroyukiNaito commented Mar 8, 2024

ensi321 left a comment

Choose a reason for hiding this comment

ensi321 commented Mar 11, 2024

HiroyukiNaito commented Mar 11, 2024

HiroyukiNaito commented Mar 13, 2024 • edited Loading

ensi321 commented Mar 14, 2024

HiroyukiNaito commented Mar 14, 2024

HiroyukiNaito commented Mar 15, 2024

ensi321 commented Mar 18, 2024 • edited Loading

HiroyukiNaito commented Mar 21, 2024

nflaig commented Jun 13, 2024

nflaig commented Oct 10, 2024

HiroyukiNaito commented Mar 6, 2024 •

edited

Loading

HiroyukiNaito commented Mar 6, 2024 •

edited

Loading

HiroyukiNaito commented Mar 7, 2024 •

edited

Loading

codecov bot commented Mar 8, 2024 •

edited

Loading

HiroyukiNaito commented Mar 13, 2024 •

edited

Loading

ensi321 commented Mar 18, 2024 •

edited

Loading