Nimbus CL < > Lodestar VC incompatibility

[barnabasbusa]

Describe the bug

We are in the process to test cross beacon <> validator client compatibility, and found a bug when testing nimbus CL with lodestar VC.

Lodestar reports:

Apr-05 14:25:06.155[]                error: Error proposing block slot=29, validator=0xb05c…679b - JSON expected key slot is undefined - Failed to produce block
Error: JSON expected key slot is undefined - Failed to produce block
    at ContainerType.fromJson (/usr/app/node_modules/@chainsafe/ssz/src/type/container.ts:347:15)
    at Object.fromJson (file:///usr/app/packages/api/src/utils/types.ts:156:44)
    at Object.fromJson (file:///usr/app/packages/api/src/utils/types.ts:198:15)
    at Object.fromJson (file:///usr/app/packages/api/src/beacon/routes/validator.ts:778:91)
    at Object.request [as produceBlockV3] (file:///usr/app/packages/api/src/utils/client/client.ts:91:50)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at BlockProposingService.produceBlockWrapper (file:///usr/app/packages/validator/src/services/block.ts:205:17)
    at BlockProposingService.createAndPublishBlock (file:///usr/app/packages/validator/src/services/block.ts:144:29)
    at async Promise.all (index 0)

Nimbus reports:

nothing out of the ordinary

Snooper between cl <> vc reports:

nothing out of the ordinary

Expected behavior

I would expect all client combinations would work.

Steps to reproduce

config.yaml:

participants:
  - cl_type: nimbus
    vc_type: lodestar
additional_services:
  - dora
snooper_enabled: true
global_log_level: debug

kurtosis run github.com/kurtosis-tech/ethereum-package --args-file config.yaml

Additional context

Current BN <> VC Compatibility list tracker

Operating system

Linux

Nimbus version or commit hash

statusim/nimbus-eth2:multiarch-latest - sha256:8e79e02b37351bdcf0a4b07c9cb5a8dd23afdf83144a0d948644b72884cdd10d

cc: @pk910

[barnabasbusa]

Same issue: ChainSafe/lodestar#6636

[nflaig]

A possible reason for this could be that Nimbus is sending execution_payload_blinded as a string value instead of boolean as per spec.

See ChainSafe/lodestar#6636 (comment), maybe someone from Nimbus can confirm this, otherwise would require more logs / information about the response payload.

[kinsi55]

What nflaig said is correct and will also cause failed proposals with Lighthouse

[tersec]

#6204