API format/protocol redux

[djrtwo]

As brought up by @skmgoldin on that last eth2 call, there is a strong desire from application developers for eth2 clients to provide a standard/conformant API.

We have this repo which is intended to be a collection of RESTful HTTP apis for various domains on the eth2 client. The APIs found in this repo are currently written in OpenAPI 3.0 in YAML. Some

So far, this repo only defines a beacon<->validator API for the purposes of writing generic "validator clients", but the intention is to host many other user-level APIs for the beacon chain (phase 0) and shard chains (phase 1). To date, the definition of this API has been largely collaborative across clients and the promise of standard/interoperable VCs seems in reach.

Prysmatic has a number of user-level APIs defined in prysmaticlabs/ethereumapis with some decent traction with block explorers and other applications, and Lighthouse has a number of APIs that have also begun to be used by various applications (link?).

There are a few things we need to work through:

1. The primary protocol and format that APIs take on in this repo
2. The user-level "beacon-node" APIs that we choose to integrate [BeaconNode APIs #25]
3. Auto-conversions and CI setup that we use to ensure that these APIs are standardized and available in multiple formats and transports as desired to be integrated by clients and used by users [API conversions and CI #26]

We'll tackle [1] here and address the other two in separate issues [#25, #26]

---

This has been addressed in a number of past conversations and the general consensus was to land on the OpenAPI format.

@paulhauner put together a argument for continuing to write the APIs in OpenAPI, and it seems that it is still the general consensus is to do so.

The alternatives are to define the API (1) in protobufs or (2) in SSZ and provide standard conversion between all three. Some arguments are for/against are made here and were discussed briefly on the last eth2 call.

---

If we choose to go the OpenAPI native route, we should do an investigation of the types currently used in this repo to ensure that they are sound for our purposes and being easily converted in alternative contexts.

Types to discuss:

• Format for bytes types
  • In eth1, bytes are largely represented as 0x00 hex-strings in APIs which is why this repo originally used the format
  • Users and developers are largely used to seeing public keys and addresses displayed in this format
  • OpenAPI recommends base64 representation and thus most ready-built parsers/converters will have native support for base64
• Format for integer types
  • This repo is currently representing uint64 as a string type (instead of native json integer) because some applications like Postman and Swagger round down integer values greater than int53
  • Native integer type is more natural for this usecase and in fact Lighthouse is using it today (as opposed to the string types defined here).

[mcdee]

I suggest the discussion here wait for #25 to be fleshed out: selecting the format before understanding the requirements is putting the cart before the horse.

[djrtwo]

I want to remind everyone of the comments that led to this restful http decision almost a year ago. The arguments made then are just as valid today

ethereum/consensus-specs#1012 (comment)

[prestonvanloon]

Is the minimum protocol still up for debate? @prysmaticlabs is on board to support restful http and I haven't heard any strong advocates for anything else recently (json-rpc, grpc, soap, etc).

[djrtwo]

The arguments for the protocol and the canonical format to define the API in are a bit intertwined in that linked thread.

Notably well definedness of the API, spec maintainability, simplicity, and wide support are desired features for the canonical definitions to be used in the spec and are discussed in the above.

[mcdee]

This repo is currently representing uint64 as a string type (instead of native json integer) because some applications like Postman and Swagger round down integer values greater than int53

Worth pointing out that this is a javascript limitation, rather than that of any particular application (see e.g. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER). It may be that some JSON parsers work around this, but it's not standard.

[djrtwo]

Closing!

Decided on API call to use OpenAPI with JSON as a first class citizen and document application/ssz where ssz types make sense. (see @mpetrunic's example here)

No action item on the repo as that's the current format. Keep SSZ types in mind as we develop the BN APIs