Consider moving ethapi out of internal

[prettymuchbryce]

Currently it seems there is no way to convert a *types.Block into the canonical JSON form using the go-ethereum package alone. This is due to the fact that ethapi resides in an Internal package and cannot be imported.

My understanding is that go-ethereum is the de-facto standard Go client for Ethereum. Whether that is a strict goal or not, I'm not sure. But Marshaling these common data structures into their canonical JSON forms would be a very useful feature.

System information

Geth version: master/HEAD
OS & Version: Windows/Linux/OSX
Commit hash : db79143

Expected behaviour

go-ethereum should allow downstream modules to marshal blocks and other common types into JSON.

Actual behaviour

ethapi.RPCMarshalBlock exists in an Internal package and cannot be imported by downstream packages.

[karalabe]

That ethapi package is a significant dumping ground. We could consider moving meaningful parts out though. @fjl?

[karalabe]

We've been discussing this a bit, but we don't fully understand the need for this. The ethapi package's role is mostly to act as a helper for the RPC package. RPC has it's own web3 quirky format, which sometimes merges a lot of different pieces of data together to form a single result json.

All in all, if you are writing your own Go code, it might be cleaner to stick to the individual Go types (e.g. block, transaction) and not use the convoluted web3 API specs. You should already be able to convert all Go Ethereum types into json by themselves.

Could you detail what you're trying to do exactly? Maybe that would help us understand better as to how that can be done, either without or with modification to the go-ethereum code.

[Pet3ris]

Hi @karalabe,

I can second the above request for making ethapi external, but I've needed that for a different reason which I can detail.

---

I'm trying to use go-ethereum's RPC API over a simulated back-end object as a library. I'm using a simulated back-end for smart contracts from another programming language and would like to have some fallback access to the simulated back-end state through the RPC API rather than having to implement each end-point one by one.

In order to set up the RPC API I'm using NewServer() and using a bi-directional pipe as the codec. The place where this approach fails is that APIs() are only available to full or light ethereum clients, but not the simulated back-end. The simulated back-end exposes GetAPIs() but this function is only available as part of internal/ethapi.

As a result I'm trying to effectively copy & paste the whole internal/api folder in my codebase but running into another related issue that I've filed.

Would love to also hear your guidance on how to achieve this and hope this helps support the overall thinking around the ethapi!

---

To summarize - from my perspective there should be a way to use APIs or GetAPIs with the simulated back-end, I wouldn't actually need the whole ethapi package to be exposed.

[prettymuchbryce]

Thank you for the consideration and taking the time to respond @karalabe.

You should already be able to convert all Go Ethereum types into json by themselves.

Can you show me an example of how to do that? I see the Block struct in core/types does not have json tags. Am I missing something?

Could you detail what you're trying to do exactly?

My use case specifically is that I would like to cache blocks fetched by go-ethereum/ethclient by serializing them into JSON in the same form they are returned from the RPC API. This way a JavaScript client could use a cached block or a block from web3 without needing separate code paths.

Given that there are client implementations in different language (Go, JavaScript, ...), it makes sense to me that there should be a canonical form to serialize/deserialize these structures for use between client implementations. In web3 this is very easy, because we can just JSON.stringify/parse a block returned from web3.

Let me know if I am misunderstanding something here.

[fjl]

@prettymuchbryce The ethclient is a bit special because it does not mirror the web3 RPC API exactly. Instead, it gives you a way to get the core/types data structures out of any implementation of the web3 RPC API. The core/types types cannot represent all the data returned by the API because the API assumes a slightly different data model.

If you want to make a generic cache, it would be much better to just decode the RPC results into json.RawMessage and cache that. You can do that by using rpc.Client instead of ethclient.Client.

[fjl]

In general, adding JSON marshaling support to types.Block is something we can do.

[fjl]

@Pet3ris running the web3 API on the simulated backend is very nice, and we do want to support it eventually. We're currently refactoring the overall structure of go-ethereum to make the high-level components more independent. I'll see about making package ethapi public after this work has completed.

[prettymuchbryce]

Makes sense. Thanks @fjl

[fjl]

The refactoring mentioned earlier is now complete, but I'll be busy with some other things for a while and can't take on moving internal/ethapi at this time.

[Pet3ris]

Thanks for the update @fjl! I've created another issue to track this specific question around using simulated back-end via json rpc #20212.