Proposal: Allowing WIT bindgen written in languages they are generating for

[Mossaka]

@ydnar and I originally posted this little proposal on this zulipchat post and I am raising an issue here for a bigger exposure.

Challenges

Currently, all the WIT binding generators are written in Rust to emit bindings in different languages. Given that the parsed WIT AST (in particular, Resolve struct) is represented in Rust data structures, this makes sense. However, this poses some challenges to language experts to develop and maintain WIT binding generators because they will have to know Rust well. This motivates the idea of developing bindgen in their own languages consuming resolved WIT packages. The question is how and in what format can we supply resolved WIT packages to bindgen written in different languages.

Details

wit-bindgen is a tool to generate guest language bindings for .wit files for languages that compile to Wasm components. It leverages wit-parser crate from wasm-tools to parse and interpret WIT packages. There is a clear difference between wit-bindgen and wit-parser:

1. wit-parser parses the input WIT files, resolves all the imports, including foreign imports, and represent all the worlds and interfaces in a struct called wit_parser::Resolved.
2. wit-bindgen uses fully-resolved WIT packages in wit_parser::Resolved to generate bindings. Roughly speaking, a new language bindgen needs to implement these Rust traits:
   1. wit_bindgen_core::Bindgen - lower and lift signatures from Wasm types to WIT types
   2. wit_bindgen_core::WorldGenerator - generate source code to files for WIT worlds
   3. wit_bindgen_core::InterfaceGenerator - generate source code to files for WIT interfaces

Ideas

1. Complete rewrite of wit-parser and wit-bindgen in each language. For example, wit-parser-go and wit-bindgen-go for WIT binding Go generator. See an example for wit-parser in Go by @jordan-rash
2. Reuse wit-parser as much as possible by serializing its exported modules (mainly wit_parser::Resolve) and deserialized to each binding generators in their respective languages for consuming. @alexcrichton has some insights into this.
3. @lann suggested to turn language generators to Wasm components by defining generator interfaces as WIT packages.

[cpetig]

I suggest to also consider exporting parts of the abi crate (which controls the lifting and lowering mechanisms; currently in move to wit-bindgen - see bytecodealliance/wasm-tools#1149 for details) to other languages as getting the ABI conventions right will be the more thankless part of language bindings generation.

[ydnar]

I think (2) makes sense, and (3) after a language has been bootstrapped.

I suggest to also consider exporting parts of the abi crate (which controls the lifting and lowering mechanisms; currently in move to wit-bindgen - see bytecodealliance/wasm-tools#1149 for details) to other languages as getting the ABI conventions right will be the more thankless part of language bindings generation.

Would it be possible to annotate the Resolve struct (or its JSON representation) with ABI information so target language generators can correctly lift/lower?

[lann]

As an extension to "turn language generators to Wasm components", if the generator interfaces can avoid resources [edit: actually I'm not sure this is even necessary] then bootstrapping could also be done by adapting the interfaces to some RPC system with widespread implementations like gRPC or JSON-RPC. This RPC adapter could itself be generated by the same system.

[ydnar]

Makes sense. Question—what would require bidirectional communication between wit-bindgen and, say, wit-bindgen-go? Versus, say, just piping JSON into another process?

I’m probably missing something! (asking because an RPC layer, even if autogenerated, is another order of complexity versus just parsing some intermediate representation)

[lann]

what would require bidirectional communication

It depends on how much wit-bindgen code you want to share with generators. With no sharing, this approach would be no better than approach 2 (serializing Resolve), but that leaves a lot of common functionality up to every bindings generator as alluded to in #640 (comment).

[ydnar]

what would require bidirectional communication

It depends on how much wit-bindgen code you want to share with generators. With no sharing, this approach would be no better than approach 2 (serializing Resolve), but that leaves a lot of common functionality up to every bindings generator as alluded to in #640 (comment).

Repeating question from above, in case it got lost: Would it be possible to annotate the Resolve struct (or its JSON representation) with ABI information so target language generators can correctly lift/lower?

[lann]

Would it be possible to annotate the Resolve struct (or its JSON representation) with ABI information so target language generators can correctly lift/lower?

I am not familiar enough with that code to be confident, but I suspect the answer is "possible but somewhat difficult, especially to maintain".

[ydnar]

Got it. How would you envision an RPC flow between wit-bindgen and a language specific tool? Specifically for ABI related code gen?

[lann]

Off the cuff, I might expect a "generator component" to have exports corresponding to e.g. WorldGenerator and Bindgen and imports corresponding to portions of what is currently in the public interface of wit-parser, e.g. some methods of Resolve.

[Mossaka]

The Pr to add Serde::Serialize to wit-parser has been merged! We are now able to consume the JSON output to write Go binding generator in Go, where @ydnar is making some progress at https://github.com/ydnar/wasm-tools-go