Serialize and deserialize protocol buffers #2667

kyleconroy · 2022-03-03T06:28:11Z

As outlined in #447, TinyGo should support common serialization formats. While most people may think of JSON, I personally want TinyGo to support protocol buffers. This issue is meant to track my progress on adding Protocol Buffer support to TinyGo.

Design

The default Protocol Buffer package (https://pkg.go.dev/google.golang.org/protobuf) makes heavy use of reflection. TinyGo only supports a small amount of the reflect package. Heavy use of reflection leads to slow code, so the folks at PlanetScale wrote vtprotobuf, which generates explicit serialization / deserialization code. This code works today with TinyGo. However, it requires the existing, slow code to compile.

So the plan is simple: Support compiling the code generated by protoc-gen-go and running and of the code contained in init() functions. Use the code generated by vtprotobuf to serialize and deserialize the data.

Roadmap

Compile code generated by https://github.com/planetscale/vtprotobuf
Compile code generated by protoc-gen-go
- feat: Add stubs for more of the reflect package #2624
- (constant -4 of type uintptr) overflows uintptr #2806
Import code generated by protoc-gen-go
- unknown import: env::sync/atomic.StoreUint32 has not been defined #2652
- Implement the PkgPath() method on reflect.Type #2668

The text was updated successfully, but these errors were encountered:

paralin · 2023-03-04T05:19:44Z

@kyleconroy This is a great idea, I'm using vtprotobuf a lot currently and it works great. Having some kind of auto-transpiler that automatically converts the reflection-based protobuf-go to use vtprotobuf would be great.

Quick question: as of March 2023, it seems like most of the reflection features are supported - is there anything missing for the existing reflection version to work currently?

dgryski · 2023-03-23T05:56:21Z

And even more reflect is working, including the last bug on you roadmap.

gitslav · 2023-06-15T00:44:23Z

@kyleconroy @dgryski looking at the roadmap is it time for the confetti emoji?

rockwotj · 2023-11-15T03:17:12Z

So it seems that the default protobuf compiler is not supported still even with the improvements to reflection. I have yet to test vtprotobuf

% ./wasmtime-v14.0.4-x86_64-linux/wasmtime run protobuf-example.wasm
panic: unimplemented: (reflect.Type).MethodByName()
Error: failed to run main module `protobuf-example.wasm`

Caused by:
    0: failed to invoke command default
    1: error while executing at wasm backtrace:
           0: 0x5a15 - <unknown>!runtime._panic
           1: 0x5939 - <unknown>!(reflect.rawType).MethodByName
           2: 0x6484 - <unknown>!(*reflect.rawType).MethodByName
           3: 0x103253 - <unknown>!(*google.golang.org/protobuf/internal/impl.MessageInfo).makeStructInfo
           4: 0xb95e6 - <unknown>!(*google.golang.org/protobuf/internal/impl.MessageInfo).init
           5: 0xb8e02 - <unknown>!google.golang.org/protobuf/proto.protoMethods
           6: 0x34c67 - <unknown>!(google.golang.org/protobuf/proto.MarshalOptions).marshal
           7: 0x2d45a - <unknown>!runtime.run$1
           8: 0x2cf76 - <unknown>!runtime.run$1$gowrapper
           9:  0xdc1 - <unknown>!tinygo_launch
          10: 0x2ce48 - <unknown>!_start
       note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information
    2: wasm trap: wasm `unreachable` instruction executed

rockwotj · 2023-11-21T18:51:07Z

FWIW I have successfully used https://github.com/planetscale/vtprotobuf to marshal/unmarshal protobuf in tinygo -target=wasi

(this can be closed then?)

wrnlb666 · 2024-04-08T14:56:37Z

How are you able to use vtprotobuf with tinygo? Isn't the generated code from vtprotobuf depends on the result of protobuf-go? I am still getting the same error with my _vtproto.pb.go file.

rockwotj · 2024-04-08T16:20:26Z

How are you able to use vtprotobuf with tinygo? Isn't the generated code from vtprotobuf depends on the result of protobuf-go?

The reflection methods are stubbed out, so it still compiles and the vtprotobuf version doesn't invoke the reflection methods, so it works fine for me.

I am still getting the same error with my _vtproto.pb.go file.

Post the error and make sure you're on the latest tinygo version

wrnlb666 · 2024-04-09T07:45:51Z

I am on the latest tinygo v0.31.1, and I am also using it with wasmtime. The error I got is exactly the same with the one you posted before without using vtprotobuf. Did you change any of the code generated by protoc --go_out ?

rockwotj · 2024-04-09T08:28:07Z

no modifications needed. Can you post a reproduction?

wrnlb666 · 2024-04-09T16:31:26Z

I am so sorry to bother u, it was me making a dumb mistake causing the error, I fixed the error and it works totally fine. Thank you so much.

paralin · 2024-04-10T22:59:15Z

vtprotobuf generates static code for Marshal and Unmarshal. We just need a version of protoc-gen-go that does not import reflect nor any of the "heavy" protobuf packages that add a lot of complexity to the resulting binary.

I have forked to protoc-gen-go-lite here and have already dropped everything but the protoc plugin, and am working on adjusting the compiler to generate bare structs without referencing reflect: aperturerobotics/protobuf-go-lite#1

Please join the effort there as this is intended to make it easier to build lightweight tinygo protobuf projects.

vtprotobuf generates static code for Marshal and Unmarshal. We just need a version of protoc-gen-go that does not import reflect nor any of the "heavy" protobuf packages that add a lot of complexity to the resulting binary. tinygo-org/tinygo#2667 Drop all unused code and fix linter warnings. Signed-off-by: Christian Stewart <christian@aperture.us>

vtprotobuf generates static code for Marshal and Unmarshal. We just need a version of protoc-gen-go that does not import reflect nor any of the "heavy" protobuf packages that add a lot of complexity to the resulting binary. tinygo-org/tinygo#2667 Drop all unused code and fix linter warnings. Drop protoimpl.EnforceVersion lines Add link to vtprotobuf-lite Drop weak fields, extensions, json marshal, errors pkg. Signed-off-by: Christian Stewart <christian@aperture.us>

vtprotobuf generates static code for Marshal and Unmarshal. We just need a version of protoc-gen-go that does not import reflect nor any of the "heavy" protobuf packages that add a lot of complexity to the resulting binary. tinygo-org/tinygo#2667 Drop all unused code and fix linter warnings. Drop protoimpl.EnforceVersion lines Add link to vtprotobuf-lite Drop weak fields, extensions, json marshal, errors pkg. Drop testprotos. Signed-off-by: Christian Stewart <christian@aperture.us>

vtprotobuf generates static code for Marshal and Unmarshal. We just need a version of protoc-gen-go that does not import reflect nor any of the "heavy" protobuf packages that add a lot of complexity to the resulting binary. tinygo-org/tinygo#2667 Drop all unused code and fix linter warnings. Drop protoimpl.EnforceVersion lines Add link to vtprotobuf-lite Drop weak fields, extensions, json marshal, errors pkg. Drop testprotos. Hand-write String() for timestamp and duration. Signed-off-by: Christian Stewart <christian@aperture.us>

paralin · 2024-04-11T06:33:39Z

I have completed protobuf-go-lite which is a reflection-free fork of protobuf-go merged with vtprotobuf and protoc-gen-go-json.

Library: https://github.com/aperturerobotics/protobuf-go-lite
Example: https://github.com/aperturerobotics/protobuf-project/tree/protobuf-lite

RPCs are available as well with starpc: https://github.com/aperturerobotics/starpc

protobuf-go-lite now supports reflection-free protobuf and protojson encoding!

This is a version of protobuf-go that does not use reflection. Reflection adds weight to the compiled binary and is not supported fully by tinygo. We can eliminate the requirement on reflect with vtprotobuf, which generates static code for Marshal and Unmarshal. tinygo-org/tinygo#2667 This commit merges vtprotobuf and protobuf-go-lite into a single repo. This allows tightly integrating the protobuf-go compiler and the vtprotobuf compiler, generating a single .pb.go file instead of multiple files. This also allows radical simplifications of the protobuf compiler plugin and the generated vtprotobuf code, eliminating the need for extra wrapper code. Major changes from upstream: - Drop all unused code and fix linter warnings. - Drop pooling - Add link to protobuf-go-lite - Drop grpc - Drop exts - Drop references to ProtoPkg - Dropped pool from vtprotobuf - Dropped MessageSet (deprecated) - Dropped any (requires reflect) - Dropped fieldmask (requires reflect) - Dropped the -wrap option for vtprotobuf - Allow import paths without slashes in protogen - Derive Go package name from the proto3 package name - Derive Go import path from the proto3 adjacency to Go packages - Generate a String() for enums which does not use reflect Signed-off-by: Christian Stewart <christian@aperture.us>

This is a reflection-less version of protobuf-go with vtprotobuf. Now starpc works without the reflect package, which is much better for supporting platforms like WebAssembly and tinygo, where every import matters in terms of managing binary size, and the reflection support may be limited. Reflection adds weight to the compiled binary and is not supported fully by tinygo. We can eliminate the requirement on reflect with vtprotobuf, which generates static code for Marshal and Unmarshal. See: tinygo-org/tinygo#2667 See: aperturerobotics/protobuf-go-lite@e0016a1 Signed-off-by: Christian Stewart <christian@aperture.us>

This commit implements protojson support by merging in a fork of: https://github.com/TheThingsIndustries/protoc-gen-go-json protoc-gen-go-json has been heavily modified to integrate cleanly with vtprotobuf and the rest of protobuf-go-lite. Currently proto2 and proto3opt are not supported w/ the json feature. This implements reflection-free protojson support for tinygo: tinygo-org/tinygo#2667 (comment) Signed-off-by: Christian Stewart <christian@aperture.us>

This was referenced Mar 3, 2022

Implement the PkgPath() method on reflect.Type #2668

Closed

feat: Codegen plugins, powered by WASM sqlc-dev/sqlc#1470

Closed

deadprogram added the enhancement New feature or request label Mar 12, 2022

This was referenced May 13, 2022

compiler: implement deferred calls to sync/atomic.* #2821

Closed

compiler: make sure zero-sized array types are as small as possible #2854

Merged

abourget mentioned this issue Jan 9, 2023

protobuf v1.28.1 compile issue + missing MapOf in reflect #3376

Open

rockwotj mentioned this issue Aug 30, 2023

transform-sdk: Introduce sr package redpanda-data/redpanda#13049

Merged

7 tasks

knqyf263 mentioned this issue Nov 7, 2024

Allow using together with protoc-gen-go knqyf263/go-plugin#67

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Serialize and deserialize protocol buffers #2667

Serialize and deserialize protocol buffers #2667

kyleconroy commented Mar 3, 2022 •

edited

Loading

paralin commented Mar 4, 2023

dgryski commented Mar 23, 2023

gitslav commented Jun 15, 2023

rockwotj commented Nov 15, 2023

rockwotj commented Nov 21, 2023 •

edited

Loading

wrnlb666 commented Apr 8, 2024

rockwotj commented Apr 8, 2024

wrnlb666 commented Apr 9, 2024

rockwotj commented Apr 9, 2024

wrnlb666 commented Apr 9, 2024

paralin commented Apr 10, 2024

paralin commented Apr 11, 2024 •

edited

Loading

Serialize and deserialize protocol buffers #2667

Serialize and deserialize protocol buffers #2667

Comments

kyleconroy commented Mar 3, 2022 • edited Loading

Design

Roadmap

paralin commented Mar 4, 2023

dgryski commented Mar 23, 2023

gitslav commented Jun 15, 2023

rockwotj commented Nov 15, 2023

rockwotj commented Nov 21, 2023 • edited Loading

wrnlb666 commented Apr 8, 2024

rockwotj commented Apr 8, 2024

wrnlb666 commented Apr 9, 2024

rockwotj commented Apr 9, 2024

wrnlb666 commented Apr 9, 2024

paralin commented Apr 10, 2024

paralin commented Apr 11, 2024 • edited Loading

kyleconroy commented Mar 3, 2022 •

edited

Loading

rockwotj commented Nov 21, 2023 •

edited

Loading

paralin commented Apr 11, 2024 •

edited

Loading