Generate a JSON client

[alecthomas]

Correct me if I'm wrong, but it seems like the auto-generated client impl is hard-coded to use protobufs over the wire:

  geoIP(request: GeoIPReq): Promise<GeoIPResp> {
    const data = GeoIPReq.encode(request).finish();
    const promise = this.rpc.request("service.Service", "geoIP", data);
    return promise.then(data => GeoIPResp.decode(new Reader(data)));
  }

Would it be possible to generate a parallel client impl that uses JSON over the wire instead?

[alecthomas]

I think ideally it would be possible to pass a protoc option to specify which wire formats to generate. This would be nice because JSON-only clients then wouldn't need to import the JS protobuf package.

[stephenh]

@alecthomas yeah, that is a good point. We'd only needed protobuf/twirp for our rpc/services, so was all we've implemented so far...

The service impl code is fairly straightforward:

https://github.com/stephenh/ts-proto/blob/master/src/main.ts#L725

And I think really it'd just be generateRegularRpc function that could be the first place to add some conditional "if output type" type logic:

https://github.com/stephenh/ts-proto/blob/master/src/main.ts#L695

Would you be interesting in tacking a crack at this?

I might be able to hack on this at some point, but all the better if you'd like to, and then could verify the impl / usage / etc. within your project / ecosystem.

[isherman]

I started to take a look at what would be involved here.

Unfortunately, protobuf.js does not support encoding/decoding to/from canonical proto3 JSON: protobufjs/protobuf.js#1304

It looks like it's partially implemented, but lacking support for important well-known-types like Timestamp and Duration.

And google's JS protobuf library has no support at all: protocolbuffers/protobuf-javascript#95

Do you see any way forward?

[isherman]

It looks like there are some open PRs that would help, e.g. protobufjs/protobuf.js#1258

Maybe the way forward is to implement this using protobufjs, without support for those well-known types, and then either fork protobuf.js, patch it at runtime, or wait for it to be fixed upstream.

[stephenh]

Hey @isherman apologies for the really late reply here. It'd be awesome if you could poke around at this.

Per the protobuf.js PRs, I think that shouldn't be a problem, because ts-proto implements its own direct support for the proto3 JSON protocol.

I.e. here is where we generate Message.fromJSON methods and handle timestamps:

https://github.com/stephenh/ts-proto/blob/master/src/main.ts#L759

We don't handle encoding/encoding Durations yet.

So, in theory, a JSON client would just be some glue that takes the service interfaces and makes a <Service>JsonClient that uses ts-proto's existing fromJSON/toJSON methods to serde the messages.

[amaury1093]

To make it really generic, we could also:

1. Change the signature of the request function to take

interface Rpc {
-  request(service: string, method: string, data: Uint8Array): Promise<Uint8Array>;
+  request<Req extends Codec, Res extends Codec>(service: string, method: string, req: Req): Promise<Res>;
}

where Codec is just:

// This is the interface that's already implemented by all types in ts-proto,
// i only gave it a name
interface Codec<T> {
  encode(message: T, writer: Writer = Writer.create()): Writer;
  decode(input: Uint8Array | Reader, length?: number): T;
  fromJSON(object: any): T;
  fromPartial(object: DeepPartial<T>): T;
  toJSON(message: T): unknown;
}

2. Then, it would be the user's responsability, when implementing the Rpc interface, to decide which wire format to use:

const implWithUint8Array: Rpc = {
  request<Req, Res>(service: string, method: string, req: Req): Promise<Res> {
    const data = Req.encode(req).finish();

    // Do whatever you used to do before, e.g. gRPC
    const wireResponse = ...;

    return Res.decode(new Reader(wireResponse))
  }
}

const implWithJSON: Rpc = {
  request<Req, Res>(service: string, method: string, req: Req): Promise<Res> {
    const data = Req.toJSON(req);

    // Do whatever you used to do before, e.g. gRPC with JSON
    const wireResponse = ...;

    return Res.fromJSON(wireResponse);
  }
}

Advantages:

• no need for an additional protoc option
• allow users to use Uint8Array, JSON for wire transfer... or do other more fancy stuff.

[disjukr]

@amaurymartiny Agree. In my case, I also need the Res codec too.

interface Rpc {
-  request(service: string, method: string, data: Uint8Array): Promise<Uint8Array>;
+  request<Req extends Codec, Res extends Codec>(
+    service: string,
+    method: string,
+    data: Uint8Array,
+    req: Req,
+    res: Res
+  ): Promise<Uint8Array>;
}

[isherman]

Thanks for picking this up @disjukr, @amaurymartiny, @stephenh.
I lost momentum on #106 when the motivation for JSON serialization became less clear in my application. Will close that now.

Your approach looks more general, an improvement for sure. One tradeoff is it's a breaking change.
But I'm no longer a stakeholder here, so I defer to the maintainer!

[vtolstov]

any moving forward for this issue?

[corwinsheahan-wf]

@stephenh

We have a use case for sending JSON data via the Rpc interface. We have sort of worked around that in our Rpc implementation by deserializing the request and then serializing it to JSON, then we deserialize our response JSON and re-serialize it back into binary. That's not ideal and requires some awkward hoops to jump through.

Additionally, and more importantly, the protobuf.js file that the binary serialization/deserialization relies upon uses the eval API, which is unsafe for use in the browser and violates our Content-Security-Policy. We can work around this by using options to skip generating the binary encode/decode methods and skip generating a client implementation. However, we'd really like to have the convenience of generated clients.

Would you be open to a contribution which implements something along the lines of either #106 or this suggestion? I imagine this would be gated behind a flag like generateGenericRpcClient or something like that (open to suggestions), so I hope that way we can implement this in a non-breaking way.

[corwinsheahan-wf]

Attempted an implementation of a generic client here