grpc / protobuf support

[StefH]

Grpc

Currently, gRPC functionality is supported via this project: https://github.com/cezarypiatek/GRPC-Mock-Server

Grpc support : request - reply

However, I want to discuss an idea here to add simple Grpc request-reply logic to WireMock.Net

This will look a bit like the GraphQL support which is added recently, combined with some ideas from MimePartMatcher.

However in case of protobuf, the binary protobuf bytes are converted to a json object. (using ProtoBufJsonConverter).

So that a normal JsonMatcher can be used.

Usage

Usage in C# will be like:

var ProtoDefinition = @"
syntax = "proto3";

package greet;

service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply);
}

message HelloRequest {
  string name = 1;
}

message HelloReply {
  string message = 1;
}
";

// Option 1: Set "useHttp2" to true to indicate that we need Http2 for Grpc
using var server = WireMockServer.Start(useHttp2: true);

// Option 2: Use "grpc://" that we need Http2 for Grpc
using var server = WireMockServer.Start(settings =>
{
    Urls = new [] { "grpc://localhost:9093" }
});

// ⭐ Define a request body json matcher (optional)
var jsonMatcher = new JsonMatcher(new { name = "stef" });

// Option 1️⃣ (define the ProtoDefinition for the request and the response)
server
    .Given(Request.Create()
        .UsingPost()
        .WithPath("/greet.Greeter/SayHello")
        
         // ⭐ Define the ProtoDefinition & the request message type & the matcher
        .WithBodyAsProtoBuf(ProtoDefinition, "greet.HelloRequest", jsonMatcher)
    )
    .RespondWith(Response.Create()
        .WithHeader("Content-Type", "application/grpc")

         // ⭐ Define the ProtoDefinition & the response message type
        .WithBodyAsProtoBuf(ProtoDefinition, "greet.HelloReply",
            // ⭐ Define the response as a object
            new
            {
                message = "hello stef"
            }
        )
        // ⭐ Set the correct Trailing Header (this is required for Grpc)
        .WithTrailingHeader("grpc-status", "0")
        
        // ⭐ Optionally, it's possible to use transformer
        .WithTransformer()
    );

// Option 2️⃣ (define the ProtoDefinition for the mapping)
server
    .Given(Request.Create()
        .UsingPost()
        .WithPath("/greet.Greeter/SayHello")
        
         // ⭐ Define the request message type & the matcher
        .WithBodyAsProtoBuf("greet.HelloRequest", jsonMatcher)
    )
    // ⭐ Define the ProtoDefinition at mapping level
    .WithProtoDefinition(ProtoDefinition)
    .RespondWith(Response.Create()
        .WithHeader("Content-Type", "application/grpc")

         // ⭐ Define the response message type
        .WithBodyAsProtoBuf("greet.HelloReply",
            new
            {
                message = "hello stef"
            }
        )
        .WithTrailingHeader("grpc-status", "0")
        .WithTransformer()
    );

// Option 3️⃣ (define the ProtoDefinition at server-level)
server
     // ⭐ Add a ProtoDefinition named "my-greeter" at server-level
    .AddProtoDefinition("my-greeter", ProtoDefinition)
    .Given(Request.Create()
        .UsingPost()
        .WithPath("/greet.Greeter/SayHello")
        .WithBodyAsProtoBuf("greet.HelloRequest", jsonMatcher)
    )
    // ⭐ Define the identifier from the ProtoDefinition at mapping level
    .WithProtoDefinition("my-greeter")
    .RespondWith(Response.Create()
        .WithHeader("Content-Type", "application/grpc")
        .WithBodyAsProtoBuf("greet.HelloReply",
            new
            {
                message = "hello stef"
            }
        )
        .WithTrailingHeader("grpc-status", "0")
        .WithTransformer()
    );

Notes

A. It's not yet possible to get the request message as a (Json) object, so you cannot yet use transformer on {{request.BodyAsJson}}
A. It is also possible to use transformer on the request (as json object).

     .WithBodyAsProtoBuf("greet.HelloReply",
            new
            {
                message = "hello {{request.BodyAsJson.name}}"
            }
        )

B. Maybe the ProtoDefinition can also defined at server-level, in that case the WithBodyAsProtoBuf will not require a full proto definition, but just a name-reference or guid.
B. It is possible to define the ProtoDefinition at server-level (see option 3)

PR

#1047

Example

I did test it in PostMan, and it works fine:

[StefH]

Tagging:
@cezarypiatek
@mcopjan
@jzabroski
@kharabasz
@sergioprates
@robertmircea
@MindaugasLaganeckas
@MJLHThomassenHadrian
@mark-hingston
@basdijkstra
@jflevesque-genetec
@sameena-ops
@leolplex
@alastairtree
@NoahLerner
@phillee007
@vitaliydavydiak
@akamud
@kashifsoofi
@ggradnig
@eli-darkly
@cal-schleupen
@oleg-nenashev

[basdijkstra]

Just wanted to say well done, @StefH ! That looks very good. I think having the ability to specify the proto definition at the server level would be a useful thing to have, especially when it contains multiple request / reply definitions. You can then load the proto definition once and define specific responses for individual requests in separate mock definitions.

[jzabroski]

This is great news.

[StefH]

I think having the ability to specify the proto definition at the server level would be a useful thing to have, especially when it contains multiple request / reply definitions. You can then load the proto definition once and define specific responses for individual requests in separate mock definitions.

This is now also possible, see example "option 3️⃣".

[basdijkstra]

Awesome! Thanks for all the good work, @StefH .

[jzabroski]

I'm curious to see what WithBodyAsProtoBuf(ProtoDefinition, "rpc Method") does internally. Are you compiling an actual proto definition file? If yes, how does it handle dependent proto definition (import statement)?

[StefH]

It's actually rpc Message Type, not rpc Method.

And internally, I'm using https://github.com/StefH/ProtoBufJsonConverter to compile the actual proto definition to C# classes and then load these classes.

Imports should work fine. (See also https://github.com/StefH/ProtoBufJsonConverter/tree/main/examples/ConsoleApp)

[jzabroski]

Got it. Let me see if I can find any bugs in your approach. I'm on vacation this week so likely 7-10 days from doing anything.

[StefH]

@jzabroski : did you have time to review the code / design ?

[jflevesque-genetec]

I haven't played much with GRPC, but I feel like option 3 is almost a must have. I've often seen repos to define rpc messages so it can be reused across teams, so having a way to import the full definition seems it would be used the most. Having a way to also define per message can also be useful when prototyping stuff, so I see 1 and 3 like something that would be used.

[StefH]

gRPC support is implemented in PR (#1047)

Read this blog post on the details and how to use it: https://mstack.nl/blogs/wiremock-net-grpc/

[cezarypiatek]

Note that there is a side project GRPC-Mock-Server which enables gRPC support, however using this approach requires a separate product which makes it more difficult to integrate and use this in your existing integration tests using WireMock.Net.

1. Thank you for mentioning my solution.
2. Can you elaborate a bit on what exactly you mean by which makes it more difficult to integrate and use this in your existing integration tests. What problem have you encountered? The GRPC Mock Server is just an extra layer on top of WireMock.Net net that translates between GRPC and the REST protocol.

[StefH]

I think that using this extra layer makes it somewhat more difficult to use because the user needs to include a new product and use a different syntax to be able to use grpc.
I tried to integrate it more in WireMock.Net so that the usage is simpler and known for users.

However if users need streaming support : then your solution uses a clever approach by using scenario's.

[cezarypiatek]

Thanks for the explanation. I agree that having all features in one product with consistent API simplifies the application.