"Official" Wireshark dissector

[FlorianWolters]

Note: I know this is not the best place to post this kind of request, but I do not know where to post this elsewhere. Hopefully other people are also interested in this.

---

I do wonder what tools the folks at Google use internally to debug and analyze the Protobuf messages transferred over the wire (if using network communication).

The defacto standard to do network analyzes is the network tool Wireshark. The functionality of Wireshark can be extended with a Protocol Dissector (i.e. a plug-in) to support a (custom) protocol.

I've researched a bit and found the following dissectors for Protobuf:

• Wireshark Protobuf Dissector
  Dissector written in Lua. Does support proto2 and UDP (without any prefixing) only. Pretty clumsy usage and buggy visualization.
• Length Prefixed Protocol Buffer Dissector
  Seems to add length prefix support to the Lua dissector in order to support TCP. Haven't tested it.
• sst-dissector
  Undocumented C dissector. Haven't tested it.
• protobuf-wireshark
  The first C dissector around. Project seems dead. Haven't tested it.
• protobuf-wireshark (fork)
  See above.

There are plenty of requests for a working and user-friendly dissector on the WWW, e.g.:

• How to make Wireshark tool support Protocol Buffers?
• Protocol Buffer Wireshark Plugin
• Wireshark dissector for a Google Protocol Buffer defined protocol
• How to parse protobuf packets in Wireshark

I would be very interested in some kind of official Wireshark protocol dissector with the following features:

1. Supports both proto3 and proto2.
2. Supports both UDP and TCP (see point 3). The port ranges for both protocols should be configurable with the GUI of Wireshark (i.e. when to activate the dissector).
3. Supports the configuration of a custom prefix with length-prefix framing. The custom prefix is configured by:
   1. The total length (in bytes) of the custom prefix.
   2. The position of the length-prefix.
   3. The length (in bytes) of the length-prefix.
      That would allow to support TCP as well as custom protocols that define additional fields.
4. Supports both generic decoding as well as specific decoding (with the help of provided .proto files).
   1. generic: Displays a aessage as [numeric field id:value] pairs. This does not require the .proto file(s).
   2. specific: Displays a message as [field name:value] pairs. This requires the .proto file(s) for the specific protocol. The addition of .proto file(s) should be possible via the GUI of Wireshark and the directory path to the official Protobuf .proto files should be configurable as well (e.g. $PROTOBUF_ROOT/include/google/protobuf).

At the end of the day it would be very nice to have some kind of official protocol dissector contributed to Wireshark.

Are there any efforts towards an official Protobuf dissector for Wireshark? What does the community think? Is this considered useful? Or did I overlook an already existing solution that satisfies the requirements stated above?

[xfxyjwf]

In Google, to send protobuf over the wire, you will be using the rpc library (the internal version of http://grpc.io/) instead of writing your own socket code, and a set of tools/libraries are provided to inspect and diagnose the rpc traffic. For example, with a command line tool you can easily write up a proto message in text format and send it to an arbitrary rpc server and get the response in text format. Hardly anyone needs to inspect the network traffic to understand the proto data being sent and it's also very hard to do because the rpc library will compress and encrypt the proto data. People who work on the rpc library probably need to use wireshark or tcpdump to inspect the network traffic, but again they don't need to inspect the proto data (I know they send payload of 1 byte in their tests). So pretty much nobody in Google will need this protobuf dessector for Wireshark and I doubt it will ever be officially supported...

[ruudsieb]

Actually after reading the OP I tried using the Length Prefixed Protocol Buffer Dissector together with the protobuf_dissector (forked). And it was pretty easy to setup and get it to work in my context which is; Wireshark on Windows, TCP connection, length prefix (little endian, for this I made a small change on the lppb), proto2 message format.

[FlorianWolters]

@ruudsieb Did you get it to work with the proto3 language?

[ruudsieb]

Sorry, forgot to mention that, no, I'm using proto2 message format.

[jfjffilk]

@FlorianWolters I am also trying to write a wireshark plugin in c to parse google protocol buffers over TCP, but not finding proper source to know how. Have you found any solution for it?

[FlorianWolters]

@jfjffilk No, as mentioned in the OP there is currently no out-of-the-box solution working for the proto3 language. For general information how-to implement a Wireshark dissector refer to Chapter 9. Packet dissection of the official Wireshark documentation. But I strongly recommend to contact the author(s) of the Wireshark Protobuf Dissector and to analyse/fork/extend that project before beginning from scratch and implementing your own dissector. Please read the OP again for problems related to the existing Protobuf dissectors.

[ndandanov]

I would also be interested in an official Wireshark dissector for protobuf and perhaps gRPC.

I agree that it may not be of much use in Google's environment, however I think that many developers and testers would benefit from it.

[ndandanov]

Sorry for double-posting, but I tried a few things after posting above.

I found out, that in the official Wireshark repo on GitHub, there is a gRPC dissector - namely the wireshark/epan/dissectors/packet-grpc.c located here: https://github.com/wireshark/wireshark/blob/master/epan/dissectors/packet-grpc.c

It seems that it was officially added in the end of September this year:
https://github.com/wireshark/wireshark/commits/master/epan/dissectors/packet-grpc.c

Note that the original Wireshark repo is located on their website. Please check https://www.wireshark.org/develop.html.

I opted to see how this dissector is working and cloned the Wireshark repo:

git clone https://code.wireshark.org/review/wireshark

My system is a 64-bit Arch Linux, so I built Wireshark with the following commands:

./autogen.sh 
./configure 
make

(using this as a reference: https://www.wireshark.org/docs/wsdg_html_chunked/ChSrcBuildFirstTime.html)

It took around 20-30 minutes to build. The Wireshark version is:

I then ran it as root (in order to be able to capture any traffic) and started capturing the loopback interface.

I cloned, built and started the official gRPC hello world server and client from the grpc-java repo:
https://github.com/grpc/grpc-java/tree/master/examples

The console outputs are, as expected:

The captured packets in Wireshark are:

You can download the .pcapng file from my repo:
https://github.com/ndandanov/wireshark-grpc/blob/master/grpc-hello-world.pcapng

Here you can see that they are decoded as TCP packets.

I clicked with the right mouse button on a packet and selected Decode As...:

And configured the packets as HTTP2:

And the packets are now correctly recognized as either HTTP2 or gRPC:

Let's take a look at the packet which carries the actual gRPC hello world request:

I had also opened the .proto file (located here https://github.com/grpc/grpc-java/blob/master/examples/src/main/proto/helloworld.proto) for reference.

The actual value of field 1 from the message HelloRequest, which is string name, is "world" (or 77 6f 72 6c 64 in hex):

And we can verify this:

The gRPC hello world response was of course "Hello world":

The Wireshark interpretation of fields and their values could definitely be extended. I did not come across any option to specify and input .proto file to extract the field names and correct value types. This leads to some doubles represented as just uint64 in Wireshark.

Nevertheless, I think this is a significant improvement in the right direction.

Please share your opinion on this dissector. Has anyone else tried it and published some results/improvements?
Is there any public information regarding the release of this dissector?
Kind regards!

[xfxyjwf]

@ndandanov Great findings! Could you also post this to grpc github repo? I think lots of grpc users will find this useful.

[ndandanov]

Hi @xfxyjwf, thank you for the kind words!
I added the above to the following issue: grpc/grpc#13586

I also recorded two videos which show the process in depth. They are somewhat lengthy and unfortunately I could not replicate the same successful packet dissection - probably due to the slightly newer development version of Wireshark.

Nevertheless, they could be of help to somebody:
gRPC Wireshark dissector - Part 1: https://youtu.be/LoGArC58Row
gRPC Wireshark dissector - Part 2: https://youtu.be/Wj9VoJQf-qc

[abhinava]

@ndandanov This is really great! Do you happen to have any packet captures (PCAP files)?

[abhinava]

@ndandanov Found it in your previous post :) https://github.com/ndandanov/wireshark-grpc/blob/master/grpc-hello-world.pcapng

[ghost]

Thank you for all the posts here. Posting them at an issue doesn't seem like the best place though. I'd recommend creating a page somewhere, and add a link to that page at https://github.com/protocolbuffers/protobuf/blob/master/docs/third_party.md. You would probably want to reach out to gRPC and figure out where to best post it there too. Thanks!

[huangqiangxiong]

Hope following information might be useful:

New features about Protobuf and gRPC dissectors have been added into Wireshark since version 3.2.0:

1. Protobuf files (*.proto) can now be configured to enable more precise parsing of serialized Protobuf data (such as gRPC).
2. The message of stream gRPC method can now be parsed with supporting of HTTP2 streaming mode reassembly feature.
3. User can specify protobuf search paths (where has *.proto files), and the UDP ports to protobuf message type maps at the Protobuf protocol preferences.
4. If your own dissectors need invoke protobuf dissector, you can pass the message type to Protobuf dissector by data parameter (in C) or pinfo->private_table["pb_msg_type"] (pinfo.private["pb_msg_type"] in lua).

Another two new features will be released in 3.3.0 or 3.4.0:

1. Protobuf fields can be dissected as wireshark (header) fields that allows user input the full names of Protobuf fields or messages in Filter toolbar for searching.
2. Dissector based on Protobuf can register itself to a new 'protobuf_field' dissector table, which is keyed with the full names of fields, for further parsing fields of BYETS or STRING type.

Preferences userguide:
https://www.wireshark.org/docs/wsug_html_chunked/ChProtobufSearchPaths.html
https://www.wireshark.org/docs/wsug_html_chunked/ChProtobufUDPMessageTypes.html

[jtattermusch]

@huangqiangxiong would you be interested in writing a guest blogpost for https://grpc.io/blog/ about how to use wireshark with grpc and what are some of the new features?

[huangqiangxiong]

OK. I like to do that. But I may spend a little time to study how to get the privilege to post article on that website.

[jtattermusch]

Btw a blogpost is created by opening a PR (e.g. grpc/grpc.io#121).