Map<_,type> appears to fail for zero types

[UserAB1236872]

I'll be honest, I'm not 100% certain if this is a prost issue of an issue with the Javascript library, but since I'm using the one included in Google's own tree:

https://github.com/google/protobuf/tree/master/js

I'm inclined to try here first. The issue is with the following code:

message Foo {
    map<string, int> bar = 1;
}

In Rust

let mut x = Foo::default();
x.bar.insert("val", 0);

Now encode and send it somewhere. Prost appears to only write the key, and not the subsequent value of 0. At least in Javascript, this causes a panic due to receiving a key with a missing value, leading me to have to use string types and conversions back and forth from numbers in order to bypass this.

This applies to all zero types, floats/doubles do this on 0.0, bools do this on false.

It's possible the specification allows this and the Javascript is in error, but again, I'm trying here first since I think this is more likely a prost bug.

[danburkert]

The language guide says that map fields are treated as repeated key/value message types, so I believe prost is at least not wrong to be making this optimization. Have you tried reproducing this against one of the more mature implementations like C++ or Java?

[danburkert]

The map encoding code goes way out of it's way to do this optimization, so at some point in the past I decided this was an important optimization, although I can't remember why now. Perhaps it's checked in the conformance tests.

[per-gron]

With equivalent C++ code, C++ protobuf generates 0a070a0376616c1000 which does include the default value, so I do not think this is required for conformance (I assume C++ protobuf is conformant).

C++ protobuf also accepts 0a070a0368656a as input (without the encoded zero). I think prost is not wrong here.

[UserAB1236872]

I'll close then

(Also, I don't believe C++ can be wrong. It's the reference implementation if I'm not mistaken?)

[danburkert]

FWIW I'm still not convinced prost should be doing this optimization. It makes the code a bit uglier, and if it causes compat issues then I think we should consider removing it.

[per-gron]

A guess as to why you could have thought of this optimization (or whatever you'd like to call it) as important: Because of Prost's way to represent messages as plain Rust structs, if this optimization wouldn't be there, a proto message with many int fields would take noticeable amount of space when serialized even when the object is just Default::default(). This is not the case for Java or C++ default constructed messages.

Not saying that it has to be kept, but that seems like a somewhat useful feature to me.

[danburkert]

I looked into what it would look like to change the behavior so that map keys are always encoded. It ends up making some of the most complicated code in prost simpler: 0753974. I'm somewhat inclined to merge it given the simplifications and compat issues. I'd really like to see this corner case added to the upstream conformance test suite, since merging this means prost will no longer be able to test that it can correctly decode such maps (unless some examples are hardcoded).

I'm curious if y'all have thoughts.

[danburkert]

A guess as to why you could have thought of this optimization (or whatever you'd like to call it) as important: Because of Prost's way to represent messages as plain Rust structs, if this optimization wouldn't be there, a proto message with many int fields would take noticeable amount of space when serialized even when the object is just Default::default(). This is not the case for Java or C++ default constructed messages.

That's definitely the case for proto3 default visibility fields, and prost does the exact same optimization, but the code that does that is distinct from map encoding/decoding, so the optimization can be backed out for maps without affecting proto3 messages. My gut is that this optimization really doesn't matter for maps much, since it will be relatively rare to have default values. It's definitely important for proto3 default visibility fields, though, since those will be the default much more often.

[brentnk]

Fwiw, this breaks with the the following configuration: [server: tonic/prost] -> [envoy] -> [dart/flutter using grpc-web]. Dart experiences a Code 15 Unrecoverable data loss or corruption.

A go server implementing the same grpc service responds with the encoded 0 (confirmed with wireshark) for my map<int32, OtherMessage> field and is decoded properly. Using index +/- 1 on both sides is a good enough workaround for now.

[osa1]

Prost appears to only write the key, and not the subsequent value of 0.

This is the right thing to do according to proto spec: https://github.com/protocolbuffers/protobuf/blob/main/docs/field_presence.md#presence-in-tag-value-stream-wire-format-serialization

When serializing, fields with no presence are not serialized if they contain their default value.

Encoding of map fields are specified in https://developers.google.com/protocol-buffers/docs/proto#maps

In proto syntax, if you have a map field like

map<int32, MyMapValue> my_map_field = 123;

It's encoded as if it was

repeated MyMapFieldEntry my_map_field = 123;

where

message MyMapFieldEntry {
    optional int32 key = 1;
    optional MyMapValue value = 2;
}

In the original example, map value type is int (which is not a valid proto type, I'm guessing the OP meant one of the valid integer types like int32 or int64), and because the default value for integer types is 0, prost is doing the right thing by omitting the value. If the receiving end fails to read the message then it's a bug in the protobuf implementation running on the receiving end.

For Dart protobuf implementation, we fix the decoding for map fields to handle this case in google/protobuf.dart#745.

Note that since both key and value fields of map entries are optional, it's also possible to omit the key field. For example

message Foo {
    map<int32, int32> bar = 1;
}

let mut x = Foo::default();
x.bar.insert(0, 0);

A spec compliant implementation should encode x as

[
    (1 << 3) | 2, // tag = 1, wire type = length delimited
    0, // length = 0, i.e. empty entry message
]

[LucioFranco]

@osa1 thanks for the explanation, so I guess we can close this now since this isn't really a bug with prost?