v1.22.0

• Overview
• Notable changes
  • Proto3 optional semantics
  • Adjustments to ease migration
• Upcoming breakage changes

Overview

This release adds support for the experimental proto3 optional semantics coming in the v3.12.0 release of the protobuf language, adjusts the API to ease migration from the old module to the new module, and fixes some minor bugs.

Notable changes

New features:

• CL/230698: all: implement support for proto3 optional semantics
• CL/229157: proto: add RangeExtensions and adjust HasExtension and GetExtension

Minor changes:

• CL/228837: all: consistently treat nil message interface as an empty read-only message
• CL/228838: proto: never return nil []byte from Marshal when successful
• CL/229558: all: improve extension validation
• CL/229277: internal/impl: inline coderInfoFields for better cache locality

Bug fixes:

• CL/229278: internal/impl: fix off-by-one error in message initialization

Proto3 optional semantics

Protocol buffers in v3.12.0 is gaining experimental support to represent true presence for scalars in the proto3 syntax. By default, scalars in proto3 lack presence, so users are unable to distinguish between an unpopulated field and whether the zero value was explicitly stored into that field. With this new language feature, users can explicitly specify the optional keyword for scalars to indicate that presence information should be preserved for that field.

This release adds support to the generator and the runtime for this feature. Generation of proto3 optional fields are represented as pointers to scalar types (similar to proto2 optional scalars).

Adjustments to ease migration

A few minor changes were made to the API to unify the behavior between the old github.com/golang/protobuf module and the new google.golang.org/protobuf module. This should aid users migrating from the former to the latter.

Changes:

• Adjusted all functions to treat a nil proto.Message as an untyped, read-only, empty message and provide sensible results given that semantic.
• Adjusted proto.HasExtension to always report false if the extension descriptor does not extend the parent message (as opposed to panicking previously).
• Added proto.RangeExtensions as a replacement for the legacy proto.ClearAllExtensions and proto.ExtensionDescs functions.
• Fixed protoreflect.ExtensionType.IsValidInterface to never panic and properly report whether the given value is of the right type.

Upcoming breakage changes

There are no new upcoming breaking changes to announce in this release.

As a reminder, the following have already been announced and may take effect in the near future:

• Deprecations announced on 2020-03-02 with the v1.20.0 release

v1.21.0

• Overview
• Notable changes
  • Code generator flags
  • Package protowire
  • Package protopack
  • Package protocmp
• Upcoming breakage changes

Overview

This release contains some improvements to the usability of the protoc-gen-go tool and the release of several packages that make it easier to interact directly with the binary wire format.

Notable changes

New features:

• CL/219298: compiler/protogen: add module= generator option
• CL/219597: compiler/protogen: make paths=import work with M overrides
• CL/223819: compiler/protogen: allow specifying the package name with M flags
• CL/219838: encoding/protowire: make package publicly available
• CL/219837: testing/protopack: make package publicly available
• CL/221432: testing/protocmp: add SortRepeated and SortRepeatedFields
• CL/226237: reflect/protodesc: add NewFiles

Minor changes:

• CL/223817: types/dynamic: make Message implement legacy message interface
• CL/223857: types/dynamicpb: fix message Zero return a read-only type

Code generator flags

A new module flag is added to the generator that controls stripping of a prefix from the generated filenames. This option enables protoc-gen-go to better generate .pb.go files for a workflow that depends on Go modules rather than a centralized Go path.

For example, suppose a protos/source.proto file has a go_package option specified as follows:

option go_proto = "github.com/example/project/foo";

and that the module root for this project is github.com/example/project.

If invoked from the module root, this flag could be potentially used as follows:

protoc --go_out=module=github.com/example/project:. protos/source.proto

This would emit a file located foo/source.pb.go rather than being located at github.com/example/project/foo/source.pb.go.

Furthermore, two minor changes have been made to the M flag, which can be used to specify the Go package path corresponding to a given .proto file:

• When used with the paths=import option, the implementation now correctly respects the effect of that option and outputs the file as import-relative, rather than always being source-relative.
• If the Go package path is provided with a ; delimiter, it can be used to specify the exact package name for the generated package. It's syntax is identical to that of the go_package option.

Package protowire

The protowire package provides functionality for parsing and packing the binary wire format. Effective use of this package requires that the user has an understanding of the wire format itself. The package is written in a performance sensitive way and currently serves as the underlying implementation of the protobuf runtime. It is the replacement for methods previously provided on the Buffer type in the legacy proto package.

Example of how to parse wire data for an encoded message:

import "google.golang.org/protobuf/encoding/protowire"

var b []byte = ... // encoded bytes of a protobuf message
for len(b) > 0 {
    num, typ, n := protowire.ConsumeField(b)
    if n < 0 {
        log.Fatalf("parse error: %v", protowire.ParseError(n))
    }
    log.Printf("FieldNumber:%d  WireType:%d  WireData:%x", num, typ, b[:n])
    b = b[n:]
}

Using the encoded output of the following protopack example, this produces:

FieldNumber:1  WireType:2  WireData:0a0d48656c6c6f2c20776f726c6421
FieldNumber:2  WireType:0  WireData:10f6ffffffffffffffff01
FieldNumber:3  WireType:2  WireData:1a0ccdcc8c3fcdcc0c4033335340

Package protopack

The protopack package is intended to be used for debugging the binary wire format or assist in the construction of testdata in the wire format. Effective use of this package requires that the user has an understanding of the wire format itself. The protopack package essentially provides a set of types that represents the wire data as a concrete syntax tree.

Given this proto3 message definition:

message MyMessage {
    string field1 = 1;
    int64 field2 = 2;
    repeated float32 field3 = 3;
}

Valid wire data for the message can be constructed as such:

import . "google.golang.org/protobuf/testing/protopack"

b := Message{
    Tag{1, BytesType}, String("Hello, world!"),
    Tag{2, VarintType}, Varint(-10),
    Tag{3, BytesType}, LengthPrefix{
        Float32(1.1), Float32(2.2), Float32(3.3),
    },
}.Marshal()

This results in the following output data:

0x0000  0a 0d 48 65 6c 6c 6f 2c  20 77 6f 72 6c 64 21 10  |..Hello, world!.|
0x0010  f6 ff ff ff ff ff ff ff  ff 01 1a 0c cd cc 8c 3f  |...............?|
0x0020  cd cc 0c 40 33 33 53 40                           |...@33S@|

Package protocmp

The SortRepeated and SortRepeatedFields options are added to the protocmp package. These options enable repeated fields to be sorted so that they are functionally treated as multisets.

The SortRepeated option sorts repeated fields of a specific element type. This option requires the user to provide a less function where the field type is inferred from the provided function type.

The SortRepeatedFields option sorts a set of repeated fields on a specific message type. This option requires the user to specify the set of fields by name. An arbitrary sort ordering is chosen for each specified repeated field.

Upcoming breakage changes

There are no new upcoming breaking changes to announce in this release.

As a reminder, the following breaking changes have already been announced and may take effect 6 months after they have first been announced:

• Announced on 2020-03-02 with the v1.20.0 release:
  • Enforcement of namespace uniqueness
  • Requirement of the go_package option
  • Deprecation of certain flags for protoc-gen-go
  • Deprecation of certain APIs in generated code

v1.20.1

Notable changes since v1.20.0:

• CL/204804: bug fix for dead-lock in the global type registry
• CL/221803: minor performance improvements for Marshal and Size operations
• CL/222678: fix signature of pseudo-internal MarshalState and UnmarshalState methods

v1.20.0

• Overview
  • Backwards compatibility
  • Package index
• Notable changes
  • Message type
  • Reflection
  • Wire serialization
  • JSON serialization
  • Text serialization
  • Extensions
  • Nil values
  • Errors
  • Generated code
  • gRPC support
  • Performance
• Upcoming breakage changes
  • Enforcement of namespace uniqueness
  • Required go_package option
  • Deprecation of protoc-gen-go flags
  • Deprecation of generated APIs

Overview

The google.golang.org/protobuf module is a major revision of the Go bindings for protocol buffers. The API has been completely overhauled with a focus on simplicity and clarity. Deprecated types and functions are removed, complicated packages are split into more manageable pieces, and internal details are separated from the public API. Particular attention was made to ensure that this implementation complies with the protocol buffer language as closely as possible to ensure proper interoperability between the Go implementation and implementations for other languages.

The first version of this module is v1.20.0. The version number is deliberately chosen to avoid overlap with existing and anticipated versions of the github.com/golang/protobuf module. This enables the version alone to unambiguously distinguish between the two modules.

The minimally supported version of Go is v1.9.

Backwards compatibility

The API in this module is not drop-in compatible with the earlier github.com/golang/protobuf module. However, the new API does strive when sensible to be compatible for easier migration.

Version v1.4.0 of the github.com/golang/protobuf module implements most of the existing API in terms of this module, which provides a number of benefits:

• Shared type registry: It ensures that the two modules share a unified type registry, which is necessary for dynamic usages of protocol buffers to work properly.
• Automatic improvements: It provides the older module with a degree of automatic upgrading whenever bug fixes and improvements are made to the newer module.
• Long-term maintainability: It keeps the older module more maintainable since it is largely a thin shim over the newer module.

Some existing features present in the github.com/golang/protobuf module are currently unavailable in this module:

• Manipulation of the wire format: The older module provides functionality for directly manipulating the raw wire format through methods on the proto.Buffer type.
• Helper functions for well-known types: The older module provides the ptypes package containing helper functions for interacting with the well-known types such as Any or Timestamp messages.

We plan to add these features to this module in the near future and improve upon them.

Package index

Summary of the packages provided by this module:

• proto: Package proto provides functions operating on protobuf messages such as cloning, merging, and checking equality, as well as binary serialization.
• encoding/protojson: Package protojson serializes protobuf messages as JSON.
• encoding/prototext: Package prototext serializes protobuf messages as the text format.
• reflect/protoreflect: Package protoreflect provides interfaces to dynamically manipulate protobuf messages.
• reflect/protoregistry: Package protoregistry provides data structures to register and lookup protobuf descriptor types.
• reflect/protodesc: Package protodesc provides functionality for converting descriptorpb.FileDescriptorProto messages to/from the reflective protoreflect.FileDescriptor.
• testing/protocmp: Package protocmp provides protobuf specific options for the cmp package.
• testing/prototest: Package prototest exercises the protobuf reflection implementation for concrete message types.
• types/dynamicpb: Package dynamicpb creates protobuf messages at runtime from protobuf descriptors.
• types/known/anypb: Package anypb is the generated package for google/protobuf/any.proto.
• types/known/emptypb: Package emptypb is the generated package for google/protobuf/empty.proto.
• types/known/timestamppb: Package timestamppb is the generated package for google/protobuf/timestamp.proto.
• types/known/durationpb: Package durationpb is the generated package for google/protobuf/duration.proto.
• types/known/wrapperspb: Package wrapperspb is the generated package for google/protobuf/wrappers.proto.
• types/known/structpb: Package structpb is the generated package for google/protobuf/struct.proto.
• types/descriptorpb: Package descriptorpb is the generated package for google/protobuf/descriptor.proto.
• types/pluginpb: Package pluginpb is the generated package for google/protobuf/compiler/plugin.proto.
• compiler/protogen: Package protogen provides support for writing protoc plugins.
• cmd/protoc-gen-go: The protoc-gen-go binary is a protoc plugin to generate a Go protocol buffer package.

Notable changes

This section summarizes notable changes in this release relative to v1.3.4 of the github.com/golang/protobuf module.

Message type

The definition of the proto.Message interface type has changed (see v1 Message and v2 Message). The new type definition contains a behaviorally-complete specification of a protobuf message. All functions in this module that operate on a proto.Message can be used with values of any concrete type which properly implements this interface.

The github.com/golang/protobuf/proto package contains functions to convert between the old and new proto.Message types (see MessageV1 and MessageV2). These functions are guaranteed to work with types generated by protoc-gen-go, however there is no guarantee that it handles hand-crafted messages or messages generated by different generators.

Reflection

A key feature of the new Go protobuf API is support for protobuf reflection, which is the ability to programmatically interact with a message at runtime. The ProtoReflect method implemented by every proto.Message returns a reflective view of that message. Using protobuf reflection instead of Go reflection is generally easier since it closely follows the protobuf data model, and also more maintainable since it relies on a stable API that is agnostic to the exact underlying implementation of a message. On the other hand, using Go reflection on protobuf messages often needs to make low-level assumptions about a particular implementation of protobuf messages that may not be stable (e.g., assuming that the Go struct field ordering exactly matches the proto message field ordering).

See the protoreflect package for more details.

Wire serialization

The API for wire serialization has changed slightly for better usability. Optional arguments for controlling serialization are specified through the proto.MarshalOptions and proto.UnmarshalOptions types. Both...