How to retrieve custom options inside of a compiler/protogen plugin?

[tgorton617]

hi all - I'm writing a template-based protoc plugin using the framework provided by compiler/protogen, and after a fair bit of digging and experimentation I've been unable to figure out how to retrieve the value of custom options during execution. Since I'm building a protoc plugin (which I don't want to recompile for every protobuf input) I can't use the generated go code (notably the E_* vars), and I also don't have a Message such as is used the example in the March blog post announcing the new protobuf go API

Here's as close as I've been able to come:

sample option file:

syntax = "proto3";

package example;

option go_package = "examples/options/proto;option";

import "google/protobuf/descriptor.proto";

extend google.protobuf.MessageOptions {
  optional string example_annotation_string = 1000;
  optional int32 example_annotation_int32 = 1001;

}

message Foo {
  option (example.example_annotation_string) = "hello";
  option (example.example_annotation_int32) = 1234;

  int32 id = 1;
}

... and a simple main.go of a plugin written in go:

package main

import (
	"fmt"
	"google.golang.org/protobuf/compiler/protogen"
	"google.golang.org/protobuf/types/pluginpb"	
)

func main() {
	protogen.Options{
	}.Run(func(gen *protogen.Plugin) error {

		gen.SupportedFeatures = uint64(pluginpb.CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL)

		for _, sourceFile := range gen.Files {
			if !sourceFile.Generate {
				continue
			}

			// setup output file
			outputfile := gen.NewGeneratedFile("./out.txt", sourceFile.GoImportPath)
			
			// pull out the particular message we know is in the file for this example
			options := sourceFile.Messages[0].Desc.Options()

			// how to get the custom options values out of the "options" protoreflect.ProtoMessage?
			// just print the value to prove they are in there
			outputfile.P(fmt.Sprintf("Options value is:%v",options))

		}
		return nil
	})
}

the output is:

Options value is:1000:"hello" 1001:1234

While this proves the data is present in the "options" descriptorpb.MessageOptions... how can I use the API to read it? This data does not appear in the result of descriptorpb.MessageOptions.GetUninterpretedOption().

[dsnet]

The Options method returns a proto.Message, which you need to type assert to one of the descriptorpb.XXXOptions types.

import (
    "google.golang.org/protobuf/proto"
    "google.golang.org/protobuf/types/descriptorpb"
    option "examples/options/proto"
)

options := sourceFile.Messages[0].Desc.Options().(*descriptorpb.MessageOptions)
ext1 := proto.GetExtension(options, option.E_ExampleAnnotationString).(string)

BTW, "examples/options/proto" is a poor choice for a Go package path. It should generally be something like github.com/user/project/... or company.com/project/....

[tgorton617]

Thanks @dsnet. I've gotten this to work if I use the generated E_ExampleAnnotationString var as you describe, but as noted in my original question I'm aiming to do this without having to recompile my protoc plugin for each new input -- my use case is a template-based generator similar to https://github.com/moul/protoc-gen-gotemplate (which uses the old protobuf go library in clearly unsupported ways) so custom options are not known at the protoc plugin's compile time.

The new protobuf go library makes almost everything I need to access quite easy -- but the one piece of information in the .proto files that I don't seem to be able to access (without compiling the generated go modules into my protoc plugin as you suggest) is the value of custom options.

I did another unsuccessful experiment along these lines: in order to avoid recompiling my protoc plugin I attempted passing the protogen.Message to a go plugin that was compiled with the generated go code (ie E_ExampleAnnotationString) -- a reasonable separation of concerns between a generic protoc plugin and a go plugin familiar with one project's custom options -- but the go plugin's code is not able to retrieve the custom option information out of the protogen.Message, I assume because the info on the options was not saved into the Message structure internals in the protoc plugin because the generated code's init() function didn't run in the protoc plugin (only in the go plugin it called).

(thanks also for the note on my go package path -- I was indeed being lazy in putting together my example)

[dsnet]

I see, can you try obtaining them dynamically through reflection?

options := sourceFile.Messages[0].Desc.Options().(*descriptorpb.MessageOptions)
options.ProtoReflect().Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
    if !fd.IsExtension() {
        continue
    }
    fmt.Println(fd, v)
    // Make use of fd and v based on their reflective properties.
    return true
})

[dsnet]

Actually, I don't think the above will work since the compiled binary will not know about any dynamically created extensions that are passed at runtime. You'll need to create a protoregistry.Types at runtime that knows the type information for all the extensions seen at runtime.

{
	var gen protogen.Plugin

	// The type information for all extensions is in the source files,
	// so we need to extract them into a dynamically created protoregistry.Types.
	extTypes := new(protoregistry.Types)
	for _, file := range gen.Files {
		if err := registerAllExtensions(extTypes, file.Desc); err != nil {
			panic(err)
		}
	}

	for _, sourceFile := range gen.Files {
		if !sourceFile.Generate {
			continue
		}

		// The MessageOptions as provided by protoc does not know about
		// dynamically created extensions, so they are left as unknown fields.
		// We round-trip marshal and unmarshal the options with
		// a dynamically created resolver that does know about extensions at runtime.
		options := sourceFile.Messages[0].Desc.Options().(*descriptorpb.MessageOptions)
		b, err := proto.Marshal(options)
		if err != nil {
			panic(err)
		}
		options.Reset()
		err = proto.UnmarshalOptions{Resolver: extTypes}.Unmarshal(b, options)
		if err != nil {
			panic(err)
		}

		// Use protobuf reflection to iterate over all the extension fields,
		// looking for the ones that we are interested in.
		options.ProtoReflect().Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
			if !fd.IsExtension() {
				return true
			}
			fmt.Println(fd, v)
			// Make use of fd and v based on their reflective properties.
			return true
		})
	}
}

func registerAllExtensions(extTypes *protoregistry.Types, descs interface {
	Messages() protoreflect.MessageDescriptors
	Extensions() protoreflect.ExtensionDescriptors
}) error {
	mds := descs.Messages()
	for i := 0; i < mds.Len(); i++ {
		registerAllExtensions(extTypes, mds.Get(i))
	}
	xds := descs.Extensions()
	for i := 0; i < xds.Len(); i++ {
		if err := extTypes.RegisterExtension(dynamicpb.NewExtensionType(xds.Get(i))); err != nil {
			return err
		}
	}
	return nil
}

[tgorton617]

Thanks @dsnet - very tricky! I confirmed that the simple Range() solution doesn't work as you expected, and that the Marshal-Unmarshal one does. I stuck my completed example in this repo for posterity.

Many thanks!

[dsnet]

Great to hear!

Anything left to do here? This is the first time I've seen a plugin implementation that needed to dynamically reinterpret the options based on the set of input sources it is operating on. It's cool to see that the new API is able to support this advanced use case. The old API had no hope for accomplishing this.

[tgorton617]

I believe I'm all set, many thanks.

Definitely an impressive use of the new reflection abilities. FWIW, while digging around I found two other examples while digging of people facing a similar same issue - this stackoverflow question (to which I posted your solution, a link to this issue and my repo) and the stringMethodOptionsExtension and similar methods inside the protoc-gen-gotemplate project, which appear to be attempting an awkward (given the lack of reflection) version of a similar solution to yours using the old API.

[shashankram]

Hi, is it possible to dynamically register field options similar to #1260 (comment)?