Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support custom options #9

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Conversation

sharonliao
Copy link

ping @pseudomuto

What is Changing?

This PR aims to fix this issue pseudomuto/protoc-gen-doc#425, to support custom, user-defined options exposing.

The custom options are unknown at compile time, so that FileDescriptorSet got from the protoc compiler won’t include the readable custom options info. To expose custom options from unknown fields, we need to register all custom extensions by recursively going through all the proto files, and then marshal and re-unmarshal the descriptor files by using the extensions registered before as a resolver. This step will decode custom field options from byte rows to a readable format.

field_option

How is it Changing?

Basically, I followed this issue golang/protobuf#1260 to implement the registering and marshal and re-unmarshal process.

golang/protobuf#1260 (comment)

@sharonliao
Copy link
Author

sharonliao commented Aug 17, 2022

ping @pseudomuto

To support the proto3 custom field option (pseudomuto/protoc-gen-doc#425), there are two tools that need to be modified.

  1. protokit
  2. protoc-gen-doc

The protoc-gen-doc leverages the protokit to create a plugin for the compiler protoc. The protokit gets the proto descriptor files from protoc and applies some necessary process on the proto descriptor, then pass it to proto-gen-doc to let it generate documentation.

The reason why the protoc-gen-doc doesn't support proto3 custom field option is that the output of compiler protoc doesn’t include the proper custom field option, to more detail, those options information is saved as byte rows in descriptor files.

To display custom field options properly on documentation, we need to

  • In protokit, register all custom extensions by recursively going through all the proto files.
  • In protokit, marshal and re-unmarshal the descriptor files by using the extensions registered before as a resolver. This is a critical step, which can decode custom field options from byte rows to readable format.
  • In protoc-gen-doc, modify option-extract functions to fetch the custom options.
  • In protoc-gen-doc, adjust the templates to display custom field options in label sections.

Change on the protokit

master...sharonliao:protokit:master

  • utils/protobuf.go
    • Register Extensions
    • Marshal and re-marshal the descriptor files by using the extensions registered before as a resolver.
  • plugin.go
    • Decode the output that comes from the compiler protoc, before passing it to protoc-gen-doc plugin.

Change on the protoc-gen-doc

  • template.go
    • Modify option-extract functions to fetch the custom options.
  • resources/markdown.tmpl
    • Adjust the templates to display custom field options in label sections.
  • resources/html.tmpl
    • Adjust the templates to display custom field options in label sections.

@renjiexu
Copy link

Any update of this PR? Really looking forward to having this feature in proto-gen-doc so we could generate custom options in the output...

@untra
Copy link

untra commented Jul 13, 2023

Ah man, I came looking for this functionality following the thread from pseudomuto/protoc-gen-doc#425

Yes please, I am also looking for custom extensions documented in our descriptorset.

@akaspin
Copy link

akaspin commented Dec 10, 2023

Fails with service method options:

panic: invalid type: got *dynamicpb.Message, want *annotations.HttpRule

goroutine 1 [running]:
google.golang.org/protobuf/internal/impl.(*messageConverter).GoValueOf(0x140006878e0, {{}, 0x102fc69a0?, 0x14000325f40?, 0x102fe68f0?})
	/Users/spin/go/pkg/mod/google.golang.org/protobuf@v1.31.0/internal/impl/convert.go:457 +0x3a8
google.golang.org/protobuf/internal/impl.(*ExtensionInfo).InterfaceOf(0x140000368a0?, {{}, 0x102fc69a0?, 0x14000325f40?, 0x103311240?})
	/Users/spin/go/pkg/mod/google.golang.org/protobuf@v1.31.0/internal/impl/extension.go:102 +0x60
github.com/golang/protobuf/proto.GetExtension({0x102fe2548?, 0x140000368a0}, 0x103311220)
	/Users/spin/go/pkg/mod/github.com/golang/protobuf@v1.5.3/proto/extensions.go:171 +0x4c8
github.com/pseudomuto/protokit.getOptions({0x102fe2548, 0x140000368a0})
	/Users/spin/go/pkg/mod/github.com/akaspin/protokit@v0.0.0-20220620195650-497fbb9da0ad/types.go:54 +0xac
github.com/pseudomuto/protokit.(*common).setOptions(0x140001929c0, {0x102fe2548?, 0x140000368a0?})
	/Users/spin/go/pkg/mod/github.com/akaspin/protokit@v0.0.0-20220620195650-497fbb9da0ad/types.go:67 +0x34
github.com/pseudomuto/protokit.parseServiceMethods({0x102fe3b40, 0x1400074ae40}, {0x140003dd0e0, 0x2, 0x102fbef80?})
	/Users/spin/go/pkg/mod/github.com/akaspin/protokit@v0.0.0-20220620195650-497fbb9da0ad/parser.go:281 +0x370
github.com/pseudomuto/protokit.parseServices({0x102fe3b40, 0x1400074ad50}, {0x14000116ab0, 0x1, 0x102fcdda0?})
	/Users/spin/go/pkg/mod/github.com/akaspin/protokit@v0.0.0-20220620195650-497fbb9da0ad/parser.go:259 +0x2fc
github.com/pseudomuto/protokit.parseFile({0x102fe3a98, 0x103352dc0}, 0x140000a9c00)
	/Users/spin/go/pkg/mod/github.com/akaspin/protokit@v0.0.0-20220620195650-497fbb9da0ad/parser.go:75 +0x354
github.com/pseudomuto/protokit.ParseCodeGenRequest(0x14000183000?)
	/Users/spin/go/pkg/mod/github.com/akaspin/protokit@v0.0.0-20220620195650-497fbb9da0ad/parser.go:45 +0xf4
github.com/pseudomuto/protoc-gen-doc.(*Plugin).Generate(0x102fe0098?, 0x14000116018?)
	/Users/spin/craft/protoc-gen-doc/plugin.go:40 +0x3c
github.com/pseudomuto/protokit.RunPluginWithIO({0x102fe0078, 0x103352dc0}, {0x102fe0098?, 0x14000116018?}, {0x102fe0058, 0x14000116020})
	/Users/spin/go/pkg/mod/github.com/akaspin/protokit@v0.0.0-20220620195650-497fbb9da0ad/plugin.go:32 +0x54
github.com/pseudomuto/protokit.RunPlugin(...)
	/Users/spin/go/pkg/mod/github.com/akaspin/protokit@v0.0.0-20220620195650-497fbb9da0ad/plugin.go:22
main.main()
	/Users/spin/craft/protoc-gen-doc/cmd/protoc-gen-doc/main.go:33 +0x98
--doc_out: protoc-gen-doc: Plugin failed with status code 2.

@haljin
Copy link

haljin commented Jan 24, 2024

It seems that problem occurs when you have google.api.http options defined in addition to your custom, I am having the same problem.

I did manage to make it not crash by changing the types.go file and doing this:

func getOptions(options proto.Message) (m map[string]interface{}) {
	protoregistry.GlobalTypes.RangeExtensionsByMessage(proto.MessageReflect(options).Descriptor().FullName(),
		func(extension protoreflect.ExtensionType) bool {

			if m == nil {
				m = make(map[string]interface{})
			}
			m[extension.TypeDescriptor().TextName()] = extension
			return true
		})

	return m
}

instead of what it did before. I barely know anything about go or the proto libraries though so I am not sure if this can be done otherwise. I'll publish a fork in a bit.

What I noticed is that it behaves differently now and I think this has something to do with registering extensions. Before I'd get

 "options": {
                "google.api.http": {
                  "rules": [
                    {
                      "method": "POST",
                      "pattern": "/{{system_id}}/group/get",
                      "body": "*"
                    }
                  ]
                }
              }

now it changes to

"options": {
                "[google.api.http]": {
                  "body": "*",
                  "post": "/{{system_id}}/system/get"
                },
                "[ngcp.api.common.routing_key]": "provisioning_service.system.get.{{system_id}}"
              }

So my custom option is here now, but the HTTP one is different. Funnily enough if I did

	if strings.Contains(string(fileDescriptor.FullName()), "google.api") {
			log.Println("Nah, drop")
			return true
		}

in the RegisterExtensions it'd actually include both versions of api.http (as well as my custom options) in the output JSON.

@haljin
Copy link

haljin commented Jan 24, 2024

Here's the fork: https://github.com/haljin/protokit

Also the thing that makes the whole thing explode is the custom parser in https://github.com/pseudomuto/protoc-gen-doc/blob/master/extensions/google_api_http/google_api_http.go

Hopefully someone more clever can make sense of this and get this stuff merged. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants