protoc-gen-go: support go_tag option to specify custom struct tags

[jedborovik]

Is it possible to define a message with custom tags? For example defining a json name that isn't just the lowercase field name or adding a bson tag.

[dsymonds]

No, there's no support for that, nor any intention to support that.

[mwmahlberg]

I'd like to add that this feature would be extremely useful for validation purposes.

[jswierad]

+1

Use case: Using protobuf generated structs along with sqlx package. Would be awesome

[mikebell-org]

Likewise the go library for cloud datastore. Basically, increasing numbers of things make use of struct tags, and without support for them you're left manually duplicating structs? If there are no plans to add this functionality, what is the recommended way to handle it?

[iain17]

I would love this feature!
This seems to somehow mitigate the need

https://github.com/favadi/protoc-go-inject-tag

[sefasenturk95]

I would love this feature aswell!

[willks]

This would be handy, as I need to validate JSON payloads for my REST API portion of the service.

[awalterschulze]

Since golang/protobuf will probably never support this. Here are some solutions.

https://github.com/mwitkow/go-proto-validators is useful for validation on proto structs.

https://github.com/gogo/protobuf can be used in two ways to have custom tags:

1. You can use the moretags extension
   https://github.com/gogo/protobuf/blob/master/test/tags/tags.proto

2. Or you can use the typedecl extension if you don't want to generate the golang struct.
   This allows you declare your own golang struct with all the tags you want and more.

[ekiyanov]

Need it as well to automate insert/update into the sql database

[qianlnk]

I've add a plugin retag for protoc-gen-go.

https://github.com/qianlnk/protobuf/tree/master/protoc-gen-go/retag

protoc --go_out=plugins=grpc+retag:. yourproto.proto

[RajeshKumar1990]

@qianlnk .thanks.I tried your plugin when i use only message then ther is no problem but when the message have sub message like this

message SendData {
	message MetaData{
		string Name = 1;//`json:"name;omitempty"`
		int64 Length = 2;//`json:"length;omitempty"`
	}
	message Chunck{
	bytes Data = 1;//`json:"data;omitempty"`
	int64 Position = 2;//`json:"position;omitempty"`
	}
}

This throws the error.Any how the retag can support this???
Thanks...

[qianlnk]

@RajeshKumar1990 I fixed it.

[atamgp]

@qianlnk I am having a problem making this work.... does it support proto3?

syntax = "proto3";
package contracts;

import "Common.proto";
//import "google/protobuf/timestamp.proto";

message ZZZ {
  string id = 1;  // `db:"id"`
  string clientId = 2;
  string datetime = 3;
} 

Converting with:
protoc -I=xyz.eu/contracts/proto --go_out=plugins=grpc+retag:xyz.eu/contracts/ xyz.eu/contracts/proto/*.proto

Result:

type ZZZ struct {
	Id       string  `protobuf:"bytes,1,opt,name=id" json:"id,omitempty"`
	ClientId string  `protobuf:"bytes,2,opt,name=clientId" json:"clientId,omitempty"`
	Datetime string  `protobuf:"bytes,5,opt,name=datetime" json:"datetime,omitempty"`
}

The db custom tag on id is missing....
Like everyone using proto, I need this in order to work with (any) orm:)
Thanks for your time

[qianlnk]

@atamgp the first letter in field must be capitalized.

[atamgp]

@qianlnk thank you for the response. I changed the proto to:

string ID = 1; // db:"id,omitempty"

In above and below example the ` before db and at the end of line are omitted by github... but they are there. Unlike your example, only I am using a non existing tag like (no json or xml).
Still no db output:

ID string protobuf:"bytes,1,opt,name=ID" json:"ID,omitempty"

If you have another idea please let me know. I am looking into the plugin code right now.
I tested that the code is called with a print statement in de init. So the environment seems ok.

[qianlnk]

@atamgp have you install my retag plugin?

git clone https://github.com/qianlnk/protobuf.git $GOPATH/src/github.com/golang/protobuf
go install $GOPATH/src/github.com/golang/protobuf/protoc-gen-go

this is my try:

syntax = "proto3";
package contracts;

//import "Common.proto";
//import "google/protobuf/timestamp.proto";

message ZZZ {
  string ID = 1;        //`db:"id"`
  string ClientID = 2;  //`db:"client_id"`
  string Datetime = 3;  //`db:"datetime"`
}

run:

protoc -I/usr/local/include -I. \
   -I$GOPATH/src \
   --go_out=plugins=grpc+retag:. \
test.proto

and result:

type ZZZ struct {
	ID       string ` protobuf:"bytes,1,opt,name=ID"         json:"ID,omitempty"  db:"id"`
	ClientID string ` protobuf:"bytes,2,opt,name=ClientID"   json:"ClientID,omitempty"  db:"client_id"`
	Datetime string ` protobuf:"bytes,3,opt,name=Datetime"   json:"Datetime,omitempty"  db:"datetime"`
}

[atamgp]

@qianlnk OK, I have it figured out a bit more :)

message ZZZZ {
string ID = 1; //db:"id"
string clientID = 2; //db:"id"
double amount = 3;
}

Things I learned:

1. I have to put the db tag on all fields of a message. Otherwise I get an exception:
   panic: runtime error: index out of range (panic: runtime error: index out of range -> indexing [1])
   For some fields it is not wanted to be mapped by an ORM. Is it possible to make this optional?
2. Like you said earlier, field name has to begin with a capital otherwise its ignored. Ofcourse its possible to misuse this aspect to realize optional fields (1) but is less clean.
3. Command path used. I have this folder structure:
   $GOPATH/src/X/Y/Z/zzz.proto
   I was using a protoc command from src folder which did not work:

Working: When calling from within Z: protoc --go_out=plugins=grpc+retag:. zzz.proto
Not working: protoc -I:X/Y/Z --go_out=plugins=grpc+retag:. X/Y/Z/zzz.proto
Working: protoc -I:. -I:X/Y/Z --go_out=plugins=grpc+retag:. X/Y/Z/zzz.proto
4) importing from other proto: I have a common.proto and a zzz.proto. zzz is importing an enum and a message from common.
working: if a use a working command from point 3) above for 1 file at a time e.g.:
protoc -I:. -I:X/Y/Z --go_out=plugins=grpc+retag:. X/Y/Z/zzz.proto
protoc -I:. -I:X/Y/Z --go_out=plugins=grpc+retag:. X/Y/Z/common.proto

not working: same commands which are working for 1 file but in order to process both proto files /*.proto instead of a specific proto. E.g.: protoc -I:. -I:X/Y/Z --go_out=plugins=grpc+retag:. X/Y/Z/*.proto
errors: .... is already defined in file ....
It is really nice to process all proto files in 1 time, can you have a look at this?

Last: It is not clean to have all proto files and generated go files in the same folder.
In my case Z is actually named proto and Y is contracts.
I want to have my generated go files put in folder Y instead of Z. Still looking for a working combination command for this...

[EwanValentine]

Has anyone found a solution for converting between type: string and type: bson.ObjectId as well?

[mavle]

Please support this it is extremely useful and cuts down on needless code bloat.

[puellanivis]

While this idea is super useful in Go, unfortunately no other language that proto supports can really make use of it. As such, changing/supporting this would be outside of the scope of the official Go protobuf package.

[maguro]

@puellanivis, that is not true. Supporting custom tags would open protobufs to a plethora of possible tooling.

[dsnet]

I'm going to re-open this issue, but it does not mean that we're going to support this. I'd like to see more discussion on the feasibility of this. Some thoughts:

• I can see the usefulness of this feature for Go-heavy projects, but protobufs were designed a language-agnostic way to interchange data. Baking more Go-specific details in the proto files seems to be antithetical to the philosophy of protobufs.
• This features makes the assumptions that generated Go protobufs are always structs with fields. What does it mean if we want the ability to generate opaque protobuf types that are interacted via methods. Performance testing has shown that techniques like lazy unmarshaling can bring significant speed benefits, but is only feasible as an opaque type.
• The proper way to support this seems to be using protobuf options, but I don't see any proposal here suggesting a syntax for how that would work.

[qianlnk]

why not add a plugin like

https://github.com/qianlnk/protobuf/tree/master/protoc-gen-go/retag

This plugin is my original idea, don't take too much effort to attain it.But I believe that many people need it, so I hope more people can join in.

[meling]

@qianlnk Looked at your retag plugin as one of several alternatives. I don't really like the idea of cloning it into my golang/protobuf folder, especially since This branch is 8 commits ahead, 28 commits behind golang:master. Would you consider making it a standalone plugin instead?

[kulak]

I am not contesting neild's post.

How does the logic above apply to JSON support in the library? ;) Why JSON gets support and XML does not? Unsigned 64 bit in JSON is serialized as string. That's a very specific support for target format...

We have to understand why people want this feature in the 1st place. The #1 motivation is to be able to tag structures for use by another library like DB layer (https://github.com/upper/db is just one example).

IMO, Protocol Buffers team needs to focus on helping developers to make it easy to assign protocol buffer generated structures to types that have required tag data. Unfortunately, it is not possible today to do this:

type DBRecord struct {}
var record DBRecord = pbRecord

The problem is with additional fields that PB generates. Those extra fields make it impossible to do an assignment of variables and requires manual field by field copy. I know that, because I had to code a lot of functions to copy data from PB and into PB.

Specification is clear here:

• Type Identity
• Type Assignability

It would be great if PB team could help us with copy data graph operations. I think there are a number of approaches that can be taken here.

[dsnet]

Those extra fields make it impossible to do an assignment of variables and requires manual field by field copy.

While I understand the desire to do this, I don't see a way for this feature to be provided without going against protobuf's goal of being backwards and forwards compatible. In particular, the ability to add new fields without breaking current usages. In order for this assignment (or a cast) to work, it requires that the memory layout of DBRecord be identical to the memory layout of the generated protobuf message type. Fundamentally, this adds a constraint that .proto source file be atomically updated with the equivalent Go type definition, which goes against one of the primary goals of protobufs.

[tandr]

We can go on and on with this feature, but if authors are very insistent on "my way or highway", there is not much we can do here.

This is one of the most requested features desperately missed from the original proto-gen-go. Missed so much that people switched to gogoprotobuf just to have it. If implemented, it would allow proto-gen-go generated code to be useful in more than one scenario that original authors thought of, and just be a good "code neighbour" by allowing the feature that is not in direct use of the library, but potentially make life much easier to devs who intend to do "one more little thing" with generated code. Continuing with neighbour analogy, it also can choose to be an old sad grumpy "get off my lawn", and just never allow kids to draw on the sidewalk because "it is not what sidewalk is for".

After 5 years of begging through this bug being open... This is actually quite sad.

[dsnet]

if authors are very insistent on "my way or highway", there is not much we can do here.

To my recollection we have never said "my way or highway". We acknowledge the benefit of this feature, but we have always pointed at technical reasons as to the detriments of this feature. Fundamentally, we prioritize adherence to the overall protobuf ecosystem and this feature goes against that priority. We understand that this is not the preference of some users, but engineering is about evaluating different options, understanding the benefits and detriments of all options and making a choice.

After 5 years of begging through this bug being open... This is actually quite sad.

This statement makes it sound like we don't care about users. I should note that the fact that we spend significant amount of time engaging with people (even if we may disagree on technical points) should hopefully show that we care.

[tandr]

I can go back starting from a very first response on this thread, and continue to comb this (or other) thread(s) to support of what I have said, but it is not a productive use of my or your time.

You reopened this thread to gather users opinion? I think it is quite clear by now that we (users) wholeheartedly voted for it. I think there is quite an obvious way to "show that we care" other than continue to discuss it.

Thank you for your time.

[mfridman]

Thank you @dsnet. You've done a great job supporting the project over the years.

As a community we aim to provide feedback on real world implementations and sought-after features. I don't have much to add outside echoing @kulak where in most/all projects I've found myself copying data from/to PB.

There is a natural flow into the DB layer and the Go ecosystem makes heavy use of tags within that layer. Capturing this in a single generated object would be a huge boost in productivity for the Go community.

I may not agree with your final stand, but respect the adherence to the overall protobuf ecosystem. Keep up the great job.

[meling]

I understand the engineering decision, but it is nonetheless sad. Some projects don’t care about compatibility with other languages. And doing manual copying between structs is error prone, and boring... I’m not hopeful that someone will pickup and fix gogo, so that it becomes compatible with this library, at least not in a releasable form in the next five years... unless someone put a full time team on it for the next year or so... it will probably be easier to start from scratch.

But I guess we can just hack this ourselves... I might do that when I get some spare time. Not likely to happen soon, so if anyone beats me to it, let me know.

[ydnar]

Candidly, we’ve concluded the cost of abandoning protos at the DB layer is lower than continuing to work with them due to lack of struct tags.

I would summarize the maintainers’ stance as the Protobuf ecosystem is more important than the language ecosystem(s) that it works within.

Given that most of us work primarily with language $x (in this case Go) more than we do with protos, this is petty sad.

[ydnar]

We should not add features to the Go protobuf implementation which are not accessible to other implementations.

You should absolutely add language-specific features which are not accessible to other implementations if it means that language’s implementation conforms to the norms and expectations of that language. Every other Protobuf implementation does. Why is Go excluded?

A way to interrogate whether a feature of Go protobufs is appropriate or not is to conduct a thought experiment: If we have a Go service that reads and writes protobuf messages using that feature, can we practically rewrite it in another language without changing the message definitions or affecting clients of the service?

Folks use Protobufs to define schemas used for other purposes than talking to a service. For example—WhatsApp uses it to serialize encrypted messages in a SQLite blob field. Square (and others) it to serialize complex data structures to MySQL.

Imagine, for example, a feature which disables UTF-8 validation of string fields. This would be useful for some Go users, since Go strings are not required to contain UTF-8 data. However, Java strings are; the Java protobuf implementation has no way to represent non-UTF-8 string data. We could not rewrite a Go service using non-UTF-8 strings in Java without changing the field definition (from string to bytes) or making changes to the Java protobuf implementation.

You mean like this Java-specific feature?

Regardless, your example (UTF-8 validation) is specious. Struct tags or the Go field names (see #555) only affect the Go implementation, and do not affect the interoperability of the protobuf messages.

Do you plan to remove json_name and JSType?

One use case is message validation: A tag will be set on some fields indicating what data it may contain, and some package will inspect those tags when checking a message for validity. There is no good way to write that validator in some other language, since it depends on Go-specific annotations.

Sure, struct tags can be used (or abused) for purposes outside of serialization. Just because there’s a footgun doesn’t mean folks are going to use it. Should you not trust your users, instead of trying to shield them from vague hypotheticals?

In both the case of validation and custom serialization, using custom field tags prevents interoperability with other protobuf implementations, are an inferior alternative to custom field options accessed via protobuf reflection, or both.

Hah, nope.

[neild]

You mean like this Java-specific feature?

I should have been clearer that I was referring to validation of proto3 string fields, which that setting does not, to my knowledge, affect.

Do you plan to remove json_name and JSType?

The json_name option is language-neutral. It affects the JSON serialization of a field, in much the same way field numbers affect the binary serialization.

The JSType option is identical to the string/[]byte hypothetical I gave as an example of something that does not impact interoperability. (I do want to add the ability to select between those two types for the Go representation of proto string and bytes fields; it just hasn't made it to the top of the priority list yet.)

[ydnar]

The JSType option is identical to the string/[]byte hypothetical I gave as an example of something that does not impact interoperability. (I do want to add the ability to select between those two types for the Go representation of proto string and bytes fields; it just hasn't made it to the top of the priority list yet.)

JSType is really close to what the OP for this issue describes.

An option to select between []byte and string representation sounds terrific. What about extending this to well-known-types, like using *string for StringValue, or time.Time for Timestamp?

[tandr]

Candidly, we’ve concluded the cost of abandoning protos at the DB layer is lower than continuing to work with them due to lack of struct tags.

@ydnar , we are about to face the same dilemma, and I am curious what did you choose as a replacement?

[ydnar]

Candidly, we’ve concluded the cost of abandoning protos at the DB layer is lower than continuing to work with them due to lack of struct tags.

@ydnar , we are about to face the same dilemma, and I am curious what did you choose as a replacement?

We took the generated structs from protoc, deleted the protobuf-specific code, leaving struct tags as necessary. We already had a translation later between our db protos and API protos, so that layer didn’t need much additional work.

[mwmahlberg]

I guess the underlying problem is that protobuf messages are considered to be DTOs, and not primary entities.
While this is a common pattern in other languages, it is diametral to Go, imho.

Go walks quite a mile to have your primary entities to be used as DTOs as well -- one of its beauties. But I guess if one wants to use protobuf messages, it seems a DT layer is mandatory, as well as adding a translation of DTOs to primary entities and vice versa.

One thing I can not wrap my head around, however, is that we are talking of protoc-gen-go: This is language specific, so it would not hurt other languages to have language specific features in the code generation -- after all, the according markup could be ignored in other languages by default.

[gattytto]

I'm going to re-open this issue, but it does not mean that we're going to support this. I'd like to see more discussion on the feasibility of this. Some thoughts:

• I can see the usefulness of this feature for Go-heavy projects, but protobufs were designed a language-agnostic way to interchange data. Baking more Go-specific details in the proto files seems to be antithetical to the philosophy of protobufs.
• This features makes the assumptions that generated Go protobufs are always structs with fields. What does it mean if we want the ability to generate opaque protobuf types that are interacted via methods. Performance testing has shown that techniques like lazy unmarshaling can bring significant speed benefits, but is only feasible as an opaque type.
• The proper way to support this seems to be using protobuf options, but I don't see any proposal here suggesting a syntax for how that would work.

this is an old issue but I still see tags in pb.go files so maybe a sane approach at a .proto3 level could be to add a CRD generator (protoc-gen-crd plugin) and then allow to use it as a base for stuff like knative objects used by controllers or maybe php ORM yml custom schemas.

[ajwerner]

I imagine there should be another issue for this discussion, and I am happy to open one, but this seemed like the place for the relevant discussion. That's especially true given the locking of #1142.

I'm sort of stumped by the outcome of #1142 given the language from @neild regarding type control. I second the question in #52 (comment) though do understand that this is something of a slippery slope and may lead towards protocol buffer files which are too closely tied to an implementation..

What I'm really stuck on is nullability. I don't follow the principled argument against adding options to interact with the behavior of protoc-gen-go w.r.t. to nullability. Running back through the though experiment, it seems that such an option does not violate the language-switching principle.

For what it's worth, the performance wins of avoiding allocations has lead cockroachdb/cockroach to use gogo/protobuf despite the fact that it is not maintained and will not obviously ever have support for the new library which is so much better. I suppose adding something like object pooling via something like what maybe is being discussed in #1192 could be helpful but tracking object lifetimes is quite hard.

It feels like this has been a prolonged philosophical debate but I feel like the philosophy got lost w/ struct tags, which I can get behind. That argument however has extended to cover other more agnostic portions of the generated code. Value types are a major feature of go to provide not just access locality but to dramatically reduce the number of allocated objects making it more reasonable for performance sensitive code despite having a garbage collector. The lack of ability to utilize this key property of the language without forking the code generator feels confusing and at odds with a commitment to support go as a first-class user of protocol buffers.

---

FWIW I do feel that controlling the type of the generated structs for messages is valuable but I don't have a clear vision on how to expose that protoc in a clean way just yet.

[ajwerner]

I've submitted #1225 as a concrete and focused issue related to the above comment which was off-topic for this issue.

[searKing]

I'd like to add that this feature would be extremely useful for validation purposes.

I have written a tool named protoc-gen-go-tag to this feature, custom struct tags for protobuf!
This seems to be nice, as it's implemented by protobuf and golang's AST.

• install: go get -u github.com/searKing/golang/tools/cmd/protoc-gen-go-tag
• compile: protoc -I . --go-tag_out=paths=source_relative:. *.proto
• examplse: https://github.com/searKing/golang/tree/master/tools/cmd/protoc-gen-go-tag/examples
• source code: https://github.com/searKing/golang/tree/master/tools/cmd/protoc-gen-go-tag

[FedorLap2006]

So, is there any news about this customization ability for protoc-gen?

[xmlking]

tested https://github.com/srikrsna/protoc-gen-gotag
it works with buf
buf.gen.yaml

version: v1beta1
plugins:
  - name: go
    out: .
    opt: paths=source_relative
  - name: gotag
    out: .
    opt: paths=source_relative
  - name: go-grpc
    out: .
    opt: paths=source_relative,require_unimplemented_servers=true
  - name: validate
    out: .
    opt: lang=go,paths=source_relative

[panol]

Need this feature,A lot of request and response are not standard json lowercase ,so we need this feature to custom json tag.