Rev of proto-gen-go to ProtoPackageIsVersion3 causing breakage

[edwarnicke]

I am hopeful that there is a simple solution to get around this issue :)

A few hours ago, this commit introduced proto.ProtoPackageIsVersion3

and set const generatedCodeVersion = 3

Our CI system (and I suspect many other peoples) do a:

go get github.com/golang/protobuf/protoc-gen-go

as part of setting up the system, and also do a:

go generate ./...

and check to make sure that regenerated files match the files in the commit (basically, checking for regen on updating the .proto files).

This has been going swimmingly until a few hours ago. In trying to debug this problem, I came to realize that the fact this worked at all is a testament to how good you guys are at backward compatibility 👍

What I discovered digging in is that we have been vendoring github.com/golang/protobuf/ v1.2.0 (because of other dependencies we have), but of course go get pulls the latest from master.

v1.2.0 lacks proto.ProtoPackageIsVersion3, so our generated code suddenly doesn't compile.

So I fell back to using:

go install ./vendor/github.com/golang/protobuf/protoc-gen-go/

in our CI. Unfortunately, this seems to result in some generation differences. If I run it locally, push, and then our CI runs it, the generated code doesn't quite match.

I get diffs like:

-var fileDescriptor_networkservice_d4de7ff5f5769d53 = []byte{
+var fileDescriptor_networkservice_5b085879d0d87bdf = []byte{

I don't precisely consider this a bug on your end, you guys are super good about backward compatibility, and at some point you do have to make the change.

Do you have any ideas about how I might navigate all of this? I suspect lots and lots of folks will shortly have similar issues...

[edwarnicke]

@dsnet do you have any insight?

[dsnet]

I'm not sure what we can do on our end, this seems to be an issue really with the Go toolchain that there isn't an easy way (or at least I don't know of one) to build a binary at a specific revision.

For the time being, you can do something like this:

$ git clone git@github.com:golang/protobuf.git && (cd protobuf && git checkout v1.2.0 && go build -o $BIN_PATH/protoc-gen-go ./protoc-gen-go) && rm -r protobuf

Be sure to have $BIN_PATH be the output path where the binary should end up.

[dsnet]

I'm going to close this as a duplicate of #751 since the general issue is that the instructions for how to install protoc-gen-go are not ideal. However, the broader issue at hand is golang/go#24250

[edwarnicke]

@dsnet Yeah... I can build the protoc-gen-go from my vendored dir (v1.2.0) but that version (unlike more recent ones) doesn’t seem to always generate the same go code for identical .proto.

At what point did proto-gen-go start generating identical go for identical proto?

[dsnet]

The output of protoc-gen-go should be deterministic. If it isn't, then that's a bug.

That said, the most likely cause of non-deterministic output is due to how protoc is invoked. The descriptor protos that are embedded in each .pb.go file contains the filepath of the proto file and depending on the -I flags to protoc, it is possible that those change.

[edwarnicke]

@dsnet Also, I don’t think is the same as #751 as it doesn’t have anything to do with modules and we are seeing it with go 1.10 (pre modules)

[edwarnicke]

@dsnet OK, that’s super helpful to know. Are the determined by the fully qualified path of included proto files? I ask, because we also just started using -I ../.../vendor in our protoc invocations.

[edwarnicke]

Which is to say, the fully qualified path of the imported protofiles (things like github.com/golang/protobufs/ptypes/empty/empty.proto) will different depending on where they are built, as we are pulling them in from vendor/ )

[dsnet]

Errr, sorry, I mispoke. It's not the -I flag. It's the how the proto file to compile was specified when given to protoc.

For example:

$ protoc --go_out=paths=source_relative:. ptypes/any/any.proto
$ mv ptypes/any/any.pb.go ptypes/any/any1.pb.go
$ (cd ptypes && protoc --go_out=paths=source_relative:. any/any.proto)
$ mv ptypes/any/any.pb.go ptypes/any/any2.pb.go 
$ (cd ptypes/any && protoc --go_out=paths=source_relative:. any.proto)
$ mv ptypes/any/any.pb.go ptypes/any/any3.pb.go 
$ cat any*.pb.go | grep RegisterFile
func init() { proto.RegisterFile("ptypes/any/any.proto", fileDescriptor_any_fea4429e4270d3cd) }
func init() { proto.RegisterFile("any/any.proto", fileDescriptor_any_f2863f9fa4d3af49) }
func init() { proto.RegisterFile("any.proto", fileDescriptor_any_0a174ad2e4d7ef58) }

Notice that all three have different values?

This behavior is surprising, but is an artifact of how protobufs were originally designed. That is, they assume a single root where all source files universally lived under in a build system similar to Bazel. It's not great for today's world of independently distributed build processes, but such is the status quo.

[edwarnicke]

OK.. so that’s a super helpful hint... let me go look for those differences and see if I can think of a solution.

[edwarnicke]

@dsnet OK... so I can't quite square this with my understanding of your explanation:

-	proto.RegisterFile("networkservice.proto", fileDescriptor_networkservice_efee0c6a43fef699)
+	proto.RegisterFile("networkservice.proto", fileDescriptor_networkservice_1d42c3056c917b94)

[dsnet]

So clearly something else is feeding into the hash. What versions of protoc-gen-go were used to produce those .pb.go files? Also what versions of the Go toolchain were used to build those binaries?

[edwarnicke]

dep has our vendored version of github.com/golang/protobuf/protoc-gen-go/ at version v1.2.0.

we are building protoc-gen-go using:

go install ./vendor/github.com/golang/protobuf/protoc-gen-go/

So I'm going to go out on a limb and say its proto-gen-go version v1.2.0 (I couldn't find any way for the protoc-gen-go binary to output a version, would love to know if there's a trick there :) ).

go version yields go1.10.5 linux/amd64

[dsnet]

And what are the versions going into the other protoc-gen-go binary being used?

[edwarnicke]

The other go version is:
go version go1.11.2 darwin/amd64
(my dev box, where I did the gen I'm pushing to our CI).

[edwarnicke]

OK... this is super weird.

I installed a downgraded go version on my laptop:
go version go1.10.5 darwin/amd64

and created a fresh new alternate GOPATH and cloned the PR that is failing our CI into it.
And ran
go generate ./...
(at top level, same as where the CI runs it).

And I am not getting different hashes. So I suspect that its not the location of the GOPATH, or the go version feeding into the hash difference... what else might it be?

[dsnet]

It's not so much that it depends directly on the Go version, but rather parts of the standard library like compress/gzip, which does not have stable output between versions.

[edwarnicke]

Ah... OK... that makes a lot of sense... hmm... I have now tried with downgrade to the same go version, but on the Mac, and seem to be OK in terms of the output not changing...

This is super weird. I appreciate your patience talking me through some of the ins and outs of how things work it makes it much much easier to figure out how to debug :)

[sjdweb]

Hi there, this issue hit me this morning. It was fine locally due to having an older version installed, but CI uses clean docker builds.

I switched to installing the vendored protoc-gen-go like @edwarnicke and all is well now!

The actual error I saw was

undefined: proto.ProtoPackageIsVersion3

[dsnet]

Hi everyone,

I apologize that 8d0c54c recently broke a lot of people's workflows. The breakage is unfortunate and several people have written to me personally requesting that we issue v1.3.0 of Go protobufs "to resolve the issue". However, I'm going to make the case that the problem at hand is not solved by any form of versioning that we do on golang/protobuf's end. Breakage is still going to happen (unless protoc-gen-go is unreasonably restricted from changing the internal implementation details of protobufs).

Fundamentally, a CI system must be hermetic such that testing a given commit of your project uses a consistent set of dependencies and provides repeatability. This implies that running the CI script on a specific version of your project today, last year, or 10 years from now should produce the exact same test results. CI scripts that hard-code a specific version of the proto package, but continues to obtain protoc-gen-go from HEAD is fundamentally not hermetic. (In fact, if any part of the CI script depends on anything from HEAD, then it is not hermetic). Breakages due to externally changing factors (e.g., 8d0c54c ) is an implicit (and undesirable) property of non-hermetic builds.

Here are things that we can do to help mitigate, but not automatically resolve the issue:

• Issue README.md: fix install instructions when used with modules #751 should fix README.md to specify how to install a specific version of protoc-gen-go. Unfortunately, this is complicated by the fact that the Go toolchain currently doesn't support directly building at a specific version. See PR README.md: document how to build a specific version of protoc-gen-go #764 for a workaround while cmd/go: allow "go get" when outside a module in module mode go#24250 (@bcmills) is still being addressed.
• Issue process: precompiled protoc-gen-go release binaries #738 requests that we release pre-compiled binaries of protoc-gen-go, which would help make it easier to download and obtain a specific version of the tool in CI scripts.

Either way, if your CI system broke as a result of 8d0c54c, then some form of change needs to happen to the CI script to ensure hermetic builds. This is not something that we can fix on golang/protobuf's end.

It is unfortunate that a change needs to be made in the user's codebase, but in this situation, I strongly believe it is actually going to put your CI script in a healthier situation.

As an example of a hermetic CI scripts, our own tests for the Go protobuf project hardcodes the exact versions of non-Go package dependencies for protoc and the go toolchain and the exact versions of Go dependencies using a go.mod file. Is it annoying to need to manually increment the versions? Yes, but well worth it in terms of guaranteeing hermetic builds.

[edwarnicke]

I'd like to second @dsnet 's point about hermetic CI. I was frankly a bit embarrassed to discover that our CI was vulnerable to this kind of thing. It was vulnerable because we were go getting (which means the version we were getting tracked master). It's a testament to the overall stability of protoc-gen-go that it took us this long to hit any issues.

@dsnet Thanks for taking the time to talk through with me my hash issues... still trying to get to the bottom of them, but your insight helped a lot.

[edwarnicke]

@sjdweb

Hi there, this issue hit me this morning. It was fine locally due to having an older version installed, but CI uses clean docker builds.

I switched to installing the vendored protoc-gen-go like @edwarnicke and all is well now!

The actual error I saw was

undefined: proto.ProtoPackageIsVersion3

Do keep an eye out for some of the other issues I've been seeing. I'm still uncertain as to whether the issues I'm hitting from downgrading to v1.2.0 are purely of my own making (most probable), but if downgrading from master to v1.2.0 is part of the cause, best to huddle together for mutual assistance in resolving them :)

[rcgoodfellow]

I'm also a bit embarrassed that our CI got caught by this too. go get -u github.com/golang/protobuf/protoc-gen-go was just too convenient.

Something that could be helpful coming from the golang/protobuf side is to put binaries at a well known stable location. In the repo release would be great, or something like the k8s release notes binaries section. Forcing users to build the tool is a bit of a complexity leak. go get handled this more or less transparently, until it didn't.

[johanbrandhorst]

dep users should version their protoc-gen-go as a required binary, like so (in Gopkg.toml):

required = [
    "github.com/golang/protobuf/protoc-gen-go",
]

You can then install protoc-gen-go from vendor:

 $ go install ./vendor/github.com/golang/protobuf/protoc-gen-go

I'm not sure what the go modules equivalent workflow is.

[longfellowone]

@trtg

Not sure what was causing the "undefined: proto.ProtoPackageIsVersion3" error for me. These are the steps I did and your go get is what fixed it.

go get -u github.com/golang/protobuf
protoc --go_out=plugins=grpc:. *.proto

using protoc --version 3.6.1

[dsnet]

I want to note some caveats with #763 (comment) suggestion:

• The issue that most people are running into is because they are building protoc-gen-go from head, but have a specific version of the proto package pinned.
• The suggestion above pins the proto package to a more recent version closer to today's head, but doesn't address underlying problem; that is, you are still building protoc-gen-go from head, but have a specific version of proto pinned.
• The README now contains instructions for how to pin a specific version of protoc-gen-go, which is a more stable path forward, than to still build the binary from head.
• Go 1.12 will make this much easier, since it will allow installing a Go binary at a specific revision. See cmd/go: allow "go get" when outside a module in module mode go#24250.

[LeoHChen]

@johanbrandhorst @edwarnicke @lavoiesl If any of you guys happen to be using go modules- this worked for me:
go get github.com/golang/protobuf@8d0c54c1246661d9a51ca0ba455d22116d485eaa
^^^
that will update your go.mod to use the specific revision needed to avoid problems

works for me. confirmed! thanks @trtg

[xiaozhoz]

problem solved.
This is because proto-go is installed by go get -u github.com/golang/protobuf/protoc-gen-go, which is the newest master branch.
you can switch to tag 1.2.0. reinstall proto-go, reprotoc all files will fix this problem.

[meysam81]

@johanbrandhorst @edwarnicke @lavoiesl If any of you guys happen to be using go modules- this worked for me:
go get github.com/golang/protobuf@8d0c54c1246661d9a51ca0ba455d22116d485eaa
^^^
that will update your go.mod to use the specific revision needed to avoid problems

Didn't solve for me. Arrrggggghhh, it's frustrating....!

[meysam81]

even tag v1.2.0 didn't solve it either

[BewareARP]

even tag v1.2.0 didn't solve it either

did you find the answer yet? Please Help Me

[meysam81]

even tag v1.2.0 didn't solve it either

did you find the answer yet? Please Help Me

No man nothing so far.

[knuckolls]

In the spirit of @dsnet's comment above about hermetic builds and because i just solved this with go modules in 1.11 I made a repeatable process you can follow so you don't have to waste all the time I did.

TLDR: I pulled the commit from master and checkout & build protobuf-gen-go on that commit, then i did a go get github.com/golang/protobuf/protoc-gen-go@c823c79ea1570fb5ff454033735a8e68575d1d0f in the folder with my existing go.mod file.

Dump of the pertinent files and build process.

go.mod -- after running the above go get command. ^^

module github.com/placeholder/protobuf-go-mod-fix

require github.com/golang/protobuf v1.2.1-0.20190205222052-c823c79ea157 // indirect

Dockerfile.build

FROM golang:1.11.5-alpine3.9 as golang-builder

RUN apk add --no-cache alpine-sdk git protobuf tree

RUN    go get -d -u github.com/golang/protobuf/protoc-gen-go \
    && cd /go/src/github.com/golang/protobuf/protoc-gen-go/ \
    && git checkout c823c79ea1570fb5ff454033735a8e68575d1d0f \
    && go install

ADD . /mnt/src/

RUN    cd /mnt/src/ \
    && tree \
    && protoc -I ./api --go_out=plugins=grpc:api api.proto \
    && go build -o ./protobuf-go-mod-fix \
    && tree \
    && file ./protobuf-go-mod-fix

Then run...

docker build -t placeholder/protobuf-go-mod-fix:latest -f ./Dockerfile.build .

➜  protobuf-go-mod-fix docker build -t placeholder/protobuf-go-mod-fix:latest -f ./Dockerfile.build .
Sending build context to Docker daemon  9.216kB
Step 1/5 : FROM golang:1.11.5-alpine3.9 as golang-builder
 ---> cb1c8647572c
Step 2/5 : RUN apk add --no-cache alpine-sdk git protobuf tree
 ---> Using cache
 ---> b93e8f8c7342
Step 3/5 : RUN    go get -d -u github.com/golang/protobuf/protoc-gen-go     && cd /go/src/github.com/golang/protobuf/protoc-gen-go/     && git checkout c823c79ea1570fb5ff454033735a8e68575d1d0f     && go install
 ---> Using cache
 ---> 16ba8398861a
Step 4/5 : ADD . /mnt/src/
 ---> ed92ca48f2cc
Step 5/5 : RUN    cd /mnt/src/     && tree     && protoc -I ./api --go_out=plugins=grpc:api api.proto     && go build -o ./protobuf-go-mod-fix     && tree     && file ./protobuf-go-mod-fix
 ---> Running in d7dfc8ca2627
.
├── Dockerfile.build
├── api
│   ├── api.proto
│   └── handler.go
├── go.mod
├── go.sum
└── main.go

1 directory, 6 files
go: finding github.com/golang/protobuf v1.2.1-0.20190205222052-c823c79ea157
< snipped .. more of these lines here>

.
├── Dockerfile.build
├── api
│   ├── api.pb.go
│   ├── api.proto
│   └── handler.go
├── go.mod
├── go.sum
├── main.go
└── protobuf-go-mod-fix

1 directory, 8 files
./protobuf-go-mod-fix: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, Go BuildID=DoJscSvpyqlOUpOPWgsH/4YDWvLflS9B7ZGPcODhk/xN3P7Yw83ibPvWI20RzE/83WgRHg5an2yiudtBC0b, not stripped
Removing intermediate container d7dfc8ca2627
 ---> 3875635d4d2a
Successfully built 3875635d4d2a
Successfully tagged placeholder/protobuf-go-mod-fix:latest

Hope that helps some of you wayward internet travelers. 👋