Gazelle is having a hard time with protobuf files

[clanstyles]

Our application is structured in the following way:

services/users/main.go
services/users/proto/user/user.proto

When ever we generate the BUILD files it seems to have a hard time importing the protobuf files. The following is the output from our BUILD file which fails.

load("@io_bazel_rules_go//go:def.bzl", "go_library")
load("@io_bazel_rules_go//proto:def.bzl", "go_proto_library")

proto_library(
    name = "brand_proto",
    srcs = ["brand.proto"],
    visibility = ["//visibility:public"],
    deps = [
        "//github.com/xxx/jscode/jsge/pkg/media/proto/media:media_proto",
        "//github.com/xxx/jscode/jsge/pkg/paging/proto/page:page_proto",
    ],
)

go_proto_library(
    name = "brand_go_proto",
    compilers = ["@io_bazel_rules_go//proto:go_grpc"],
    importpath = "github.com/xxx/jscode/jsge/services/brands/proto/brand",
    proto = ":brand_proto",
    visibility = ["//visibility:public"],
    deps = [
        "//github.com/xxx/jscode/jsge/pkg/media/proto/media:go_default_library",
        "//github.com/xxx/jscode/jsge/pkg/paging/proto/page:go_default_library",
    ],
)

go_library(
    name = "go_default_library",
    srcs = [
        "brand.go",
        "brand.micro.go",
    ],
    embed = [":brand_go_proto"],
    importpath = "github.com/xxx/jscode/jsge/services/brands/proto/brand",
    visibility = ["//visibility:public"],
    deps = [
        "//jsge/pkg/media/proto/media:go_default_library",
        "//jsge/pkg/paging/proto/page:go_default_library",
        "//jsge/vendor/github.com/golang/protobuf/proto:go_default_library",
        "//jsge/vendor/github.com/micro/go-micro/client:go_default_library",
        "//jsge/vendor/github.com/micro/go-micro/server:go_default_library",
    ],
)

It fails with the error code:

ERROR: /home/styles/go/src/github.com/xxx/jscode/jsge/services/brands/proto/brand/BUILD.bazel:14:1: no such package 'github.com/xxx/jscode/jsge/pkg/media/proto/media': BUILD file not found on package path and referenced by '//jsge/services/brands/proto/brand:brand_go_proto'
ERROR: Analysis of target '//jsge/apis/merchants:merchants' failed; build aborted: no such package 'github.com/xxx/jscode/jsge/pkg/media/proto/media': BUILD file not found on package path
INFO: Elapsed time: 0.602s

If I try to change the deps path by hand, and remove the github.com/xxx/jscode then it will find the right path but still fail to load:

ERROR: /home/styles/go/src/github.com/xxx/jscode/jsge/services/brands/proto/brand/BUILD.bazel:4:1: Generating Descriptor Set proto_library //jsge/services/brands/proto/brand:brand_proto failed (Exit 1)
github.com/xxx/jscode/jsge/pkg/paging/proto/page/page.proto: File not found.
github.com/xxx/jscode/jsge/pkg/media/proto/media/media.proto: File not found.
jsge/services/brands/proto/brand/brand.proto: Import "github.com/xxx/jscode/jsge/pkg/paging/proto/page/page.proto" was not found or had errors.
jsge/services/brands/proto/brand/brand.proto: Import "github.com/xxx/jscode/jsge/pkg/media/proto/media/media.proto" was not found or had errors.
jsge/services/brands/proto/brand/brand.proto:42:5: "PagingRequest" is not defined.
jsge/services/brands/proto/brand/brand.proto:47:4: "PagingResponse" is not defined.
jsge/services/brands/proto/brand/brand.proto:91:5: "PagingRequest" is not defined.
jsge/services/brands/proto/brand/brand.proto:96:5: "PagingResponse" is not defined.
jsge/services/brands/proto/brand/brand.proto:127:5: "Media" is not defined.
Target //jsge/apis/merchants:merchants failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.570s, Critical Path: 1.29s
INFO: 0 processes.

Any ideas how to fix this?

I think bazelbuild/bazel#3867 is the same issue?

[jayconrod]

Yep, bazelbuild/bazel#3867 is the issue.

Whenever you build a proto_library with Bazel (for in any language, not just Go), Bazel thinks the imports are relative to a repository root (either in your local repository or in any of your external repositories). I'm guessing brand.proto is importing github.com/xxx/jscode/jsge/pkg/paging/proto/page.proto. Since the file you want to import is actually jsge/pkg/paging/proto/page.proto, protoc won't find it when invoked by Bazel.

Since this is an issue with proto_library, there's not much Gazelle or rules_go can do about it. If bazelbuild/bazel#3867 is implemented, you'll be able to adjust proto_library import paths. Until then, you won't be able to build proto_library rules without modifying them.

Unfortunately, the best advice I can offer at the moment is to check in pre-generated .pb.go files and include those in your go_library rules. If you add a comment # gazelle:proto disable in your root build file, it will ignore .proto files and will include .pb.go files.

[clanstyles]

Thanks @jayconrod I'll give that a try. When I re-run Gazelle, it will remake all of the customized build files right? How can I prevent that?

[clanstyles]

This seems to be a step in the right direction. I added # gazelle:proto disable and that worked.

ERROR: /home/styles/go/src/github.com/xxx/jscode/jsge/apis/merchants/BUILD.bazel:21:1: GoCompile jsge/apis/merchants/linux_amd64_stripped/merchants~/github.com/xxx/jscode/jsge/apis/merchants.a failed (Exit 1)
GoCompile: missing strict dependencies:
	/home/styles/.cache/bazel/_bazel_styles/7d04995eee8b8aa40bbdb626339502ac/sandbox/3756034888499404562/execroot/__main__/jsge/apis/merchants/handler.go: import of "github.com/xxx/jscode/jsge/services/brands/proto/brand"
Known dependencies are:
	github.com/xxx/jscode/jsge/micro
	github.com/gorilla/mux
	github.com/micro/go-micro/client
	github.com/micro/go-micro/errors
	github.com/micro/go-web
	github.com/gorilla/mux
	github.com/micro/go-micro/client
	github.com/micro/go-micro/errors
	github.com/micro/go-web
Check that imports in Go sources match importpath attributes in deps.

I think Gazelle is out of touch with the dependencies. I was using Dep (haven't moved to vgo).

[jayconrod]

When I re-run Gazelle, it will remake all of the customized build files right? How can I prevent that?

Gazelle doesn't replace existing build files; it generates rules and merges those rules with rules in existing files (creating those files if they don't exist). So rules, attributes, and comments that Gazelle didn't generate will be intact.

I think Gazelle is out of touch with the dependencies. I was using Dep (haven't moved to vgo).

When you re-run Gazelle, it should update all the dependencies. In the go_library rule //jsge/apis/merchants:go_default_library, you should see the rule //jsge/services/brands/proto/brand:go_default_library in deps. That rule should have importpath = "github.com/xxx/jscode/jsge/services/brands/proto/brand".

If any part of that isn't right, let me know and we can figure out what went wrong.

[jmillikin-stripe]

bazelbuild/bazel#3867 has been fixed upstream, and will be released in Bazel 0.22

[jmhodges]

Do y'all have a strat for handling this in external dependencies until 0.22 is released? I just got burned by it when I upgrading to a new k8s.io/apimachinery that had this issue.

[jmhodges]

Oh! I totally missed that go_repository had gained the build_file_proto_mode = "disable", capability. Thanks!

[jmhodges]

I think this one can be closed now! 🎉