go_repository: errors when protobuf imports don't match Go module root directory

[alexeagle]

What version of gazelle are you using?

0.22.0

What version of rules_go are you using?

0.24.1

What version of Bazel are you using?

3.5.0

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

Macos darwin x64

What did you do?

Copy the getting started instructions to an empty project https://github.com/bazelbuild/bazel-gazelle#running-gazelle-with-bazel

go.mod contains

module example.com/m

require (
    github.com/hashicorp/vault v1.2.3
)

generate go.deps following readme

bazel run //:gazelle -- update-repos -from_file=go.mod -to_macro=deps.bzl%go_dependencies

deps.bzl now has

    go_repository(
        name = "com_github_hashicorp_vault_sdk",
        importpath = "github.com/hashicorp/vault/sdk",
        sum = "h1:b65cSyZqljnCPzzsUXvR4P0eXypo1xahQyG809+IySk=",
        version = "v0.1.14-0.20190909201848-e0fbf9b652e2",
    )

Add the proto incantation to /WORKSPACE

http_archive(
    name = "com_google_protobuf",
    strip_prefix = "protobuf-3.13.0",
    urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.13.0.tar.gz"],
)
load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
protobuf_deps()

now build the hashicorp/vault/sdk

bazel build @com_github_hashicorp_vault_sdk//...

What did you expect to see?

should build that thing

What did you see instead?

ERROR: /private/var/tmp/_bazel_alex.eagle/a7021cb2659431f216432bfcf968a0a7/external/com_github_hashicorp_vault_sdk/plugin/pb/BUILD.bazel:5:14: no such package '@com_github_hashicorp_vault_sdk//sdk/logical': BUILD file not found in directory 'sdk/logical' of external repository @com_github_hashicorp_vault_sdk. Add a BUILD file to a directory to mark it as a package. and referenced by '@com_github_hashicorp_vault_sdk//plugin/pb:pb_proto'

Observations:
That code has a proto import from sdk/...
https://github.com/hashicorp/vault/blob/70497f67a5c073094a2d5fb42ab44c21a2a3f8e8/sdk/plugin/pb/backend.proto#L7

gazelle generates this:

proto_library(
    name = "pb_proto",
    srcs = ["backend.proto"],
    visibility = ["//visibility:public"],
    deps = [
        "//sdk/logical:logical_proto",
        "@com_google_protobuf//:timestamp_proto",
    ],
)

this is wrong because we are in an external workspace com_github_hashicorp_vault_sdk so the dep should have been //logical:logical_proto

[jayconrod]

So it looks like this particular proto imports sdk/logical/identity.proto and sdk/logical/plugin.proto. The paths are relative to the repository root directory rather than the module root directory (which Gazelle thinks is the repository root directory). Gazelle will attempt to resolve the dependency using its index; failing that, it does a static conversion to //sdk/logical:logical_proto, which doesn't exist.

This can be worked around by adding this to the go_repository rule:

build_directives = ["gazelle:proto_import_prefix sdk"],

That directive is usually needed when protos are imported with some other prefix like github.com/a/b. There's also a proto_import_strip_prefix directive.

It's hard for Gazelle to do something like this automatically, since go_repository in module mode doesn't actually know whether a module is defined in its repository's root directory or not. go_repository is usually fetching a module zip file from a proxy, and there might not even be a repository behind that.

So I'm not sure I see a clear way forward here. I've been thinking that go_repository should not generate proto rules (it's rare that it does the right thing), and there should be a separate gazelle_repository rule that's more flexible but may require more control.

[alexeagle]

Our workaround for now was to disable generating protos in third-party rules. We added 14 instances of
build_file_proto_mode = "disable_global"
across all go_repository rules so it would have been better to make it a separate, opt-in ability.

[ob]

I'm seeing something similar with https://github.com/kubernetes-sigs/controller-tools, I wonder if it's the same problem:

All I did was clone that repo, create WORKSPACE and BUILD.bazel as per the Setup Section, and run:

$ bazel run //:gazelle
$ bazel run //:gazelle -- update-repos -from_file=go.mod -to_macro=deps.bzl%go_dependencies
$ bazel build //cmd/...
INFO: Invocation ID: 901741d9-c8be-4b47-87ed-ea9ab7200bcc
ERROR: /Users/obonilla/tmp/controller-tools/cmd/controller-gen/BUILD.bazel:3:11: no such package '@io_k8s_sigs_controller_tools//pkg/crd': The repository '@io_k8s_sigs_controller_tools' could not be resolved and referenced by '//cmd/controller-gen:controller-gen_lib'
ERROR: /Users/obonilla/tmp/controller-tools/cmd/controller-gen/BUILD.bazel:3:11: no such package '@io_k8s_sigs_controller_tools//pkg/genall/help': The repository '@io_k8s_sigs_controller_tools' could not be resolved and referenced by '//cmd/controller-gen:controller-gen_lib'
ERROR: Analysis of target '//cmd/controller-gen:controller-gen_lib' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.149s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (2 packages loaded, 0 targets configured)

[jayconrod]

@ob That doesn't look related to protos. Please open a new issue.

[ob]

@jayconrod sorry, my bad... I was going by the Issue title and error message. I should've read more carefully the issue to notice it was protobuf related. Filed #924. Thanks!