Getting segfault when running gopackagesdriver on v0.40.0 & v0.41.0

[vtsao]

What version of rules_go are you using?

v0.40.0

What version of gazelle are you using?

v0.24.0

What version of Bazel are you using?

bazel 6.2.1

Does this issue reproduce with the latest releases of all the above?

Seems like it only occurs with rules_go v0.40.0 as v0.39.1 works fine.

What operating system and processor architecture are you using?

Linux 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Any other potentially useful information about your toolchain?

No.

What did you do?

echo {} | ./tools/gopackagesdriver.sh

where gopackagesdriver.sh has the following contents:

#!/usr/bin/env bash
exec bazel run -- @io_bazel_rules_go//go/tools/gopackagesdriver "${@}"

What did you expect to see?

No error/panic.

What did you see instead?

Loading:
Loading:
Loading: 0 packages loaded
Analyzing: target @io_bazel_rules_go//go/tools/gopackagesdriver:gopackagesdriver (0 packages loaded, 0 targets configured)
INFO: Analyzed target @io_bazel_rules_go//go/tools/gopackagesdriver:gopackagesdriver (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Target @io_bazel_rules_go//go/tools/gopackagesdriver:gopackagesdriver up-to-date:
  bazel-bin/external/io_bazel_rules_go/go/tools/gopackagesdriver/gopackagesdriver_/gopackagesdriver
INFO: Elapsed time: 0.177s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/external/io_bazel_rules_go/go/tools/gopackagesdriver/gopackagesdriver_/gopackagesdriver
Running: [bazel info --tool_tag=gopackagesdriver --ui_actions_shown=0]
Running: [bazel query --tool_tag=gopackagesdriver --ui_actions_shown=0 --ui_event_filters=-info,-stderr --noshow_progress --order_output=no --output=label --nodep_deps --noimplicit_deps --notool_deps @io_bazel_rules_go//:stdlib]
Running: [bazel build --tool_tag=gopackagesdriver --ui_actions_shown=0 --show_result=0 --build_event_json_file=/tmp/gopackagesdriver_bep_1220076710 --build_event_json_file_path_conversion=no --experimental_convenience_symlinks=ignore --ui_event_filters=-info,-stderr --noshow_progress --aspects=@io_bazel_rules_go//go/tools/gopackagesdriver:aspect.bzl%go_pkg_info_aspect --output_groups=go_pkg_driver_json_file,go_pkg_driver_stdlib_json_file,go_pkg_driver_srcs --keep_going @io_bazel_rules_go//:stdlib]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x58e8f5]

goroutine 1 [running]:
main.(*PackageRegistry).walk(0xc0000ac1c0, 0xc00008fae0, {0xc000220060, 0x1b})
        external/io_bazel_rules_go/go/tools/gopackagesdriver/packageregistry.go:79 +0x55
main.(*PackageRegistry).Match(0xc0000b9d90, {0xc00022e130, 0x1, 0x0})
        external/io_bazel_rules_go/go/tools/gopackagesdriver/packageregistry.go:102 +0x1ba
main.(*JSONPackagesDriver).GetResponse(0xc0000ac1a0, {0xc00022e130, 0xc000028340, 0xc00022e130})
        external/io_bazel_rules_go/go/tools/gopackagesdriver/json_packages_driver.go:51 +0x25
main.run()
        external/io_bazel_rules_go/go/tools/gopackagesdriver/main.go:111 +0x5c5
main.main()
        external/io_bazel_rules_go/go/tools/gopackagesdriver/main.go:115 +0x2b

[fmeum]

@JamyDev Do you happen to have an idea what could cause this?

[JamyDev]

Yeah I just ran into this and have a potential patch as well

[JamyDev]

Effectively the recent changes to the driver here remove a critical piece that mapped a request for stdlib into a request for all subpackages of stdlib, since stdlib itself doesn't exist in the stdlib pkg.json output

[DolceTriade]

Nice fix! Was running into this as well.

[pmacustodio]

I am having the same SIGSEGV at the same line using 0.40.1 and gazelle 0.24.0

external/io_bazel_rules_go/go/tools/gopackagesdriver/packageregistry.go:79

Do not know how to call the driver directly but it happens when gazelle is doing "something" with org_golang_x_crypto:

bazel_gazelle/internal/go_repository.bzl:209:18: org_golang_x_crypto: 
gazelle: finding module path for import crypto/ecdh: finding module path for import crypto/ecdh: template: main:1:9: executing "main" at <.Module.Path>: nil pointer evaluating *modinfo.ModulePublic.Path 
gazelle: finding module path for import crypto/ecdh: finding module path for import crypto/ecdh: template: main:1:9: executing "main" at <.Module.Path>: nil pointer evaluating *modinfo.ModulePublic.Path 
panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x54bb55]  
goroutine 1 [running]: main.(*PackageRegistry).walk(0xc00013c300, 0xc00058dae0?, {0xc0002b6540?, 0x37?})
 	external/io_bazel_rules_go/go/tools/gopackagesdriver/packageregistry.go:79 +0x55 main.(*PackageRegistry).Match(0xc00013c300, {0xc000138000, 0xc4, 0xc000293d80?}) 	
    external/io_bazel_rules_go/go/tools/gopackagesdriver/packageregistry.go:112 +0x2fa main.(*JSONPackagesDriver).GetResponse(0xc000343000?, {0xc000138000?, 0xc00009b4f0?, 0xc000138000?}) 	
    external/io_bazel_rules_go/go/tools/gopackagesdriver/json_packages_driver.go:51 +0x25 main.run() 	
    external/io_bazel_rules_go/go/tools/gopackagesdriver/main.go:111 +0x5c5 main.main() 	
    external/io_bazel_rules_go/go/tools/gopackagesdriver/main.go:115 +0x2b 

Version 0.39.1 shows a DEBUG with the <.Module.Path>: nil pointer evaluating *modinfo.ModulePublic.Path but no panic. Versions 0.40.0 and 0.40.1 panic.

[andrewmbenton]

I am also seeing a panic with rules_go@0.40.1. Seems like there is still an issue and this should either be renamed and reopened, or I can create a new issue if preferred.

INFO: Running command line: bazel-bin/external/rules_go~0.40.1/go/tools/gopackagesdriver/gopackagesdriver_/gopackagesdriver
Running: [bazelisk info --tool_tag=gopackagesdriver --ui_actions_shown=0]
INFO: Invocation ID: 9a23c939-6e81-4d7c-a71d-08d22db2b952
Running: [bazelisk query --tool_tag=gopackagesdriver --ui_actions_shown=0 --ui_event_filters=-info,-stderr --noshow_progress --order_output=no --output=label --nodep_deps --noimplicit_deps --notool_deps @@rules_go~0.40.1//:stdlib]
Running: [bazelisk build --tool_tag=gopackagesdriver --ui_actions_shown=0 --show_result=0 --build_event_json_file=/tmp/gopackagesdriver_bep_4114190248 --build_event_json_file_path_conversion=no --experimental_convenience_symlinks=ignore --ui_event_filters=-info,-stderr --noshow_progress --aspects=@@rules_go~0.40.1//go/tools/gopackagesdriver:aspect.bzl%go_pkg_info_aspect --output_groups=go_pkg_driver_json_file,go_pkg_driver_stdlib_json_file,go_pkg_driver_srcs --keep_going @io_bazel_rules_go//:stdlib]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x54c015]

goroutine 1 [running]:
main.(*PackageRegistry).walk(0xc0000ba430, 0xc000161ae0?, {0xc00021c000?, 0x1b?})
	external/rules_go~0.40.1/go/tools/gopackagesdriver/packageregistry.go:79 +0x55
main.(*PackageRegistry).Match(0xc0000ba430, {0xc00021e000, 0x1, 0xc000289d80?})
	external/rules_go~0.40.1/go/tools/gopackagesdriver/packageregistry.go:112 +0x2fa
main.(*JSONPackagesDriver).GetResponse(0xc0000ba410?, {0xc00021e000?, 0xc00007d4f0?, 0xc00021e000?})
	external/rules_go~0.40.1/go/tools/gopackagesdriver/json_packages_driver.go:51 +0x25
main.run()
	external/rules_go~0.40.1/go/tools/gopackagesdriver/main.go:111 +0x5c5
main.main()
	external/rules_go~0.40.1/go/tools/gopackagesdriver/main.go:115 +0x2b

[fmeum]

CC @JamyDev

[ian-h-chamberlain]

I noticed some comments, e.g. #3659 (comment), mentioning that this could be related to bzlmod, but in our case we are not using bzlmod at all yet — we have these versions:

• Bazel 5.4.0
• rules_go 0.41.0
• Gazelle 0.24.0
• Go 1.21

Our workaround for now has been to patch rules_go by reverting #3606 and #3524 — once those are reverted, it seems to resolve the crash (i.e. work how it did before we upgraded to 0.40.0). It's unfortunate because we would like to use the feature implemented in #3524 too but having general language server features is more useful in the meantime.

[pziggo]

I can confirm the workaround mentioned by @ian-h-chamberlain (reverting both commits) also works in my context.

• Bazel 6.3.0
• bzlmod enabled
• rules_go 0.41.0

Some additional remarks from my debugging efforts so far:

1. it is mandatory to use repo_name = "io_bazel_rules_go" with bzlmod
   As of today, the IDs for the stdlib packages will always be prefixed with this module name (see https://github.com/bazelbuild/rules_go/blob/master/go/tools/builders/stdliblist.go#L110)

roots: map[@//my/package:lib:{} @//my/package:lib_test:{} @rules_go//:stdlib:{}]
packageIDs: [@io_bazel_rules_go//stdlib:crypto/ecdh ... ]

2. in my context, the label for stdlib looks weird as it doesn't treat stdlib as a package but as a target:

	Running: [bazel build --tool_tag=gopackagesdriver --ui_actions_shown=0 --show_result=0 --build_event_json_file=<STRIP> --build_event_json_file_path_conversion=no --experimental_convenience_symlinks=ignore --ui_event_filters=-info,-stderr --noshow_progress --aspects=@@rules_go~0.41.0//go/tools/gopackagesdriver:aspect.bzl%go_pkg_info_aspect --output_groups=go_pkg_driver_json_file,go_pkg_driver_stdlib_json_file,go_pkg_driver_srcs --keep_going //my/package:lib //my/package:lib_test @io_bazel_rules_go//:stdlib]

root for walk: @io_bazel_rules_go//:stdlib
packageIDs: [@io_bazel_rules_go//stdlib:internal/goarch ...]
pkg: (*main.FlatPackage)(nil) <- causing the SIGSEGV

3. in my context, the canonical repository name for the packages inside the workspace are not prefixed with the double @@ while assembling the list of roots:

roots: map[@//my/package:lib:{} @//my/package:lib_test:{} @io_bazel_rules_go//:stdlib:{}]
packageIDs: [@io_bazel_rules_go//stdlib:crypto/ecdh ... @@//my/package:lib @@//my/package:lib_test]

[fmeum]

We probably need to wait for #3659 to be fixed before we can fully address this, which in turn requires bazelbuild/bazel#19508 and thus Bazel 6.4.0 (hopefully we can still get it in).

[pziggo]

@fmeum I agree, for a proper fix the output of bazel query must be consistent. But I would still propose #3700 as intermediate fix to workaround this limitation. What do you think?

[seanmorton-afs]

I noticed some comments, e.g. #3659 (comment), mentioning that this could be related to bzlmod, but in our case we are not using bzlmod at all yet — we have these versions:

* Bazel 5.4.0

* rules_go 0.41.0

* Gazelle 0.24.0

* Go 1.21

Our workaround for now has been to patch rules_go by reverting #3606 and #3524 — once those are reverted, it seems to resolve the crash (i.e. work how it did before we upgraded to 0.40.0). It's unfortunate because we would like to use the feature implemented in #3524 too but having general language server features is more useful in the meantime.

TL;DR; for anyone new to this issue: an official fix has been merged but only for bazel version +6.4.0, for anyone on an older bazel version, this advice from Ian to create a patch from these two revert commits fixes the problem.

[JamyDev]

fwiw we have some patches at Uber that resolve this segfault. I'm looking at contributing this back, but not without writing some integration tests for this driver first.

Patch 1 fixes the panic, but may mean some packages don't get resolved. For us this was fixed with adding some extra attrs to the aspect in the driver.
rulesgo-gpd-panic.patch

Seemed that for us the trouble was that in case of code generated through one of our custom rules that defines a plugins attribute, bazel wouldn't run the aspect on those plugins (it wasn't an attr in the aspect, so that's sort of expected), and causing missing data in the registry of the packagesdriver. Now that shouldn't segfault, hence patch 1, but the root cause was elsewhere.