x/sys/cpu: respect CPU feature overrides specified in GODEBUG

[Lekensteyn]

golang.org/x/sys/cpu does not allow CPU feature detection to be overridden, unlike internal/cpu. This results in surprising behavior while debugging software that do not use internal/cpu but the external golang.org/x/sys/cpu package.

@martisch already mentioned this difference in #32102 (comment).

The use case was forcing qtls in https://github.com/lucas-clemente/quic-go to use ChaCha20-Poly1305 instead of AES128-GCM in TLS 1.3 for debugging purposes.

What version of Go are you using (go version)?

$ go version
go version go1.12.9 linux/amd64

Does this issue reproduce with the latest release?

Yes, reproduced with golang/sys@5fe476d.

What did you do?

Create detect.go:

package main

import (
	"golang.org/x/sys/cpu"
)

func main() {
	print("cpu.X86.HasAES: ", cpu.X86.HasAES, "\n")
}

Run with:

./detect
GODEBUG=cpu.all=off ./detect
# demonstrate that GODEBUG is interpreted
GODEBUG=cpu.allx=off ./detect

What did you expect to see?

cpu.X86.HasAES: true
cpu.X86.HasAES: false
GODEBUG: unknown cpu feature "allx"
cpu.X86.HasAES: true

What did you see instead?

cpu.X86.HasAES: true
cpu.X86.HasAES: true
GODEBUG: unknown cpu feature "allx"
cpu.X86.HasAES: true

[martisch]

I think we should port over all CPU feature masking options that internal/cpu supports. Individual feature settings and masking all features. This will also help packages imported into the standard library that use x/sys/cpu (see #32102) to be aligned with what internal/cpu will report.

Adding others to discuss if it is ok for x/ packages to interpret GODEBUG.
/cc @randall77 @bradfitz @bcmills @aclements

[bradfitz]

Remind me why we have both internal/cpu and golang.org/x/sys/cpu? It'd sure be nice if we only had one (x/sys/cpu) and the Go runtime could just use it.

[bcmills]

To the extent that x/sys/cpu and internal/cpu remain separate, I agree that x/sys/cpu should support all of the same GODEBUG options as internal/cpu. Perhaps there should be a GODEBUG parser somewhere in one of the core x/ repos?

I also agree with @bradfitz that ideally in the long term we should find some way to consolidate on only x/sys/cpu.

[randall77]

runtime needs to import internal/cpu, so it's not trivial to use x/sys/cpu as a drop-in replacement. For example, x/sys/cpu imports encoding/binary, which imports reflect, which imports runtime.

I think there is a lot of scope to share an implementation (or large parts thereof), but it isn't trivial.

[bradfitz]

The encoding/binary dependency is not necessary.

And the only use of runtime is to look at runtime.GOARCH which we could instead inject during init and hide that file using build tags when vendoring the module into std.

It seems like we could make golang.org/x/sys/cpu a leaf package and drop internal/cpu.

[martisch]

Other complications for why runtime does not use x/sys/cpu:

• some constructs like maps can not be used anywhere as those parts of the runtime are not initialised yet when internal/cpu runs code.
• internal/cpu supports more architectures/platforms as it is has access to non public runtime internals like auxv information to derive hwcap bits which x/sys/cpu can not use directly unless we expose a runtime API or update unsafe coupling with any related runtime change.
• historical: x/sys/cpu was forked from internal/cpu because it was not public.

Let me ask the other way around when we already think about sharing a common implementation:
Why cant we just make internal/cpu public as package cpu in standard lib?

[bradfitz]

Why cant we just make internal/cpu public as package cpu in standard lib?

We've tried to stop adding new packages to std, preferring everything to be public where it can have its own release cadence. (e.g. fix bugs today, don't wait 6 months.... https://golang.org/doc/faq#x_in_std)

[martisch]

We've tried to stop adding new packages to std, preferring everything to be public where it can have its own release cadence. (e.g. fix bugs today, don't wait 6 months.... https://golang.org/doc/faq#x_in_std)

I am aware of that but I think that point will lose some of its appeal when we hide the coupling with runtime internals under the hood with build tags and unsafe. We also have exposed other std lib/runtime functions in new std lib packages lately like bytes/hash that are coupled to the runtime.

If we will have a serious bug in CPU feature detection I would want to fix that for the runtime in a point release too instead of waiting 6 months.