encoding/binary: add var NativeEndian; also x/sys/cpu.IsBigEndian

[bradfitz]

I'd like to revisit #35398 (proposal: encoding/binary: add NativeEndian). Sorry. I know the history & people's opinions already. But I come with a story! 😄

Go 1.19 added support forGOARCH=loong64 (https://go.dev/doc/go1.19#loong64)

So naturally somebody wanted to compile tailscale.com/cmd/tailscaled with GOOS=linux GOARCH=loong64. Compilation failed.

It turned out we have four different native endian packages in our dependency tree:

• https://pkg.go.dev/tailscale.com/util/endian (const Big = false, var Native = binary.LittleEndian)
• https://pkg.go.dev/github.com/josharian/native (var Endian = binary.LittleEndian)
  • example dep path: "tailscale.com/net/interfaces" => "github.com/mdlayher/netlink" => "github.com/mdlayher/netlink/nlenc" "github.com/josharian/native"
• https://pkg.go.dev/github.com/u-root/uio/ubinary (var NativeEndian = binary.LittleEndian)
  • example dep path: "tailscale.com/wgengine/router" => "tailscale.com/net/dns" => "tailscale.com/net/tstun" => "github.com/insomniacslk/dhcp/dhcpv4" => "github.com/u-root/uio/uio" => "github.com/u-root/uio/ubinary"
• wireguard-go's upcoming wireguard/endian (in review at https://github.com/WireGuard/wireguard-go/pull/64/files#diff-8b7f475f3cbc65d1d51dd1c959d0232aad2aed0fbf7967207a97bfb0f94abdba) because it doesn't take external dependencies outside of std or golang.org/x/*

So we had to update all four, along with the various requisite go.mod bumps.

Some observations:

• they're all very similar
• people don't like taking dependencies on other big packages. @josharian's github.com/josharian/native that he mentioned in proposal: encoding/binary: add NativeEndian #35398 (comment) is closest, but lacks the constant that we ended up needing in Tailscale. So everybody makes their own local copies instead. That works until a new GOARCH comes along. Maybe that's rare enough? But I'm sure more riscv* variants will come along at some point.

x/sys/cpu already has this code:

https://cs.opensource.google/go/x/sys/+/refs/tags/v0.3.0:cpu/byteorder.go;l=44

And it has even more GOARCH values (for gccgo) than any other package has!

So everybody has a different subset of GOARCH values it seems.

I know people don't want to encourage thinking about or abusing endianness, but it's a reality when talking to kernel APIs. And this is kinda ridiculous, having this duplicated incompletely everywhere.

It would've been neat if Go could've added loong64 and had a bunch of code in the Go ecosystem just work right away and not require adjusting build tags.

Alternatively, if std and x/sys/cpu are too objectionable: what about new build tags?

/cc @josharian @mdlayher @hugelgupf @zx2c4 @yetist @jwhited @raggi

[mdlayher]

Yes please. Another option: unsafe.NativeEndian to imply that you shouldn't use this unless you are aware of the implications?

[ianlancetaylor]

CC @robpike

[josharian]

@josharian's github.com/josharian/native that he mentioned in #35398 (comment) is closest, but lacks the constant that we ended up needing in Tailscale.

FWIW, I know the author of that package, and he’s amenable to adding a constant.

But an x/sys/cpu or build tag option would be better. :)

[robpike]

You know my position about byte order. Yes, it's sometimes necessary but it's used far more than it should be. I'd prefer to take up @josharian's first suggestion and not have the standard library promote the concept.

[bradfitz]

I'd prefer to take up @josharian's first suggestion and not have the standard library promote the concept.

What about const IsBigEndian = false in x/sys/cpu? That's not the standard library. And that cpu package is very much about telling you what the CPU does.

[robpike]

I'm not going to die on this hill, but know too that big/little endian is not the full story for some architectures. Maybe Go will never support a nutty layout, but who knows?

In other words, is a boolean sufficient? It might be nowadays, but I honestly don't know. It's probably sufficient to support the architectures encoding/binary does, so maybe it's enough.

At least your suggestion doesn't promote the idea of "native" byte order explicitly.

But: A single third-party package that just told you the byte order by computing it (this can only be done unsafely, which is a BIG part of why I dislike the concept) seems like the real right answer to me.

[raggi]

Is part of the problem here more one of naming, e.g. "NativeEndian", perhaps something closer to what this is used for in most good cases is the systems ABI endianness. I'm not sure of a good short name to describe that, but perhaps if we had one it would be less objectionable?

[robpike]

@raggi Not especially. It's not what it's called, it's what it represents, an internal detail that is almost always (not always, but almost always) irrelevant. A long history of bad C code has taught people that it matters more than it does.

Only unsafe code can care.

[DeedleFake]

I made this exact mistake recently. I'm working on a pure Go Wayland protocol implementation. Wayland, despite being essentially a network protocol, is constrained to Unix domain sockets for various practical reasons. Since everything's on the same machine anyways, I assume, the developers decided to just use native endianness for data being sent. Early on in my project, I created a global var byteOrder binary.ByteOrder that I then set in an init() by detecting the native endianness via unsafe. It was only later that I realized that this was 100% pointless, serving literally no purpose other than hurting the performance ever so slightly. It's pointless because all it's actually doing is getting the raw bytes of various types. This is just as easily, and far more efficiently, accomplished by just, for example, doing something like *(*[4]byte)(unsafe.Pointer(&v)) to get the raw bytes of either a int32 or uint32. I wrote a couple of functions, such as func Bytes[T ~int32 | ~uint32](v T) [4]byte, to simplify it a bit further and that was the end of it.

I have to agree with @robpike. I used to wonder why this was missing but now I just don't see the point. All it would do would be to obscure stuff that probably should be unsafe. I took a look through several packages that use the above referenced native endianness packages and none of them require a native endianness. For example, Tailscale doesn't use it much, and every usage could easily be replaced with an unsafe conversion, the same goes for Cloudflare's usage of it, and I'm not even sure what's going on in this one. On the other hand, wireguard-go has a discussion that talks about unsafe casting, but only seems to consider casting an entire struct as an alternative instead of casting just the slices that they're already handling individually anyways, which seems a bit odd to me if the goal is remove the dependency.

Unless I'm missing something, none of these seem like they actually require any kind of NativeEndian and their insistence on using a library to detect how their system reads and writes memory instead of just reading and writing that memory and letting the computer do what it's designed to do is causing issues that they just simply wouldn't otherwise have. It's also notable that every usage that I saw directly calls the methods on the ByteOrder implementation, rather than passing it to binary.Read() or binary.Write(). Calling the methods directly is even more pointless, since you have to know the size in advance anyways.

Endianness is generally not something that should affect anything outside of network protocols and file formats. In other words, it only really matters for stuff where data could be read by two processes that might be running on different computers. In those cases, the endianness needs to be explicitly declared as something standardized between the two, hence binary.LittleEndian and binary.BigEndian. I don't know what the point of a native endian implementation of binary.ByteOrder would be other than to confuse people into thinking that there's a similarity in the usage between native endianness and predefined endianness.

[zx2c4]

You have to be careful with alignment on some archs with that sort of unsafe cast.

Typically what I've done in C is memcpy to the destination and let gcc figure out whether that has to be byte by byte or can be word-wise, with load store folding removing the insufficiency.

[DeedleFake]

You have to be careful with alignment on some archs with that sort of unsafe cast.

How so, and how does a NativeEndian implementation help with that? If you can reference the memory, the computer shouldn't care what type you're treating the bytes as I would think. For example, what's the difference between

v.hdrLen = endian.Native.Uint16(b[2:])

and

v.hdrLen = *(*uint16)(unsafe.Pointer(&b[2]))

Genuinely curious. It's reading the same bytes in the same order. If there are issues with reading those bytes that seems like it should be something the compiler should handle transparently.

[josharian]

@DeedleFake the compiler only coalesces byte-by-byte memory reads/writes if it knows that it is safe to do so, based on the architecture, alignment, whatnot. The unsafe version says to the compiler: just do it, whether or not it is safe.

[DeedleFake]

What actually happens if you attempt to do

v.hdrLen = *(*uint16)(unsafe.Pointer(&b[2]))

on something like MIPS and the alignment isn't correct? Do you just get junk data? Does it panic? That's not something I've run into before, despite working with some relatively low level stuff on MIPS a bit a fair while back.

Edit: Never mind. I think I answered my own question.

[tklauser]

In the current implementation added by CL 463218 we can neither do:

binary.NativeEndian == binary.LittleEndian

or

binary.NativeEndian == binary.Little

because the underlying types nativeEndian and {little,big}Endian don't match and the code won't compile:

./endian_test.go:18:28: invalid operation: binary.NativeEndian == binary.LittleEndian (mismatched types binary.nativeEndian and binary.littleEndian)
./endian_test.go:21:28: invalid operation: binary.NativeEndian == binary.BigEndian (mismatched types binary.nativeEndian and binary.bigEndian)

Is this intended?

Context is that I wanted to add the following test to x/sys/cpu which currently fails on both little and big endian machines:

func TestIsBigEndian(t *testing.T) {
        var want binary.ByteOrder = binary.LittleEndian
        if cpu.IsBigEndian {
                want = binary.BigEndian
        }
        if binary.NativeEndian != want {
                t.Errorf("IsBigEndian = %v not consistent with binary.NativeEndian: got %T, want %T",
                        cpu.IsBigEndian, binary.NativeEndian, want)
        }
}

In Russ' comment #57237 (comment) I noticed:

We can't use a type alias for nativeEndian, nor can do...

var NativeEndian bigEndian // or littleEndian

with build tags because then

x := binary.NativeEndian
x == binary.LittleEndian

or even

binary.NativeEndian == binary.LittleEndian

would only compile on certain machines.

I'm not sure I understand this part of the comment, i.e. why the code above would only compile on certain machines in case nativeEndian is a type alias. Both littleEndian and bigEndian are defined regardless of GOARCH.

I think changing to a type alias would solve the problem state above and make TestIsBigEndian pass on both little and big endian machines. Or am I missing something?

[randall77]

On a little-endian machine, then binary.NativeEndian == binary.LittleEndian would work, but binary.NativeEndian == binary.BigEndian doesn't. It gives the error

invalid operation: binary.NativeEndian == binary.BigEndian (mismatched types binary.littleEndian and binary.bigEndian)

Something like binary.NativeEndian == any(binary.BigEndian) would work though.

And you can always do binary.NativeEndian.Uint16([]byte{0, 1}) == 1.

[gopherbot]

Change https://go.dev/cl/494856 mentions this issue: cpu: define IsBigEndian on wasm

[gopherbot]

Change https://go.dev/cl/494858 mentions this issue: cpu: add test for IsBigEndian

[gopherbot]

Change https://go.dev/cl/501615 mentions this issue: all: update x/sys to HEAD

[ianlancetaylor]

As far as I can tell this is completed. Please comment if I'm missing something.

[gopherbot]

Change https://go.dev/cl/524675 mentions this issue: _content/doc/go1.21: mention encoding/binary.NativeEndian

[dmitshur]

Also moving back to Go1.21 milestone, since that's when this was completed.
(Gopherbot automatically moved it to the next milestone because it was still open.)

[dolmen]

See also github.com/dolmen-go/endian.Native which exists since 2016 and uses code generation to keep up-to-date with GOARCH.

Also, contrary to the current flawed implementation of binary.NativeEndian my package supported and still supports comparing with == to binary.LittleEndian and binary.BigEndian (which was expected by Rob in #57237 (comment)).

[gopherbot]

Change https://go.dev/cl/581655 mentions this issue: encoding/binary: fix NativeEndian to support == with Little/BigEndian