proposal: encoding/binary: add NativeEndian

[josharian]

This was previously discussed at https://groups.google.com/forum/#!topic/golang-nuts/3GEzwKfRRQw.

The rationale given there (by multiple people, these words by Paul Borman) was:

This comes up when talking with the kernel and also with networking between processes on the same machine (or like machines) that might be using a traditionally defined structure rather than some dynamic structure.

Rob replied:

[...] almost every time I hear someone asking what the native byte order is, it's a mistake.

Russ wrote:

Put [var hbo = binary.LittleEndian] in a file named byteorder_amd64.go and it stops being a hack. It need not be in the standard library.

I'd like to add another rationale for adding it: encoding/binary can act as a poor man's vectorization. With a bit of care, you can read eight bytes into a uint64 with it, operate on the uint64, and write back eight bytes, and have the code emitted be just a MOVQ, XORQ, MOVQ. It will also be correct and safe across all architectures and alignments, because it is pure Go. See #31586 (comment) and #35381.

With neither implicit nor explicit vectorization anywhere on the horizon, this is pretty useful. However, this is most important in performance-sensitive code like crypto, where using a non-native endianness incurs a non-trivial penalty (best case a bswap).

Russ suggested using build tags. There are a few downsides to this. First, the obvious implementation (var nativeEndian = binary.LittleEndian) introduces a memory load into the hot path, since the compiler can't prove that nativeEndian is never modified and changes a direct call to an indirect one (albeit a highly predictable one). Second, the burden of maintaining a list of all architectures organized by endianness falls on all clients, rather than one centralized location.

Another option would be to add an endianness constant somewhere (package runtime? math/bits? encoding/binary?).

[rsc]

First, the obvious implementation (var nativeEndian = binary.LittleEndian) introduces a memory load into the hot path, since the compiler can't prove that nativeEndian is never modified and changes a direct call to an indirect one (albeit a highly predictable one).

In this case nativeEndian will have type binary.littleEndian. Pretty sure the compiler can prove that type never changes. :-)

I still agree with Rob that NativeEndian is almost always a mistake. Let's solve vectorization a better way than misleading API hacks in encoding/binary that we will never be able to remove.

[ddevault]

I have a different use-case, which cannot reasonably be solved in a more clever way by attacking the high-level use-cases. I am writing a Wayland protocol implementation for Go, which communicates with third-party client/server implementations over a Unix socket. The byte order for this protocol is defined as host endianness.

[josharian]

I needed this yet again, so I made this: https://github.com/josharian/native