encoding/binary: add func Append

[lmb]

I'd like to propose adding a function with the following signature:

// Append the binary representation of data to buf.
//
// buf may be nil, in which case a new buffer will be allocated. See [Write] on
// which data are acceptable.
//
// Returns the (possibly extended) buffer containing data or an error.
func Append(buf []byte, order AppendByteOrder, data any) ([]byte, error)

This is useful when repeatedly encoding the same kind of value multiple times into a larger buffer and is a natural extension to #50601. A related proposal wants to add similar functions to other packages in encoding: #53693.

Together with #53685 it becomes possible to implement a version of binary.Write that doesn't allocate when using common io.Writer. See my comment for writeBuffer(). Roughly (untested):

func write(w io.Writer, order binary.AppendByteOrder, data any) error {
    buf := writeBuffer(w, binary.Size(data))
    binary.Append(buf[:0], order, data)
    _, err := w.Write(buf)
   return err
}

If the CLs to avoid escaping in reflect APIs lands, Append would allow encoding with zero allocations.

I think it might also allow encoding into stack allocated slices, provided the compiler is (or becomes) smart enough:

buf := binary.Append(make([]byte, 0, 128), binary.LittleEndian, v)

[ianlancetaylor]

CC @dsnet

[dsnet]

SGTM. Write is the last API in binary that lacks an append-like equivalent.

[lmb]

I've been playing with the implementation a bit, and I'd like to extend my proposal to cover an equivalent function for Read.

// Decode data from buf according to the given byte order.
//
// Returns the number of bytes read from the beginning of buf or io.ErrUnexpectedEOF.
func Decode(buf []byte, order ByteOrder, data any) (int, error)

The rationale is similar to Append: using this new function makes it possible to implement zero-allocation decode for common io.Reader.

I struggled to come up with a good name for the function, what is the inverse of Append? Renaming it to Encode seems clearer to me, and follows existing unexported types. The proposal would become:

func Decode(buf []byte, order ByteOrder, data any) (int, error) // aka Read
func Encode(buf []byte, order AppendByteOrder, data any) ([]byte, error) // aka Write

[rsc]

We have used Append as the word for all of the other appending encoders, both in this package and others; it is confusing to change to Encode now (for example, why does Encode take an AppendByteOrder and not an EncodeByteOrder? why is it Encode but AppendVarint? and so on). Let's keep using Append.

[rsc]

If we need the decoding version, it could be Parse.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

Looking at unicode/utf8, which has both AppendRune and DecodeRune, it would be fine to have Decode+Append here. But sometimes you are not actually appending, and it seems okay to add Encode too, like utf8 has EncodeRune. But Encode would not be an appender. It would return an error if the buffer was not large enough.

If we did that, we'd have all three:

func Decode(buf []byte, order ByteOrder, data any) (int, error) 
func Encode(buf []byte, order ByteOrder, data any) (int, error)
func Append(buf []byte, order AppendByteOrder, data any) ([]byte, error)

That would match utf8 better.

[gopherbot]

Change https://go.dev/cl/579157 mentions this issue: encoding/binary: add Append, Encode and Decode

[lmb]

I've dusted off the code I've had lying around and mailed a CL. Encode and Decode were straight forward additions. One interesting wrinkle: I've opted to implement all other primitive functions in terms of Append, which means I need to go from ByteOrder to AppendByteOrder:

func toAppendByteOrder(order ByteOrder) AppendByteOrder {
	switch order := order.(type) {
	case littleEndian:
		return order
	case bigEndian:
		return order
	case nativeEndian:
		return order
	case AppendByteOrder:
		return order
	default:
		return appendableByteOrder{order}
	}
}

toAppendByteOrder can only be implemented this way in encoding/binary because the types are not exported. Doing a type switch vs comparing interface values (which is possible outside of the package) does make a difference in the benchmarks.

P.S.: ToAppendByteOrder is necessary if a dependent package takes ByteOrder as an argument but would like to use Append internally for performance reasons. Without it the choice of encoding primitive „leaks“ into the API and may require breaking changes. I’d like to extend the proposal to include the function (or I can make a separate proposal).

[rsc]

Sorry but we're not going to add ToAppendByteOrder. It seems okay to have some duplication of special cases between Encode and Append.

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

The proposal is to add:

func Decode(buf []byte, order ByteOrder, data any) (int, error)
func Encode(buf []byte, order ByteOrder, data any) (int, error)
func Append(buf []byte, order AppendByteOrder, data any) ([]byte, error)

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

The proposal is to add:

func Decode(buf []byte, order ByteOrder, data any) (int, error)
func Encode(buf []byte, order ByteOrder, data any) (int, error)
func Append(buf []byte, order AppendByteOrder, data any) ([]byte, error)

[aclements]

We've been going around and around on the implementation of this and I think I just realized the central issue that's giving us trouble: why does Append take an AppendByteOrder instead of a ByteOrder? In the general case, it already pre-computes the size of the entire encoded value and grows the slice by that much. At that point, there's no need for the Append* methods of AppendByteOrder. In the current CL, it uses the Append methods for the fast path, but I'm not sure there's much value in that over just growing the slice.

[rsc]

Dropping AppendByteOrder works for me.

[gopherbot]

Change https://go.dev/cl/587096 mentions this issue: encoding/binary: adjust docs for Append, Encode and Decode