encoding: provide append-like variants

[dsnet]

There's been a shift in recent years to provide Append-like variants of APIs:

• utf8.AppendRune
• binary.AppendXXX
• binary.LittleEndian.AppendXXX
• fmt.AppendXXX

as these are generally easier to work with than Put-like APIs where you need to know the size of the buffer beforehand, prepare a buffer of the right size, and then call the Put-like operation.

Such APIs are missing for the encoding/hex, encoding/base32, and encoding/base64 packages. I propose we add the following:

package hex
func AppendEncoded(dst, src []byte) []byte
func AppendDecoded(dst, src []byte) ([]byte, error)

package base32
func (*Encoding) AppendEncoded(dst, src []byte) []byte
func (*Encoding) AppendDecoded(dst, src []byte) ([]byte, error)

package base64
func (*Encoding) AppendEncoded(dst, src []byte) []byte
func (*Encoding) AppendDecoded(dst, src []byte) ([]byte, error)

This would simplify many callers of these packages, where the only reason to call the Len method is to obtain the size of the expected output buffer, do some manual memory management, and then call the Encode or Decode equivalent.

---

For example, this logic in the json package:

go/src/encoding/json/encode.go

Lines 839 to 860 in c111091

	e.WriteByte('"')
	encodedLen := base64.StdEncoding.EncodedLen(len(s))
	if encodedLen <= len(e.scratch) {
	// If the encoded bytes fit in e.scratch, avoid an extra
	// allocation and use the cheaper Encoding.Encode.
	dst := e.scratch[:encodedLen]
	base64.StdEncoding.Encode(dst, s)
	e.Write(dst)
	} else if encodedLen <= 1024 {
	// The encoded bytes are short enough to allocate for, and
	// Encoding.Encode is still cheaper.
	dst := make([]byte, encodedLen)
	base64.StdEncoding.Encode(dst, s)
	e.Write(dst)
	} else {
	// The encoded bytes are too long to cheaply allocate, and
	// Encoding.Encode is no longer noticeably cheaper.
	enc := base64.NewEncoder(base64.StdEncoding, e)
	enc.Write(s)
	enc.Close()
	}
	e.WriteByte('"')

could be simplified as:

b := e.AvailableBuffer()
b = append(b, '"')
b = base64.StdEncoding.AppendEncoded(b, s)
b = append(b, '"')
e.Write(b)

This assumes we had some way to unify bytes.Buffer with append-like APIs; see #53685 (comment).

The resulting code is both simpler and more performant since we can append directly into the underlying bytes.Buffer without going through an intermediate encodeState.scratch buffer. Also, the caller doesn't have to do manual memory management checking whether the encoded output fits within the encodeState.scratch.

[mvdan]

As far as names go, we could also consider EncodeAppend and DecodeAppend, even if they aren't as consistent with the usual "verb then object" method names like MarshalJSON or AppendUint.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[gopherbot]

Change https://go.dev/cl/504884 mentions this issue: encoding: add AppendEncode and AppendDecode

[gopherbot]

Change https://go.dev/cl/520755 mentions this issue: encoding/json: use base64.Encoding.AppendEncode

[mitar]

@dsnet: Thank you for adding those. I was also looking into how to append encoded and now I see it will be out in upcoming 1.22. Great!

But I am still curious how this interacts with bytes.Buffer.

To me, the following works (using what is available with 1.21):

	v := []byte{'a', 'b'}
	b := &bytes.Buffer{}
	data := make([]byte, base64.StdEncoding.EncodedLen(len(v)))
	base64.StdEncoding.Encode(data, v)
	b.Write(data)

But if I do:

	v := []byte{'a', 'b'}
	b := &bytes.Buffer{}
	b.Grow(base64.StdEncoding.EncodedLen(len(v)))
	base64.StdEncoding.Encode(b.AvailableBuffer(), v)
	fmt.Println(b.String())

if fails with a panic. I must say I do not understand why based on discussion in #53685, it seems this should also work?

[mitar]

OK, I made it work:

	v := []byte{'a', 'b'}
	buf := &bytes.Buffer{}
	l := base64.StdEncoding.EncodedLen(len(v))
	buf.Grow(l)
	b := buf.AvailableBuffer()[:l]
	base64.StdEncoding.Encode(b, v)
	buf.Write(b)
	fmt.Println(buf.String())

No I understand why AppendEncoded is needed. :-)

[gopherbot]

Change https://go.dev/cl/547755 mentions this issue: doc: add release notes for changes to encoding packages