spec: review/clarify uses of "slice of bytes"

[griesemer]

This is a follow-up issue for #23536: Comb through documentation (spec, builtins) to verify the use of the term "slice of bytes". The spec appears to mean by this any type whose underlying type is []byte. But not all implementations agree.

[mdempsky]

I just ran into this issue. I think it's valuable to at least come to a shared agreement on what we think the right behavior is, so we can start aligning the implementations.

I think the spec cases relevant here are:

1. Conversions: string -> []byte, string -> []rune, []byte -> string, []rune -> string
2. copy([]byte, string)
3. append([]byte, string...)

My 2c is that the spec should allow this compilation unit:

package p

type (
	String string
	Byte   byte
	Bytes  []Byte
	Rune   rune
	Runes  []Rune
)

var (
	s  String
	bs Bytes
	rs Runes
)

func f() {
	bs = Bytes(s)
	rs = Runes(s)

	s = String(bs)
	s = String(rs)

	copy(bs, s)
	bs = append(bs, s...)
}

That is, I think we should liberally interpret "slice of bytes" and "slice of runes" to mean "slice type with element type of underlying type byte/rune". I.e., defined slice types and (user-)defined element type should be allowed.

gccgo accepts this.

cmd/compile and go/types currently reject the copy call, claiming that Bytes's element type (Byte) is different from byte.

go/types currently additionally rejects the append call too.

(Also notably, cmd/compile's typechecker allows String(Bytes(nil)) and String(Runes(nil)), but then walk.go fails to generate the appropriate conversions when calling runtime.slicebytetostring and runtime.slicerunetostring, so compilation fails. It has a similar problem with the append call too.)

[mdempsky]

/cc @griesemer @ianlancetaylor

[griesemer]

@rsc, @ianlancetaylor, @robpike Do you have any opinions here?

We're inconsistent at the moment and different implementations (cmd/compile, gccgo, go/types) differ for operations where this matters.

[bradfitz]

Personally, if there's any ambiguity that needs to be specified, I'd prefer to be a permissive as possible, letting copies/appends/conversions work as expected if the underlying representation is the same.

[ianlancetaylor]

I'm inclined to agree with @mdempsky 's comment #23814 (comment) . I don't see any particular reason to restrict operations on a "slice of bytes" to only the type []byte.

[josharian]

I just rediscovered this in #36965.

Is there sufficient consensus here that I can fix cmd/compile’s implementation to use a maximally liberal interpretation of “byte slices?

Other things to check on once the desired direction is clear: is the spec clear enough? Do go/types, package reflect, cmd/compile, and gccgo agree?

[josharian]

(The reason I’d like to fix cmd/compile now/soon is that I’m touching this code anyway for #36890.)

[griesemer]

@josharian The reason for this issue is that not all implementations agree. @mdempsky 's comment explains how they disagree.

I think we should first get the spec to be crystal clear before making changes. It sounds like we're leaning towards making the interpretation more liberal, so this should not break future code, but enable more. This is not urgent, either.

[griesemer]

Marking for Go 1.15.

[griesemer]

It looks like the consensus here is that "slice of bytes" should be maximally liberal, which would mean any slice of elements whose underlying type is bytes.

[ianlancetaylor]

@griesemer This issue has milestone 1.18; shall we move to 1.19 or backlog?

[griesemer]

This is documentation that we might be able to address. Prefer to leaving open for now.

[griesemer]

Moving to 1.19.

See also: test cases.

[gopherbot]

Change https://go.dev/cl/412094 mentions this issue: spec: clarify "slice of bytes" and "slice of runes" through examples

[griesemer]

The above CL adjusts the spec to interpret slice of bytes/runes in the most liberal way (underlying type of the slices' element types must be byte/rune) for conversions. This matches the current implementation (and closely matches 1.17 - see the CL description).

The behavior for copy/append is (and always was) more strict; also, also no explicit conversion happens in those cases, so the more strict behavior can be justified.

Since this issue is about "slice of bytes/runes", terminology that is only used in the Conversions section of the spec, the CL addresses only that aspect.

If we want to change the behavior for append/copy, we probably need to file a mini-proposal.

[gopherbot]

Change https://go.dev/cl/412095 mentions this issue: test: add tests for string/[]byte/[]rune conversions

[zigo101]

Maybe, the reflect implementation should also be made consistent with this change: #23536 (comment)

[griesemer]

@go101 Filed #53523.