reflect: add a function for treating C memory as a Go slice

[bcmills]

Now that cgo supports exporting Go functions to C, it would be useful for Go functions to be able to read C memory without making unnecessary copies. In particular, we would like to be able to read (and write) a C char* using functions that operate on a Go []byte (e.g. bytes.Reader or io.Reader.Read).

reflect.NewAt and reflect.ArrayOf get us partway there: we can treat the pointer to C memory as a Go pointer to an array of the correct underlying length, then use reflect.Value.Slice to convert that to a slice. Unfortunately, reflect.ArrayOf strands a small amount of memory for a unique reflect.Type for every distinct length used: http://play.golang.org/p/lc3QnMnKP_

We could consider using reflect.SliceHeader instead, but it "cannot be used safely or portably" (https://golang.org/pkg/reflect/#SliceHeader).

So it seems that we don't have an efficient solution for this conversion.

The simplest solution seems like it would be to add a function to the reflect package to construct the slice directly. Following the example of reflect.MakeSlice and reflect.NewAt, it could be something like:

    // SliceAt returns a slice value for the specified slice type, length, and capacity,
    // using p as the address of the first element and leaving its contents unchanged.
    func SliceAt(typ Type, len, cap int, p unsafe.Pointer) Value

[minux]

We already have C.GoBytes that converts a unsafe.Pointer and a length to []byte, why do we need another? After converting the pointer to a byte slice, it's very easy to convert to []T for arbitrary T.

[bcmills]

We already have C.GoBytes that converts a unsafe.Pointer
and a length to []byte, why do we need another?

1. My understanding was that C.GoBytes makes a copy. Is that not the case?
   (If correct, that implies less efficiency - particularly for large arrays - and no mutative APIs such as io.Reader.Read.)

2. It doesn't work for element types other than byte. (For example, you could envision a C API that wants to use an array of wchar_t or uint32_t.)

After converting the pointer to a byte slice, it's very easy to convert to []T for arbitrary T.

Not without copying the underlying array. Or am I missing something?

[minux]

You can convert a unsafe.Pointer p to any []T without copying as follows: (*[(1 << 31) / unsafe.Sizeof(T{})]T)(p)[:len:len]

[bcmills]

That isn't portable: on 64-bit platforms it only expresses sizes up to 2GB. (And you can't work around it by increasing the constant because then it doesn't work on 32-bit platforms.)

It also seems likely to interact poorly with analysis tools: it temporarily produces a pointer to an "array" with elements in invalid memory.

[minux]

There is way to use correct cap on all platforms: (*[1<<(^uint(0)>>27&63)/unsafe.Sizeof(T{}) - 1]T)(p)[:len:len] If the code breaks some analysis tool, then it's a bug of the analysis tool. The code is fully conforming and portable. If p is not pointing to Go heap, it should also work with future GC enhancements.

[bcmills]

1. Expecting CGo users in general to come up with those constants on their own is fairly user-hostile, when we could just provide a function that does it correctly.

2. Where in the spec does it say that pointers to arrays with arbitrary lengths are valid? gc is not the spec, and gc is also not the only Go implementation. (One could easily envision a compiler that, say, sanity-checks that all pointers to arrays point to addresses that are mapped in the program's address space.)

[bcmills]

Also, those constants aren't even correct on amd64:

$ cat test/foo.go

package main

import (
        "fmt"
        "reflect"
        "unsafe"
)

func main() {
        s := []byte("Hello, ⚛")
        n := len(s)
        p := reflect.ValueOf(s).Pointer()

        s2 := (*[1<<(^uint(0)>>27&63)/unsafe.Sizeof([]byte{}) - 1]byte)(p)[:n:n]
        fmt.Println(string(s2))
}

$ go run test/foo.go
# command-line-arguments
test/foo.go:14: type [384307168202282324]byte larger than address space
test/foo.go:14: cannot convert p (type uintptr) to type *[384307168202282324]byte

That comes from here:
https://tip.golang.org/src/cmd/compile/internal/gc/align.go?h=address+space

To me, the fact that even you - someone deeply familiar with the Go language and its implementations - couldn't get those constants right strongly suggests that we should not expect more casual cgo users to do so.

[randall77]

I agree that we should probably find a solution to this problem. We want to make cgo easier & safer.

Using SliceHeader is ok, that "cannot be used safely" warning is aimed at people trying to store Go pointers in its Data field. Storing C pointers in the Data field should be ok. The rest of the warning is basically the same as the one for unsafe.Pointer.
It's still tricky, though, you'd have to do:

h := reflect.SliceHeader{uintptr(p), n, n}
s := *(*[]T)(unsafe.Pointer(&h))

(In an ideal world, SliceHeader.Data should be an unsafe.Pointer, not a uintptr. Too late now.)

I'm hesitant to put the fix in reflect, it is too heavyweight. Maybe just something in package unsafe?

// MakeSlice makes a slice out of ptr/len/cap and stores it to s.
func MakeSlice(s, ptr unsafe.Pointer, len, cap int)

[bcmills]

Using SliceHeader is ok, that "cannot be used safely" warning is aimed at people trying to store Go pointers in its Data field. Storing C pointers in the Data field should be ok.

If that's the case, then just a doc fix might be sufficient. (I suggested a similar approach in some code I was reviewing and @ianlancetaylor said not to do it.)

I'm hesitant to put the fix in reflect, it is too heavyweight. Maybe just something in package unsafe?

Maybe? But unsafe doesn't otherwise have anything to do with built-in data structures like slices, and it's not entirely clear to me how you'd get from that to a value of slice type. Would you convert "s" to a pointer-to-slice, or to some other type (e.g. the slice itself)? If it's a pointer-to-slice, is the slice header allocated on the Go heap?

reflect.Value.Interface() at least makes the answers to those questions consistent, which is why it seems like a more natural fit to me - and SliceAt isn't that far off from the existing reflect.MakeSlice and reflect.NewAt.

[bcmills]

In particular, this snippet of existing reflect code would be equivalent to SliceAt modulo the memory leakage for the reflect.ArrayOf call:

    func SliceAt(typ reflect.Type, len, cap int, p unsafe.Pointer) reflect.Value {
      return reflect.NewAt(reflect.ArrayOf(len, typ.Elem()), p).Elem().Slice(0, len)
    }

[randall77]

You'd do something like this:

var s []T
unsafe.MakeSlice(unsafe.Pointer(&s), ptr, len, cap)
...use s...

Doing it this way gets around the polymorphic return type you would need if MakeSlice returned a slice, or the allocation you would need if MakeSlice returned an unsafe.Pointer to a slice.

I agree that this is functionality that unsafe hasn't had historically. But I think it would be the right place for this operation. On the other hand, reflect.NewAt has a similar feel, and maybe both it and MakeSlice should be together.

[ianlancetaylor]

I don't see how to use reflect.SliceHeader safely unless we exempt it from the Go 1 guarantee or promise that we will never change the representation of slices. Since in the past we've discussed changing the representation of slices, it seems unwise to commit to the current representation for all time and for all implementations. Same for reflect.StringHeader, of course.

[randall77]

To use reflect.SliceHeader you need to use unsafe, and unsafe is exempt from the Go 1 guarantee.

[oec]

FWIW, this is not just a cgo related issue, it already occurs in some of the syscal-wrappers. For example, an implementation of syscall.Mmap needs to do basically the same construction. It would be helpful if there is an API - even under unsafe - available that hides the implementation details and memory-layout for slices away. (Assuming the API doesn't need to change or can be augmented with an refined version).