Description
What version of Go are you using (go version
)?
N/A, but mostly 1.14
Does this issue reproduce with the latest release?
No. It is only expected to reproduce some day in the future. This is sort of a future-proofing thing, where there's been a lot of thought and effort going into writing correct code against a hypothetical future moving implementation of GC, but since we don't have one, we don't know whether we're succeeding...
So, in general, right now, the go GC does not move things, and we know it won't move things. However, it could, so we try to write code that's robust against this. I was previously thinking about this in a now-declined proposal, #32115, which I think was approaching it in the wrong way.
On further study of some of the use cases for this, I've concluded that there is an actual general need for a way to express "this memory has to not-move because something somewhere will be relying on this address in a way the runtime can't do anything about". One example is pointers in parameters passed to ioctl
calls; some ioctl calls take data structures which may contain pointers, and the Go runtime can't really be expected to know about all such potential things, but worse, by the time the ioctl call starts and go knows it should be pinning things, the things indirectly referenced by some such data structure could have already been moved.
But there's another potential case, which is alignment. So far as I can tell, while Go presumably preserves its own alignment requirements on data, that doesn't mean that it preserves anything else's alignment requirements for data. Consider a []byte which is being used to store a buffer of raw data which can be type-punned and interpreted as some other data structure. Go has no way of detecting that the alignment requirements are significant.
pkg/unsafe says:
It is valid both to add and to subtract offsets from a pointer in this way. It is also valid to use &^ to round pointers, usually for alignment. In all cases, the result must continue to point into the original allocated object.
So imagine that I'm doing something which interacts, possibly via cgo, with hardware. I might actually care about alignment larger than anything Go cares about. For instance, say I want a 16KB block that is 16KB-aligned. I can do that at all in Go:
b := make([]byte, 32768)
p := unsafe.Pointer(&b[16383])
p = unsafe.Pointer(uintptr(p)&^16383)
So far as I know, this gives me a valid unsafe.Pointer which definitely refers to 16KB of 16KB-aligned memory... Except that a hypothetical moving implementation might move b. And I assume it's smart enough that it can detect that p is a pointer into that same object, and fix it, but it's less obvious to me that the new location will definitely be aligned the same way.
There's only a handful of cases where this applies, and in the absence of an actual moving implementation, it's obviously trivial to "implement" the desired functionality that things don't move while pinned. On the other hand, some of these things are so far as I can tell impossible to write safely without something like pin. (You can get some of the desired behavior by using mmap to request anonymous regions that runtime doesn't manage, but then you have to be sure you're never storing anything in those
regions that contains pointers into runtime-managed memory that could be the only pointers to those things...)
An approximate sketch of an API:
unpin := runtime.Pin(addr)
defer unpin()
// code that uses addr and can assume it won't move
Alternatively:
runtime.Pin(ctx, addr)
where ctx is an arbitrary context, and the pin lasts until the context is cancelled.
It should probably be permissible for a pin to never get cancelled, with the caveat that if there's a lot of uncancelled pins, it could be bad.