cmd/cgo: document rules for pointer sharing

[rsc]

We need to document what the rules are.

[cznic]

If not already considered a part of the issue, let me please ask for one more, I believe closely related thing above the subject: Pinning guarantees rules, ie. either defining an explicit mechanism, maybe an annotation //go:pinned, for making things with static addresses; or having an indirect mechanism, like eg. that b := make([]byte, n, m) under certain specified conditions produces an immovable &b[0], etc. Actually supporting a byte slice is enough (for me). What about automatically, always and everywhere pinning any unsafe.Pointer???

[cespare]

Does this issue supersede #8310?

[ianlancetaylor]

@cznic Do you have a need for pinning beyond passing pointers between Go and C?

I don't think we are likely to implement a //go:pinned annotation. For one thing, pinning is a concept associated with specific pointers, and annotations in comments are a concept associated with specific variables (or functions).

[dr2chase]

If we were going to pin things, I would pin any memory allocated that had a cgo-related type (C.uint, C.char, etc) or array of cgo-related type. So your example might be
b := make([]C.char, n, m)
Since the purpose of these C types is to pass them (and pointers to them) to C, and passing them to C is the main reason for pinning, this seems to me like a good match.

[chai2010]

@dr2chase Good idea! +10086

[cznic]

@ianlancetaylor

Do you have a need for pinning beyond passing pointers between Go and C?

Yes I do. I'm looking into a cgo alternative which translates, when possible, C code to Go code with equivalent semantics. The translated code uses the unsafe package, yet it has only the same inherent memory unsafeness of the original C code, nothing to lose here; but the call-into-C overhead is gone. Also, installing such translated-to-Go C library is as easy as go get.

In this scenario, p = malloc(n); can be translated to something like p = unsafe.Pointer(&make([]byte, n)[0]), plus keeping p reachable. *p escapes and will not be put in the stack and thus not moved by the current Go compilers, AFAIK. But there are no guarantees that the same will hold for any future/other Go implementations.

I assume that moving the backing array of the above example by the Go runtime at GC is actually fine as long as p stays as a translated-from-C variable only - then it would get correctly updated. The problem arises when p itself is stored in the C heap space, now a Go byte slice backing array, and becomes invisible to the Go runtime wrt to updating due to moving heap items around, which is so far a completely legal thing to do.

[dr2chase]

I'm not sure how much experience you have with this sort of thing, but it might be worth looking at how this was done in other languages (e.g., Modula-3 or Modula-2+, Susan Owicki wrote a paper on this a long long time ago http://dl.acm.org/citation.cfm?id=567539 ). There is a standard pattern that has worked in other languages that use copying (object-moving) garbage collectors -- stuff for native code is allocated by malloc, if the GC is supposed to backstop its lifetime management you wrap that malloc-pointer into a heap-allocated object with a finalizer that calls free.

One thing to be aware of is that Go stacks can move, and Cgo contains a handshake to ensure that stuff passed to C is stored on the GC'd heap (because right now, that memory does not move) -- this means that if you write code that (you think) avoids a heap allocation for something you pass to C, that actually you are getting both a copy AND a heap allocation.

[ianlancetaylor]

@cznic That's a pretty specialized use case, and you can, and probably should, resolve it by implementing your own syscall.Mmap based memory allocator (after all, the C heap is not garbage collected, so why should your translation of the C heap be garbage collected). At the moment I don't see it as a convincing reason to add a complex feature to the Go GC.

[cznic]

@ianlancetaylor

resolve it by implementing your own syscall.Mmap based memory allocator...

Yep. Of course it would be easier to use the existing allocator. But not only that. The intention is to, in the next step, be able to also produce (slower) code, which however is not using package unsafe and qualifies (some of) the translated C libs to be used in restricted environments like the GAE/GCE.

I don't have yet a working proof of concept for the 'next step', but so far I haven't figured out any other blocking issue - except for the pinning problem (only with future/other Go implementations). It can be worked around, but at the cost of slowing down the "safe" mode even more.

Don't get me wrong, I'm not really pushing for the pinning feature. I am asking for it, but please read it as a request for it to be considered and if seen fit for future implementation, then its specification should be, IMO, part of the rules @rsc asked to be defined in this issue.

While I do agree that the implementation might be complex, there exist GC languages which support pinning, proving it's feasible.

@dr2chase
I have no access to the Owicki's paper. Maybe her technique is the same as discussed elsewhere?

... if you write code that (you think) avoids a heap allocation for something you pass to C ...

What I am trying to do does not use cgo/does not pass things to C.

[dr2chase]

@cznic Owicki's paper discusses how you interoperate traced and untraced stuff in a language without dropping down into Unsafe. It influenced the design of Modula-2+, which influenced the design of Modula-3, then along came Java, and we forgot all that stuff because who needs anything better than JNI? The rules follow from the needs of the garbage collector and tracing -- traced can contain pointers to untraced, but not vice-versa, because the GC (in most designs) does not include untraced memory in its root set -- untraced memory is treated as a likely source of bugs and is not to be relied on (and decades of experience confirms this), and this is especially true in the usual use case for untraced memory, which is "interfacing with some other language that doesn't enforce type safety or array bounds". It's a perfectly reasonable model for Cgo.

I can't quite figure out how you end up needing pinning without using unsafe, if that's your eventual goal. I suspect you'll find yourself wanting to use unsafe to convert pointer-to-foo into a singleton slice of foo, also, but that doesn't create a need for pinning.

[rsc]

Duplicate of #12416