Golang Shared library behavior in forked processes

[praveenkmurthy]

1. What version of Go are you using (go version)?
   go1.7 linux/amd64
2. What operating system and processor architecture are you using (go env)?
   Ubuntu 16.04 LTS X86-64
3. What did you do?
   If possible, provide a recipe for reproducing the error.
   A complete runnable program is good.
   A link on play.golang.org is best.

Issue #15538 establishes that a process which has loaded a Go shared library (and therefore, instantiated the Go runtime) cannot call fork() and make use of Go code in child processes.

Issue #15556 establishes that if library is loaded after fork(), everything works fine.

Also I understand there is no support for dlclose() for the go shared library.

My use case is w.r.t., using a go shared library as a module in apache web server.
I have written a PKCS#11 shared library using Golang. This library is loaded when apache httpd loads mod_nss.

Following is the apache httpd startup flow ( as observed from the stack trace )

• Apache httpd main process loads the mod_nss module which in turn loads the PKCS#11 Golang library.
• Loads and validates the certificates for the server
• Forks a child process to serve the http request
• Child process again loads the mod_nss which in turn again loads the PKCS#11 Golang shared library.

https://play.golang.org/p/_Eaqnne7vt

The code is specific to PKCS#11 library. This is reproducible with any Go shared library which uses net.http package

1. What did you expect to see?

Since the child process is loading the shared library again, I expected the golang shared library runtime to be in a consistent state.

1. What did you see instead?

Golang shared library hangs in net.http package waiting for futex

GDB Output:
#1 0x00007f32795fb263 in runtime.futexsleep (addr=0xc820022108, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:40
#2 0x00007f32795e0ff8 in runtime.notesleep (n=0xc820022108) at /usr/local/go/src/runtime/lock_futex.go:145
#3 0x00007f32796041a2 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:1685
#4 0x00007f32796052c2 in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2033
#5 0x00007f3279605741 in runtime.park_m (gp=0xc820000600) at /usr/local/go/src/runtime/proc.go:2140
#6 0x00007f327962c9b3 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:233
#7 0x00007fffd5e52918 in ?? ()
#8 0x00007f327962e49a in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:629
#9 0x00007f32795bf494 in _cgoexp_6ce49e0b1287_G_GenerateRandom (a=0x7fffd5e529d0, n=32) from /home/osboxes/goWorkspace/libcryptoki_idps.so

[bradfitz]

• Apache httpd main process loads the mod_nss module which in turn loads the PKCS#11 Golang library.
• ....
• Forks a child process to serve the http request

If you're loading Go code before a fork, all bets are off, no?

/cc @ianlancetaylor

[minux]

But the process already forked (and leaves Go runtime is an inconsistent state) and I think dlopen will not reload a DSO if it has already been loaded (even if by the parent process before fork)

[praveenkmurthy]

As I don't have control over the apache code, I was wondering if there is any way out in this scenario.
Doesn't reloading the library in child process help the cause ?

[ianlancetaylor]

There is no such thing as "reloading a shared library". If your process has already loaded the library, then loading it again (calling dlopen again) only increments a reference count, nothing else. This applies whether the initial dlopen was done by the current process or by some parent process.

I do not know of any way to make this work using shared libraries.

[praveenkmurthy]

Yeah, I guess if dlclose() was supported, it would have worked. I could have used the pthread_atfork handlers to do dlclose() and load it again in the child process.
Is there anything planned to support dlclose() for the Golang shared libraries ?

[ianlancetaylor]

There are no current plans to make dlclose work. It's hard because it requires finding all the threads and shutting them down.

I'm going to close this as infeasible.

[minux]

On a side note: should Go shared library register an atfork handler that terminate the whole program with a nice error message rather that panic later?

[ianlancetaylor]

A fork shortly followed by an execve is fine.