os/exec: Start sometimes creates processes with signals blocked with go1.5 on OSX

[gabrielrussell]

I can recreate this by running a test of our application in a loop. Eventually a process is started that has many of its signals blocked. This ends up causing a error because we're running a process that expects to be able to receive signals.

My attempts to create a smaller program that reproduces this has only failed. I would assume that some bit of the complexity of the application is triggering the bug.

I'll be happy to provide any information that I can.

Gabriel Russell

[ianlancetaylor]

What does the loop that is running the program look like? Is this a loop written in Go or in shell or something else?

How are you testing whether signals are blocked?

[gabrielrussell]

It's a shell script that runs a "go test ..." in a loop. That said, we don't only see this in testing, we also see it on our production systems.

Its a c++ application doing daemonizing double fork where the grandparent doesn't exit until the grandchild sends a signal to the grandparent indicating that it's ready to do work.

We've tested that we can fix the problem by adding code to our c++ application to unblock signals before doing anything else, but this is not a whole solution for us because the c++ application is already widely distributed.

[ianlancetaylor]

I'm sorry, I don't grasp the details here, and the details matter. I see now that your original issue title refers to exec.Start, implying that you are using the os/exec package. You say that you have a shell script running "go test" in a loop.

Is that Go program using os/exec?

How exactly are you starting the program whose signals are blocked?

What does the C++ daemonizing application have to do with this? Is that the test, or is that the production system?

Which signal(s) exactly are you using?

Do you have a small reproduction case that you could share?

Thanks.

[gabrielrussell]

We have an application written in go who's main responsibility is to run
another program that's written in c++.

The c++ program needs to to be executed without signals blocked.

Sometimes the go program executes the c++ program with it's signals blocked.

We have tests for our go program, and some of them execute the c++ program.

If we execute one of those tests over and over, via a bash while loop, we
eventually see the c++ program get executed with signals blocked.

We need to send and receive SIGUSR2

Our attempts to create a small reproduction case have failed.

Thanks.

On Thu, Nov 5, 2015 at 5:29 PM, Ian Lance Taylor notifications@github.com
wrote:

I'm sorry, I don't grasp the details here, and the details matter. I see
now that your original issue title refers to exec.Start, implying that you
are using the os/exec package. You say that you have a shell script running
"go test" in a loop.

Is that Go program using os/exec?

How exactly are you starting the program whose signals are blocked?

What does the C++ daemonizing application have to do with this? Is that
the test, or is that the production system?

Which signal(s) exactly are you using?

Do you have a small reproduction case that you could share?

Thanks.

—
Reply to this email directly or view it on GitHub
#13164 (comment).

[ianlancetaylor]

Thanks for the details. Does your Go program, the one that executes the C++ program, use cgo?

[ianlancetaylor]

Also, you said this was Darwin, but what is your GOARCH value?

[gabrielrussell]

Yes, there is some cgo.

GOARCH="amd64"

[ianlancetaylor]

I assume your cgo code never calls sigaction or sigprocmask.

Sorry, I realized that may have misunderstood something. When you say that the signals are blocked, what precisely do you mean? Do you mean that the signal is in the blocked signal mask as set by sigprocmask or pthread_sigmask? Or do you mean that the signal is ignored because sigaction was used to set the signal handler to SIG_IGN?

[gabrielrussell]

I assume that they're blocked, ala sigprocmask as opposed SIG_IGN'd.

The piece of code that when inserted into the c++ code masks the problem is:

sigset_t unblockSignalMask;
sigemptyset(&unblockSignalMask);
sigprocmask(SIG_SETMASK, &unblockSignalMask, NULL);

And, I don't think that unblocking a signal should unignore it, but I may be wrong.

Also, we disabled the cgo function calls and re-ran our tests without any change in outcome.

[ianlancetaylor]

I agree that they must be blocked if calling sigprocmask fixes it.

I asked about cgo because a program that uses cgo uses a different code path when creating a new thread. When creating a new thread it is necessary to temporarily block all signals, so that the new thread does not receive a signal before it has registered an alternate signal stack. However, the code uses pthread_sigmask, which only affects the signal mask for the calling thread. My assumption is that execve uses the signal mask of the thread that calls execve, or the process-wide signal mask. Anyhow, that is probably not the issue.

Does your program use the os/signal package at all?

[sielaq]

I'm having similar problem: Linux/amd64, exec.Run() ( or Start() )
I have created 2 app to recreate this:
Go:
https://gist.github.com/sielaq/5120a4f06cefc2138ed9
C:
https://gist.github.com/sielaq/9867be2d1b65ffec658c

Go app starts C app (that kills others C app currently running).
When app compiled with go 1.4 -> all C app receive SIGTERM from each other
When app compiled with go 1.5 -> sometimes C app cannot trap SIGTERM

to make it easier to reproduce it:
bash -c "while true; do ./app; done"
try to break it ^C multiple times -> as an effect you should see that C apps start to cumulate in process list.

[ianlancetaylor]

Thanks for the sample code. I suspect this is an unexpected consequence of https://golang.org/cl/10173.