Non-deterministic restart order of daemons

[ngrash]

Hi @cortesi, thanks for modd and devd. Both are fantastic productivity tools in my opinion.

I am having trouble with a workflow in modd and I'm not totally sure that it's not a user error.

Summary

It seems the restart order of daemons is currently non-deterministic. They do receive their signals in order but if a signal leads to a restart this might lead to a race condition.

Explanation

I have a web application that serves it's own assets and I want to use devd to enable live reloading during development.

Consider the following modd.conf:

**/*.go !**/*_test.go {    
    prep: go build ./cmd/myapp
    daemon: ./myapp -listen :8080
    daemon: devd -w 'assets/**/*' -w 'templates/**/*' http://localhost:8080    
}

myapp renders templates and serves assets. Devd is only used for live-reloading which works fine for assets and templates.

But here is what happens when I change a go file:

18:50:09: prep: go build
>> done (554.741561ms)
18:50:10: daemon: ./myapp -listen :8008
>> sending signal hangup
18:50:10: daemon: devd -w 'assets/**/*' -w 'templates/**/*' http://localhost:8080
>> sending signal hangup
18:50:10: Received signal - reloading
18:50:10: daemon: ./myapp -listen :8080
exited: signal: hangup
18:50:10: daemon: devd -w 'assets/**/*' -w 'templates/**/*' http://localhost:8080
18:50:10: GET /
	reverse proxy error: dial tcp [::1]:8080: connect: connection refused
	<- 500 Internal Server Error 
18:50:10: daemon: ./myapp -listen :8080
>> starting...

As you can see, devd triggers a reload before my application daemon restarts. This leads to my browser showing a blank page until I manually refresh.

Looking at the code in daemon.go I think I figured out what is happening.

1. Daemons are kept alive by a loop in func (d *Daemon) Run() that keeps restarting the daemon unless it was stopped.
2. DaemonsPens are restarted by looping over the daemons and restarting each.
3. Daemons are restarted by sending a signal.

The race condition is between the restart loop of the DaemonPen and the run loop of the Daemon. If the run loop restarts my daemon before the restart loop of the DaemonPen restarts the next daemon, everything is fine. Which never happened for me btw.

Towards a solution

The problem would vanish if modd waited for each daemon to process the signal and/or restart before proceeding with the next daemon. However I am aware that signals are asynchronous, so I think a solution will not be that simple.

I'm willing to contribute code but wanted to hear what you think before starting to work on a solution.

[cortesi]

Hi there. Thanks for the complete problem description.

It would be great to have a deterministic order here, but there's a problem: there's no consistent way to tell when a daemon has completely responded to a signal. This is super-dependent on the specific behaviour of the daemon in question.

In your particular case, what might work would be to trigger the devd reload when the persistent assets are changed by myapp, rather than when the Go source files change.

I'll keep this issue open while I think about this.

[SirMetathyst]

what's wrong with a wait command?

**/*.go !**/*_test.go {    
    prep: go build ./cmd/myapp
    daemon: ./myapp -listen :8080
    wait: 2s
    daemon: devd -w 'assets/**/*' -w 'templates/**/*' http://localhost:8080    
}

[SirMetathyst]

Yep. Just tested it with mine, works great:

**/*.go www/*.css www/*.html {
    prep: go build -o ./cmd/inkpad/inkpad ./cmd/inkpad/main.go
    daemon +sigterm: ./cmd/inkpad/inkpad
}

www/*.css www/*.html cmd/inkpad/inkpad {
    prep: make generate
    prep: sleep 2
    daemon: devd --modd --livewatch --watch=www/*.html --watch=www/static/css/style.css http://localhost:8080
}