Premature timeout in lock acquisition

[boxysean]

consul version for both Client and Server

Client: v0.8.0
Server: v0.8.0

consul info for both Client and Server

Client:

$ consul info
agent:
        check_monitors = 3
        check_ttls = 0
        checks = 3
        services = 3
build:
        prerelease = 
        revision = '402636f
        version = 0.8.0
consul:
        known_servers = 5
        server = false
runtime:
        arch = amd64
        cpu_count = 2
        goroutines = 89
        max_procs = 2
        os = linux
        version = go1.8
serf_lan:
        encrypted = false
        event_queue = 0
        event_time = 2317
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 8
        member_time = 357889
        members = 33
        query_queue = 0
        query_time = 463

Server:

$ consul info
agent:
        check_monitors = 0
        check_ttls = 0
        checks = 0
        services = 1
build:
        prerelease = 
        revision = '402636f
        version = 0.8.0
consul:
        bootstrap = false
        known_datacenters = 1
        leader = false
        leader_addr = <snip>
        server = true
raft:
        applied_index = 14353657
        commit_index = 14353657
        fsm_pending = 0
        last_contact = 7.256981ms
        last_log_index = 14353657
        last_log_term = 83434
        last_snapshot_index = 14353126
        last_snapshot_term = 83434
        latest_configuration = [<snip>]
        latest_configuration_index = 14347722
        num_peers = 4
        protocol_version = 2
        protocol_version_max = 3
        protocol_version_min = 0
        snapshot_version_max = 1
        snapshot_version_min = 0
        state = Follower
        term = 83434
runtime:
        arch = amd64
        cpu_count = 1
        goroutines = 425
        max_procs = 1
        os = linux
        version = go1.8
serf_lan:
        encrypted = false
        event_queue = 0
        event_time = 2317
        failed = 9
        health_score = 0
        intent_queue = 0
        left = 141
        member_time = 357889
        members = 175
        query_queue = 0
        query_time = 463
serf_wan:
        encrypted = false
        event_queue = 0
        event_time = 1
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 8471
        members = 5
        query_queue = 0
        query_time = 1

Operating system and Environment details

Ubuntu 14.04

Description of the Issue (and unexpected/desired result)

I have 10 procs waiting for a lock for 10 minutes, each proc taking 60 seconds. The first few procs will exit as expected, but the rest will exit together around 3-4 minutes in with message Shutdown triggered or timeout during lock acquisition.

Looks to be related to #2399.

Reproduction steps

This does it for me, every time:

for ((i=0;i<10;i++)); do consul lock -verbose -name=test_lock60 -try=600s test_lock60 sleep 60 & done

Log Fragments or Link to gist

$ for ((i=0;i<10;i++)); do consul lock -verbose -name=test_lock60 -try=600s test_lock60 sleep 60 & done        
[1] 3303
[2] 3304
[3] 3305
[4] 3306
[5] 3307
[6] 3308
[7] 3309
[8] 3310
[9] 3311
[10] 3312
$ Setting up lock at path: test_lock60/.lock
Attempting lock acquisition
Setting up lock at path: test_lock60/.lock
Attempting lock acquisition
Setting up lock at path: test_lock60/.lock
Attempting lock acquisition
Setting up lock at path: test_lock60/.lock
Attempting lock acquisition
Setting up lock at path: test_lock60/.lock
Attempting lock acquisition
Setting up lock at path: test_lock60/.lock
Attempting lock acquisition
Setting up lock at path: test_lock60/.lock
Attempting lock acquisition
Setting up lock at path: test_lock60/.lock
Attempting lock acquisition
Setting up lock at path: test_lock60/.lock
Attempting lock acquisition
Setting up lock at path: test_lock60/.lock
Attempting lock acquisition
Starting handler 'sleep 60'
Child terminated, releasing lock
Starting handler 'sleep 60'
Cleanup aborted, lock in use
Child terminated, releasing lock
Cleanup aborted, lock in use
Starting handler 'sleep 60'
Child terminated, releasing lock
Starting handler 'sleep 60'
Cleanup aborted, lock in use
Child terminated, releasing lock
Starting handler 'sleep 60'
Cleanup aborted, lock in use
Shutdown triggered or timeout during lock acquisition
Shutdown triggered or timeout during lock acquisition
Shutdown triggered or timeout during lock acquisition
Shutdown triggered or timeout during lock acquisition
Shutdown triggered or timeout during lock acquisition
$
[1]   Exit 1                  consul lock -verbose -name=test_lock60 -try=600s test_lock60 sleep 60
[2]   Exit 1                  consul lock -verbose -name=test_lock60 -try=600s test_lock60 sleep 60
[3]   Done                    consul lock -verbose -name=test_lock60 -try=600s test_lock60 sleep 60
[4]   Done                    consul lock -verbose -name=test_lock60 -try=600s test_lock60 sleep 60
[5]   Exit 1                  consul lock -verbose -name=test_lock60 -try=600s test_lock60 sleep 60
[6]   Exit 1                  consul lock -verbose -name=test_lock60 -try=600s test_lock60 sleep 60
[7]   Exit 1                  consul lock -verbose -name=test_lock60 -try=600s test_lock60 sleep 60
[8]   Done                    consul lock -verbose -name=test_lock60 -try=600s test_lock60 sleep 60
[9]-  Done                    consul lock -verbose -name=test_lock60 -try=600s test_lock60 sleep 60
$ Child terminated, releasing lock
Cleanup succeeded
[10]+  Done                    consul lock -verbose -name=test_lock60 -try=600s test_lock60 sleep 60