Script checks can overlap

[nh2]

To my surprise Consul runs script checks in an overlapping fashion instead of serially.

This can lead to lots of processes and memory leakage if the script check is short or the timeout is long.

It also means that an older script check can overwrite the result written by a younger script check.

For example, assume you put date into the script check. Then, it can theoretically happen that the date output captured by the script and show time going backwards (for example, if the first date was CPU-scheduled unluckily).

I would have assumed that consul runs no more than 1 instance of a script check at any given time.

At least not unless some special flag is set.

consul version for both Client and Server

Client: irrelevant
Server: Consul v0.9.3

consul info for both Client and Server

Client: irrelevant

Server:

# consul info
agent:
	check_monitors = 1
	check_ttls = 0
	checks = 1
	services = 1
build:
	prerelease = 
	revision = 
	version = 0.9.3
consul:
	bootstrap = false
	known_datacenters = 1
	leader = true
	leader_addr = 10.0.0.2:8300
	server = true
raft:
	applied_index = 95568
	commit_index = 95568
	fsm_pending = 0
	last_contact = 0
	last_log_index = 95568
	last_log_term = 495
	last_snapshot_index = 90278
	last_snapshot_term = 479
	latest_configuration = [{Suffrage:Voter ID:10.0.0.3:8300 Address:10.0.0.3:8300} {Suffrage:Voter ID:10.0.0.1:8300 Address:10.0.0.1:8300} {Suffrage:Voter ID:10.0.0.2:8300 Address:10.0.0.2:8300}]
	latest_configuration_index = 1
	num_peers = 2
	protocol_version = 2
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Leader
	term = 495
runtime:
	arch = amd64
	cpu_count = 1
	goroutines = 231
	max_procs = 1
	os = linux
	version = go1.9
serf_lan:
	coordinate_resets = 0
	encrypted = false
	event_queue = 0
	event_time = 165
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 468
	members = 5
	query_queue = 0
	query_time = 1
serf_wan:
	coordinate_resets = 0
	encrypted = false
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 231
	members = 3
	query_queue = 0
	query_time = 1

Operating system and Environment details

NixOS

Description of the Issue (and unexpected/desired result)

Consul runs script checks in an overlapping fashion instead of serially.

Reproduction steps

Define a check with interval 1s, make it sleep 10 seconds. Observe processes in ps or htop.

[slackpad]

Hi @nh2 that's not the intended behavior. Will take a look.

[nh2]

For people who read this, here's a question whether this entire probelm is indeed already solved: #3567 (comment)