consul watch handler runs on consul reload / restart

[giladsh1]

Overview of the Issue

consul watch handler runs on reload or when restarting the service.
I've setup a simple watch on a key -

{
  "watches": [
    {
      "type": "key",
      "key": "devops/salt-config/gilad",
      "handler_type": "script",
      "args": ["bash", "/root/test_handelr.sh"]
    }
  ]
}

The key is not changes, but when reloading the service, consul runs the handler.
Please see logs below.

IMHO this is a major issue, since configuration change is sometimes required, but you don't want to runs unnecessary scripts that could potentially break your environment.

This could be related to another issue - #4179

Consul info for both Client and Server

Client info

agent:
	check_monitors = 3
	check_ttls = 0
	checks = 7
	services = 6
build:
	prerelease =
	revision = 28141971
	version = 1.2.0
consul:
	known_servers = 3
	server = false
runtime:
	arch = amd64
	cpu_count = 2
	goroutines = 115
	max_procs = 2
	os = linux
	version = go1.10.1
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 1186
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 415
	member_time = 22158
	members = 1019
	query_queue = 0
	query_time = 1

Server info

agent:
	check_monitors = 0
	check_ttls = 0
	checks = 4
	services = 0
build:
	prerelease =
	revision = e716d1b5
	version = 1.2.2
consul:
	bootstrap = false
	known_datacenters = 2
	leader = true
	leader_addr = 10.224.9.55:8300
	server = true
raft:
	applied_index = 1097012
	commit_index = 1097012
	fsm_pending = 0
	last_contact = 0
	last_log_index = 1097012
	last_log_term = 8
	last_snapshot_index = 1082924
	last_snapshot_term = 4
	latest_configuration = [{Suffrage:Voter ID:229cbf97-30ba-1e53-e02c-ccb3eeb46f28 Address:10.224.9.55:8300} {Suffrage:Voter ID:94cb10fb-3d55-f62a-3881-4743cdc4f300 Address:10.225.20.171:8300} {Suffrage:Voter ID:1822c202-23e1-b611-c72e-ff117ad70613 Address:10.226.20.21:8300}]
	latest_configuration_index = 1096050
	num_peers = 2
	protocol_version = 3
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Leader
	term = 8
runtime:
	arch = amd64
	cpu_count = 2
	goroutines = 5878
	max_procs = 2
	os = linux
	version = go1.10.1
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 5
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 36
	member_time = 5679
	members = 245
	query_queue = 0
	query_time = 1
serf_wan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 2463
	members = 6
	query_queue = 0
	query_time = 1

Operating system and Environment details

CentOS Linux release 7.4.1708 (Core)

Log Fragments

   2018/08/29 14:18:35 [INFO] agent: Caught signal:  hangup
    2018/08/29 14:18:35 [INFO] agent: Reloading configuration...
WARNING: LAN keyring exists but -encrypt given, using keyring
    2018/08/29 14:18:35 [DEBUG] http: Request PUT /v1/agent/reload (5.320895ms) from=127.0.0.1:38813
    2018/08/29 14:18:35 [DEBUG] agent: Node info in sync
    2018/08/29 14:18:35 [DEBUG] agent: Node info in sync
    2018/08/29 14:18:35 [DEBUG] http: Request GET /v1/kv/devops/salt-config/gilad (863.512µs) from=127.0.0.1:38815
    2018/08/29 14:18:35 [DEBUG] agent: watch handler '[bash /root/test_handelr.sh]' output: 14:18:35.746253021

[pierresouchay]

I wonder if this PR #4554 might not fix this issue

[giladsh1]

@pierresouchay any way to validate this will indeed fix the issue?
It would be awesome if you could fix this bug in the next version of Consul..

[pierresouchay]

@vaLski Did validated something similar with the PR #4554, it might have insights on this

[vaLski]

@pierresouchay I don't think that #4554 can relate in any way to this report.

@giladsh1 IMHO, this is an intended behavior and most probably it will remain like that. All watches and event handlers are executed exactly once during consul reload/restart. From my experience, tools invoked as consul {key[prefix],event} watch handlers, should be idempotent. They should produce the same output and state regardless of the fact how many times they have been executed. When running those during reload/restart the handlers in question receive the current global state of the system and apply it to the local environment.

There was a bug which was fixed back in 1.0.1 - #4179. During consul reload, it was running each watch N times, where N was the number of reloads. The more reloads, the more times it will run each watch handler. However this is now fixed.

[pearkes]

I believe this is a duplicate of #571 so I'm going to close this. If not, please let us know and we can re-open.

@vaLski Thanks for that extra context!