Service discovery data is inconsistant with local Consul agent

[JC000]

We've been pointing fingers as to which component is responsible for being incorrect. I think we have the answer below, which was data taken during one of the inconsistent periods.

[clduser@talaria-prod-00101-184-204-1gv ~]$ curl -s 'http://localhost:8500/v1/health/service/talaria?passing=true&pretty' | grep -e '"Node": {' | wc -l
125
[clduser@talaria-prod-00101-184-204-1gv ~]$ curl -s http://localhost:9390/metrics | grep sd_instance_count
# HELP xmidt_talaria_sd_instance_count The current number of service instances of a given type
# TYPE xmidt_talaria_sd_instance_count gauge
xmidt_talaria_sd_instance_count{service="talaria[prod rum]{passingOnly=true}"} 115
[clduser@talaria-prod-00101-184-204-1gv ~]$

We can no longer blame the local agent, as it evidently has the same data as before the event window.

[kcajmagic]

A PR has been submitted against go-kit to fix this issue.

[kcajmagic]

Code has been merged