Daemonset services are listed after the nodes is decommissioned from cluster.

[tejnar]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Daemonset services are listed in consul UI and api after the node is decommissioned from cluster.

Reproduction Steps

1. Install consul server without the agents.
2. Deploy Daemonset(Fluent-bit or node-exporter) to cluster.
3. Delete any node in cluster.

Actual Behaviour: Consul UI is displaying the deleted daemonset service. API is also listing the services which aren't part of the cluster.
API command: curl -i http://127.0.0.1:8500/v1/catalog/service/fluent-bit

Attached the values.yaml as values.txt file.
values.txt

/ $ consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = dae670fe
version = 1.14.4
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 100.64.54.123:8300
server = true
raft:
applied_index = 69775
commit_index = 69775
fsm_pending = 0
last_contact = 21.178669ms
last_log_index = 69775
last_log_term = 8
last_snapshot_index = 65546
last_snapshot_term = 8
latest_configuration = [{Suffrage:Voter ID:72b147fe-690f-0c42-f07e-d42709b9c255 Address:100.64.54.123:8300} {Suffrage:Voter ID:fe9ca4ff-3608-45ee-fc25-d4774c1e6f9b Address:100.64.28.115:8300} {Suffrage:Voter ID:5a56a260-d45e-6748-2c13-fb17877d273b Address:100.64.40.151:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 8
runtime:
arch = amd64
cpu_count = 2
goroutines = 804
max_procs = 2
os = linux
version = go1.19.4
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 3
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 12
members = 3
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 6
members = 3
query_queue = 0
query_time = 1

EKS: Kubernetes version: 1.21
Amazon VPC CNI
version: 1.0.3
appVersion: 1.14.4

[mr-miles]

Sounds like the same issue as hashicorp/consul#15908 - deleting nodes leaves orphans in the service catalog as the event watcher seems not to receive the events that it is expecting

[david-yu]

The following PR is now merged which may address your issues: #2571. This should be released in consul-k8s 1.2.x, 1.1.x, and 1.0.x by mid August timeframe for our next set of patch releases.