'Service does not exist' should not be fatal

[nicholascapo]

Occasionally we have a server that crashes or is rebooted un-cleanly. On restart consul dies with an error:

==> Starting Consul agent...
==> Error starting agent: ServiceID "foo" does not exist

This requires manual intervention to remove the service from the consul data-dir and restart the services.

This error should not be fatal, and should simply result in the service not being available in the catalog.

An additional note: In this case the services are registered using the HTTP API not the config files.

[cetex]

Same here, it happens often on rebooting the cluster (which i'm doing due to testing)
Rebooted the cluster and got this error.
root@s2:~# consul agent -server $CONSUL_BOOTSTRAP -log-level=debug -bind=:: -advertise=10.255.255.$CONSULID -data-dir $CONSUL_STORAGE -ui-dir /usr/local/share/consul-web/dist/ -retry-join=10.255.255.1 -retry-join=10.255.255.2 -retry-join=10.255.255.3 -retry-join=10.255.255.4 -retry-join=10.255.255.5 -client=::
==> WARNING: It is highly recommended to set GOMAXPROCS higher than 1
==> Starting Consul agent...
==> Error starting agent: ServiceID "aurora" does not exist

Current status: Can't start any consul agent, datacenter is currently down. consul data-dir most likely needs to be erased and kv/store needs to be recreated from scratch..

[armon]

Thanks for the report, tagging as a bug!

[cetex]

Great. How tricky is this to fix?
It's currently one of (very few things) holding us back from moving forward with deploy on consul of the rest of the DC.

[ryanuber]

I don't think this one will be too hard to fix. I'll take a look at it soon, although you'd have to build from master to get the fix since we have no release dates planned yet.

[cetex]

sure. i can build it and distribute it from my own repo as long as there's something that works. :)

[ryanuber]

Fixed by 04a2fae. We now warn and purge the erroneous check. Thanks!

[nicholascapo]

Sweet, thanks Ryan

On Wed, Mar 11, 2015, 18:27 Ryan Uber notifications@github.com wrote:

Closed #769 #769.

—
Reply to this email directly or view it on GitHub
#769 (comment).

[cetex]

Awesome, way faster than i thought possible ;)

[tonglil]

Mm, I think this is still happening?

==> Starting Consul agent...
==> Error starting agent: Failed to register check 'CPU Info Check': ServiceID "key-value store" does not exist &{cpu CPU Info Check This check reports the stats of the CPU, and does not check for unhealthy state. key-value store   {/etc/consul.d/monitor cpu   10m0s   0 0  This check reports the stats of the CPU, and does not check for unhealthy state.}}

I defined the service in a check.

{
  "check": {
    "id": "cpu",
    "name": "CPU Info Check",
    "script": "/etc/consul.d/monitor cpu",
    "interval": "10m",
    "service_id": "key-value store",
    "notes": "This check reports the stats of the CPU, and does not check for unhealthy state."
  }
}

Running

Consul v0.6.3
Consul Protocol: 3 (Understands back to: 1)

[ryanuber]

Hi @tonglil,

Is this check inside of a config file? This fix is only to purge health checks which were submitted from the REST API, and whose services no longer exist when Consul is restarted. If the check is defined in a configuration file, you have to remove the check or the config file which contains it.

Hope that helps.

[tonglil]

Ah gotcha, thanks for the clarification @ryanuber.

Do you know what the reasoning is for making it fatal for config files?

[ryanuber]

No problem. I think it was mainly just the most logical course of action when a check is attempted to register against an unknown service. You would otherwise get a running Consul agent without the check or the service you specified, which seems rather unexpected. Let me know if I missed anything!