CLI healtheck only listening on localhost, not reachable for Kubernetes

[Timmitry]

A cli health check was introduced in #452 after discussion in #403. The readme describes how to add it to Kubernetes. We did exactly that, but the health check does not work.

We can enter the kubernetes-worker-pod and perform curl localhost:7001, which works just fine. However, a curl <ip>:7001 does not work, with a connection refused.

Unfortunately, Kubernetes will try to ping the pod via its ip, and is therefore not successful, which prevents the containers from starting, as the health checks are never successful.

Do people use this health check with Kubernetes, and if so, how did they solve this problem? 😆
If not, how could we allow the binding on all interfaces? We tracked this down to this code:

good_job/lib/good_job/probe_server.rb

Lines 17 to 24 in 9b61ea6

	def start
	@handler = Rack::Handler.get(RACK_SERVER)
	@future = Concurrent::Future.new(args: [@handler, @port, GoodJob.logger]) do |thr_handler, thr_port, thr_logger|
	thr_handler.run(self, Port: thr_port, Logger: thr_logger, AccessLog: [])
	end
	@future.add_observer(self.class, :task_observer)
	@future.execute
	end

Where we wildly guessed that providing BindAddress: "0.0.0.0" in line 20, which will then be passed on to Webrick here:

https://github.com/ruby/webrick/blob/968bc8c4392c0ff8fa8147ca50bc47c74a811886/lib/webrick/server.rb#L111

However, I do not really understand Rack and Webrick, so that is just a first guess, and before experimenting with that (which always involves a deployment...), I first wanted to raise this as an issue here!

Let me know if you need any more information, and best regards 🙂

[bensheldon]

@Timmitry thank you for digging into this! And I'm bummed it didn't work out of the box. But we can fix it.

I think binding on 0.0.0.0 is right. I took a look at sidekiq-alive and that's what it binds on, though they pass it as a Host: kwarg.

Would you be up for trying it empirically and reporting back? Otherwise I can try to set up a local kube and give it a try.

[Timmitry]

Hey @bensheldon, thanks for answering!

I just tested this locally with a Rails project that uses GoodJob.

First I started good job with GOOD_JOB_PROBE_PORT=7001 bundle exec good_job. I was then able to curl localhost:7001, but not curl <local_ip_address>:7001. This is exactly the problem that we have on Kubernetes, that it only binds on localhost, but not on all interfaces.

I then opened the Gem with bundle open good_job and changed this line

good_job/lib/good_job/probe_server.rb

Line 20 in 9b61ea6

thr_handler.run(self, Port: thr_port, Logger: thr_logger, AccessLog: [])

to include the Host as you suggested with:

thr_handler.run(self, Port: thr_port, Host: "0.0.0.0", Logger: thr_logger, AccessLog: [])

Restarting good_job as before, I was now able to curl <local_ip_address>:7001 🎉

If you want, I can make a PR for this change. I made a PR for this change. Do you feel like we need an ENV-Variable GOOD_JOB_PROBE_HOST for this setting?