Cherry-pick #11265 to 7.0: Log Kibana GET /api/status response on test failure

[ycombinator]

Cherry-pick of PR #11265 to 7.0 branch. Original message:

To help debug #11261.

[ycombinator]

Interestingly enough, this PR failed in Travis CI and printed out the Kibana GET /api/status response (which is exactly what this PR is trying to do):

command [go test -tags integration -cover -coverprofile /tmp/gotestcover-806603798 github.com/elastic/beats/metricbeat/module/kibana/stats]: exit status 1
Building elasticsearch
Step 1/2 : FROM docker.elastic.co/elasticsearch/elasticsearch:6.6.0
 ---> 13aa43015aa1
Step 2/2 : HEALTHCHECK --interval=1s --retries=300 CMD curl -f http://localhost:9200/_xpack/license
 ---> Using cache
 ---> 2dc711e5f1ed
Successfully built 2dc711e5f1ed
Successfully tagged metricbeat5dc5a3d869897b23131599c7099908d63a856c7e_elasticsearch:latest
Creating metricbeat5dc5a3d869897b23131599c7099908d63a856c7e_elasticsearch_1 ... 
Step 1/2 : FROM docker.elastic.co/kibana/kibana:6.6.0
 ---> dfc685453eaa
Step 2/2 : HEALTHCHECK --interval=1s --retries=300 CMD curl -f http://localhost:5601/api/status | grep '"disconnects"'
 ---> Using cache
 ---> 543752fb78b4
Successfully built 543752fb78b4
Successfully tagged metricbeat5dc5a3d869897b23131599c7099908d63a856c7e_kibana:latest
Creating metricbeat5dc5a3d869897b23131599c7099908d63a856c7e_kibana_1 ... 
	stats_integration_test.go:60: 
			Error Trace:	stats_integration_test.go:60
			Error:      	Should be empty, but was [HTTP error 500 in stats: 500 Internal Server Error]
			Test:       	TestFetch
	stats_integration_test.go:61: 
			Error Trace:	stats_integration_test.go:61
			Error:      	Should NOT be empty, but was []
			Test:       	TestFetch
--- FAIL: TestData (31.03s)
	stats_integration_test.go:115: Kibana GET /api/status response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
	stats_integration_test.go:76: getting kibana version key not found
FAIL
coverage: 13.3% of statements
FAIL	github.com/elastic/beats/metricbeat/module/kibana/stats	195.367s

[ruflin]

Does this mean we probably need to improve the health check?

[ycombinator]

Does this mean we probably need to improve the health check?

I'm not sure. The health check is already calling the same API that is failing in CI. Here's what the health check looks like:

beats/metricbeat/module/kibana/_meta/Dockerfile

Line 2 in 4920cd6

HEALTHCHECK --interval=1s --retries=300 CMD curl -f http://localhost:5601/api/status | grep '"disconnects"'

Also, I'm stumped that the failure is a 401. AFAIK, Security isn't even enabled in Elasticsearch for these tests!

[ruflin]

We have a different check here: https://github.com/elastic/beats/blob/master/testing/environments/snapshot.yml#L31

[ycombinator]

It's probably a good idea to make all healthchecks for a service consistent across the various Dockerfiles and docker-compose files we have across beats. I'll make a separate PR (to master, since this PR here is a backport PR) to make this happen for the stack modules at least.

Two things, however:

1. You mentioned https://github.com/elastic/beats/blob/master/testing/environments/snapshot.yml. I thought this file was only being used by the python system tests, not the golang integration tests (which is what are failing here). My understanding was that the golang integration tests called compose.EnsureUp which ends up recursively looking for a docker-compose.yml file. If I'm following the code correctly, it will end up finding https://github.com/elastic/beats/blob/7.0/metricbeat/docker-compose.yml, which for the Kibana container, references https://github.com/elastic/beats/tree/7.0/metricbeat/module/kibana/_meta.

   So, in the case of the CI failure that happened for this PR earlier, could you explain to me where https://github.com/elastic/beats/blob/master/testing/environments/snapshot.yml comes into play?

2. I'm still very confused why Kibana returned a 401. I'm going to make a debugging PR to master that will log Docker logs for integration tests and keep running that PR until we see this 401 error again.

[ycombinator]

Let's continue the conversation here but I'm going to merge this backport PR now as it's green and it does what it set out to do, which is to log the Kibana API response in case of an error.