Enable system tests for auditbeat #5945

ruflin · 2017-12-21T22:56:30Z

There were system tests and system integration tests for auditbeat but so far they were not run.
Integration tests enabled
Module config changed to module that works on more systems. Limitation for linux removed.

This is prerequisite for #5938

ruflin · 2017-12-21T23:30:36Z

jenkins, test it

andrewkroh

LGTM

andrewkroh · 2017-12-22T00:03:33Z

jenkins, test it

Seems like Jenkins never started for this one.

ruflin · 2017-12-22T01:19:20Z

Seems like it refuses to start even with our comments :-(

ph · 2017-12-22T18:19:24Z

@ruflin Yes, there is an issue with github hook infra and github are working on it.

ph · 2017-12-22T18:19:41Z

Jenkins test this please

ruflin · 2018-02-06T02:51:44Z

Rebased this on master to see if it builds correctly.

andrewkroh · 2018-02-06T17:50:47Z

auditbeat/Makefile

 GOX_OS?=linux windows ## @Building List of all OS to be supported by "make crosscompile".
 DEV_OS?=linux
+TESTING_ENVIRONMENT?=snapshot-noxpack
+GOPACKAGES=$(shell go list ${BEAT_PATH}/... | grep -v /vendor/)


Do you know what requires this line? I'd assume this is the default in libbeat/scripts/Makefile.

I just copied over from Metricbeat. It does not seem to be needed anymore. I also removed it from Metricbeat.

andrewkroh · 2018-02-06T17:51:40Z

auditbeat/docker-compose.yml

+
+  elasticsearch:
+    extends:
+      file: ../testing/environments/${TESTING_ENVIRONMENT}.yml


If you follow this file you will find that it references Dockerfile-snapshot which no longer exists.

beats/testing/environments/snapshot-noxpack.yml

Lines 10 to 11 in f3de72e

context: ./docker/elasticsearch

dockerfile: Dockerfile-snapshot

Strange, metricbeat depends on the same one and still works. Change.

@andrewkroh I think I just found a potential issue. Metricbeat has it's own elasticsearch instance for the testing of the module. This is reused also for the tests where data is sent to ES like the template loading.

It could now be argued that this a good thing as template loading etc. is tested with a stable version and it's up to libbeat to test its own functionality against master as each beat just uses the provided functionality. At the same time this was not intended.

The above explains how the noxpack file could have become outdated without breaking the tests.

ruflin · 2018-02-27T05:17:22Z

It seems the test on Jenkins is failing because the mac slave tries to start the environment which it shouldn't because TEST_ENVIRONMENT=0 is set. I need to further investigate.

This PR is a subset of elastic#5945 to remove the Metricbeat changes from the PR.

andrewkroh · 2018-03-08T00:13:31Z

auditbeat/Makefile

-SYSTEM_TESTS=false
-TEST_ENVIRONMENT=false
+SYSTEM_TESTS=true
+TEST_ENVIRONMENT=true


Would setting TEST_ENVIRONMENT?=true solve the issue? IIRC we set TEST_ENVIRONMENT=0 in the Jenkins darwin environment.

good catch, will try

* Cleanup Metricbeat composer and Makefile This PR is a subset of #5945 to remove the Metricbeat changes from the PR.

* There were system tests and system integration tests for auditbeat but so far they were not run. * Integration tests enabled * Module config changed to module that works on more systems. Limitation for linux removed. Change noxpack path readd es_beats path remove kibana snapshot file one more check for xpack0 to see if timeout becomes less of a problem readd go packages unify metricbeat / auditbeat to find problem update travis, rename beat-path as double used remove kibana because not needed modify build flag to run as unit test add file tree test to unit tests add ? for test env

ruflin · 2018-03-12T17:01:19Z

auditbeat/docker-compose.yml

+  proxy_dep:
+    image: busybox
+    depends_on:
+      elasticsearch: { condition: service_healthy }


@exekias In compose version 2 depends_on has to be an array and does not support the health check: https://docs.docker.com/compose/compose-file/compose-file-v2/#depends_on 2.1 is required for the above notation. But 2.1 is not supported by the module script. As there is no compose usage yet in auditbeat I will check if I can get away with using 2.1 here.

If you want to use the uncoupled testing code you need to switch to something like this: https://github.com/elastic/beats/blob/master/metricbeat/docker-compose.yml

Where there is no proxy_dep and health checks are managed from testing framework

It seems to work with 2.1 for now as there are no usage of the testing framework.

ruflin · 2018-03-13T06:33:06Z

The test failure from autodiscovery should not be related, still posting it here for @exekias

github.com/elastic/beats/libbeat/version
17:14:47 ?   	github.com/elastic/beats/libbeat/api	[no test files]
17:14:50 command [go test -race -cover -coverprofile /tmp/gotestcover-568559804 github.com/elastic/beats/libbeat/autodiscover]: exit status 1
17:14:50 --- FAIL: TestAutodiscover (0.08s)
17:14:50 	<autogenerated>:1: 
17:14:50                           
17:14:50 	Error Trace:	autodiscover_test.go:183
17:14:50 		
17:14:50 	Error:      	Not equal: 
17:14:50 		
17:14:50 	            	expected: 1
17:14:50 		
17:14:50 	            	actual  : 2
17:14:50 		
17:14:50 	Test:       	TestAutodiscover
17:14:50 	<autogenerated>:1: 
17:14:50                           
17:14:50 	Error Trace:	autodiscover_test.go:184
17:14:50 		
17:14:50 	Error:      	Should be true
17:14:50 		
17:14:50 	Test:       	TestAutodiscover
17:14:50 panic: runtime error: index out of range [recovered]
17:14:50 	panic: runtime error: index out of range
17:14:50 
17:14:50 goroutine 3 [running]:
17:14:50 testing.tRunner.func1(0xc420134000)
17:14:50 	/var/lib/jenkins/.gvm/versions/go1.9.4.linux.amd64/src/testing/testing.go:711 +0x5d9
17:14:50 panic(0x90b4a0, 0xbdebf0)
17:14:50 	/var/lib/jenkins/.gvm/versions/go1.9.4.linux.amd64/src/runtime/panic.go:491 +0x2a2
17:14:50 github.com/elastic/beats/libbeat/autodiscover.TestAutodiscover(0xc420134000)
17:14:50 	/var/lib/jenkins/workspace/elastic+beats+pull-request+multijob-linux/beat/libbeat/label/ubuntu/src/github.com/elastic/beats/libbeat/autodiscover/autodiscover_test.go:185 +0x1e76
17:14:50 testing.tRunner(0xc420134000, 0x986d68)
17:14:50 	/var/lib/jenkins/.gvm/versions/go1.9.4.linux.amd64/src/testing/testing.go:746 +0x16d
17:14:50 created by testing.(*T).Run
17:14:50 	/var/lib/jenkins/.gvm/versions/go1.9.4.linux.amd64/src/testing/testing.go:789 +0x569
17:14:50 FAIL	github.com/elastic/beats/libbeat/autodiscover	0.457s
17:14:51 ok  	github.com/elastic/beats/libbeat/autodiscover/builder	1.052s	coverage: 32.6% of statements

@andrewkroh Ready for an other review / merge.

andrewkroh · 2018-03-13T12:05:55Z

Thanks for adding this test!

ruflin added Auditbeat review labels Dec 21, 2017

ruflin requested a review from andrewkroh December 21, 2017 22:56

ruflin mentioned this pull request Dec 21, 2017

Add dashboard test loading for auditbeat #5938

Merged

andrewkroh approved these changes Dec 21, 2017

View reviewed changes

ruflin force-pushed the enable-auditbeat-system-tests branch from 8f3b496 to a78b6a4 Compare December 27, 2017 02:47

ruflin force-pushed the enable-auditbeat-system-tests branch 2 times, most recently from 6aed4aa to f5ed930 Compare January 4, 2018 23:03

ruflin force-pushed the enable-auditbeat-system-tests branch from f5ed930 to b79d60d Compare February 6, 2018 02:51

andrewkroh reviewed Feb 6, 2018

View reviewed changes

ruflin force-pushed the enable-auditbeat-system-tests branch from b79d60d to 4836590 Compare February 12, 2018 00:56

ruflin mentioned this pull request Feb 18, 2018

Fix #6382. In snapshot-noxpack.yml: set XPACK to 0 in the kibana image #6383

Merged

ruflin force-pushed the enable-auditbeat-system-tests branch from a21591d to ab8e48d Compare February 26, 2018 08:41

ruflin added a commit to ruflin/beats that referenced this pull request Feb 27, 2018

Cleanup Metricbeat composer and Makefile

0ecbe84

This PR is a subset of elastic#5945 to remove the Metricbeat changes from the PR.

ruflin mentioned this pull request Feb 27, 2018

Cleanup Metricbeat composer and Makefile #6481

Merged

andrewkroh reviewed Mar 8, 2018

View reviewed changes

ruflin force-pushed the enable-auditbeat-system-tests branch from ae26160 to 3e9299b Compare March 11, 2018 13:36

exekias pushed a commit that referenced this pull request Mar 12, 2018

Cleanup Metricbeat composer and Makefile (#6481)

6bec4b1

* Cleanup Metricbeat composer and Makefile This PR is a subset of #5945 to remove the Metricbeat changes from the PR.

ruflin added 2 commits March 12, 2018 11:07

go to version 2

6e11a0c

ruflin force-pushed the enable-auditbeat-system-tests branch from 3e9299b to 6e11a0c Compare March 12, 2018 16:10

ruflin commented Mar 12, 2018

View reviewed changes

switch to 2.1

e620a2c

andrewkroh merged commit 3bb5cb6 into elastic:master Mar 13, 2018

ruflin deleted the enable-auditbeat-system-tests branch March 13, 2018 12:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable system tests for auditbeat #5945

Enable system tests for auditbeat #5945

ruflin commented Dec 21, 2017

ruflin commented Dec 21, 2017

andrewkroh left a comment

andrewkroh commented Dec 22, 2017

ruflin commented Dec 22, 2017

ph commented Dec 22, 2017

ph commented Dec 22, 2017

ruflin commented Feb 6, 2018

andrewkroh Feb 6, 2018

ruflin Feb 12, 2018

andrewkroh Feb 6, 2018

ruflin Feb 12, 2018

ruflin Feb 12, 2018

ruflin commented Feb 27, 2018

andrewkroh Mar 8, 2018

ruflin Mar 11, 2018

ruflin Mar 12, 2018

exekias Mar 12, 2018

ruflin Mar 13, 2018

ruflin commented Mar 13, 2018

andrewkroh commented Mar 13, 2018

	context: ./docker/elasticsearch
	dockerfile: Dockerfile-snapshot

Enable system tests for auditbeat #5945

Enable system tests for auditbeat #5945

Conversation

ruflin commented Dec 21, 2017

ruflin commented Dec 21, 2017

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh commented Dec 22, 2017

ruflin commented Dec 22, 2017

ph commented Dec 22, 2017

ph commented Dec 22, 2017

ruflin commented Feb 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Feb 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Mar 13, 2018

andrewkroh commented Mar 13, 2018