[WIP/Ideas] Config matrices for integration tests #7957

jsoriano · 2018-08-13T17:46:23Z

Thoughts and experiments for having the possibility of integration tests with more configurations in an easier way, and to continue migrating things from python to go. Not to be merged yet (even if miraculously tests pass 😃 )

Here one metricset of the redis module is tested with three versions and two operating systems. It also makes better use of paralellization. Runners could be easily shared per module.

Some things done here:

Scenario definition (still docker-compose.yml) parametrized and moved to the module level
A helper function to run test suites in multiple scenarios
Some things to ease concurrent testing (Host configuration automatically taken from the scenario, compose project and lock per scenario...)
Implementation of scenario cleanup (docker-compose down -v)

houndci-bot · 2018-08-13T17:46:38Z

libbeat/tests/compose/runner.go

+	return scenarios
+}
+
+func (r *TestRunner) Run(t *testing.T, tests ...func(t *testing.T, host string)) {


exported method TestRunner.Run should have comment or be unexported

houndci-bot · 2018-08-13T17:46:38Z

libbeat/tests/compose/runner.go

+	"testing"
+)
+
+type TestRunner struct {


exported type TestRunner should have comment or be unexported

andrewkroh · 2018-08-13T21:27:42Z

metricbeat/module/redis/keyspace/keyspace_integration_test.go

-	compose.EnsureUp(t, "redis")
-
-	addEntry(t)
+	runner.Run(t, func(t *testing.T, host string) {


Nice idea 👍

I am starting to think in the merge of this little monster and one of my main concerns is that this tests structure modifies the whole integration test files even if only for the indentation level. This is already making the maintenance of the branch a bit painful, as any change on integration tests require a manual merge when updating the branch. And for the same reasons it will be also painful when merged for open PRs and backports.

I am thinking on a second option, in which there is a simple test that calls the runner and the actual tests are defined as private functions. Find in this gist both options for a simple test file and their diffs with master. Option 1 is the one I'm following by now, option 2 would be my new proposal, in option 2 the differences are actually only in the lines that would need to be changed, and there are less indentation levels.

@andrewkroh @ruflin opinions?

any disadvantages for option 2 besides having more functions? +1 on 2 so far.

The more I think about it, the more I am leaning about second option too. One disadvantage can be that one needs to add the test function to the suite for it to be run, but most of integration tests have only a fetch and a data function and is not so frequent to add more tests.
I'll go for option 2.

ruflin · 2018-08-14T07:56:36Z

jenkins, test this

houndci-bot · 2018-08-14T17:27:41Z

libbeat/tests/compose/runner.go

+	"github.com/pkg/errors"
+)
+
+type TestRunner struct {


exported type TestRunner should have comment or be unexported

houndci-bot · 2018-08-14T18:10:39Z

metricbeat/module/redis/mtest/runner.go

+)
+
+var (
+	Runner = compose.TestRunner{


exported var Runner should have comment or be unexported

libbeat/tests/compose/runner.go

houndci-bot · 2018-08-14T18:10:39Z

libbeat/tests/compose/runner.go

+	Timeout  int
+}
+
+type Suite map[string]func(t *testing.T, host string)


exported type Suite should have comment or be unexported

houndci-bot · 2018-08-14T18:10:39Z

libbeat/tests/compose/runner.go

+	rand.Seed(time.Now().UnixNano())
+}
+
+type TestRunner struct {


exported type TestRunner should have comment or be unexported

ruflin

++ on having this.

ruflin · 2018-08-15T12:49:18Z

metricbeat/module/redis/mtest/runner.go

+	DataRunner = compose.TestRunner{
+		Service: "redis",
+		Options: map[string][]string{
+			"REDIS_VERSION": []string{


+1 on always having the newest version here which is GA.

ruflin · 2018-08-15T12:49:45Z

metricbeat/module/redis/mtest/runner.go

+var (
+	Runner = compose.TestRunner{
+		Service: "redis",
+		Options: map[string][]string{


This is great. Perhaps in the future this could even be generated from the fields.yml file.

Yes, I want to keep this as a struct with basic data in case we move it to a yaml file.

ruflin · 2018-08-15T12:52:05Z

libbeat/tests/compose/runner.go

+		timeout = 300
+	}
+
+	for _, s := range r.scenarios() {


Would be nice to still have something like NO_COMPOSE=1 that can be set if the environment is already running on the local machine so it will not start up all docker containers. Perhaps that still exists ...

I have just added something like this, look at 89f5f0e

I prefer to do it as a host override instead of only a feature disable for two reasons:

Depending on how you are running the environment you want to test with you are going to need to override the host in any case

"Running environment" is going to be trickier now, as each service can be running with multiple versions and configurations

Add the possibility to override some options of compose runner using environment variables: - If there is a variable with the upercased name of the service + "_HOST" (i.e. REDIS_HOST for "redis" service), no scenario is created and the value of the variable is injected to the test in the host variable. - If there is a variable named as one of the options, this single value is used instead of the list of values in the options matrix.

libbeat/tests/compose/runner.go

houndci-bot · 2018-08-21T08:55:31Z

metricbeat/module/kafka/partition/partition_integration_test.go

-	var offsetBefore int64 = 0
-	var offsetAfter int64 = 0
+		var offsetBefore int64 = 0
+		var offsetAfter int64 = 0


should drop = 0 from declaration of var offsetAfter; it is the zero value

houndci-bot · 2018-08-21T08:55:31Z

metricbeat/module/kafka/partition/partition_integration_test.go


-	var offsetBefore int64 = 0
-	var offsetAfter int64 = 0
+		var offsetBefore int64 = 0


should drop = 0 from declaration of var offsetBefore; it is the zero value

metricbeat/module/kafka/mtest/runner.go

houndci-bot · 2019-03-06T17:27:49Z

metricbeat/module/couchbase/testing.go

-		dsn = "http://Administrator:password@localhost:8091"
-	}
-	return dsn
+func GetEnvDSN(host string) string {


exported function GetEnvDSN should have comment or be unexported

ruflin · 2019-03-07T13:53:23Z

metricbeat/module/aerospike/mtest/runner.go

+	// Runner is the compose test runner for aerospike
+	Runner = compose.TestRunner{
+		Service: "aerospike",
+		Options: compose.RunnerOptions{


The config files I started to introduce here could be a place where we have this documented in the future: https://github.com/elastic/beats/pull/11131/files#diff-7d1232f25fa4ea3006884196c00228c7R3 (not for this PR)

Add a helper to automatically obtain the host of a service in metricbeat integration tests so tests don't depend on environment variables or current running docker services. This is needed to allow in the future to run multiple versions of the same service, is also helpful when running tests locally. Environment variables with the format SERVICE_HOST, being SERVICE the uppercased name of the compose service can be used to override the automatic value. `NO_COMPOSE` environment variable can be used now to avoid stopping the running containers too. Add ports to all services in docker compose files, required for the helper. Part of elastic#7957 for python-based integration tests.

Add a helper to automatically obtain the host of a service in metricbeat integration tests so tests don't depend on environment variables or current running docker services. This is needed to allow in the future to run multiple versions of the same service, is also helpful when running tests locally. Environment variables with the format `SERVICE_HOST`, being SERVICE the uppercased name of the compose service can be used to override the automatic value. `NO_COMPOSE` environment variable can be used now to avoid stopping the running containers too. Add ports to all services in docker compose files, required for the helper. Part of #7957 for python-based integration tests.

jsoriano · 2019-09-02T13:35:28Z

Closing this as it is not going to be merged as is, and many of the ideas explored here have been already merged. Also another approach for end to end testing that could cover testing with multiple versions is being explored in https://github.com/elastic/metricbeat-tests-poc

jsoriano added in progress Pull request is currently in progress. :Testing labels Aug 13, 2018

houndci-bot reviewed Aug 13, 2018

View reviewed changes

andrewkroh reviewed Aug 13, 2018

View reviewed changes

houndci-bot reviewed Aug 14, 2018

View reviewed changes

jsoriano force-pushed the compose-test-runner branch from e609b43 to c463a75 Compare August 14, 2018 18:19

jsoriano added 8 commits August 15, 2018 10:11

Initial implementation of compose test runner

f67d8d7

Parallel

39cb492

More parallel

1f12a3e

--no-ansi for compose

56729d7

Randomize project name

ce4ab51

Seed random generator for compose project names

c2b8060

Migrate info metricset

166709d

Run test suites

0f1a8d6

jsoriano force-pushed the compose-test-runner branch from c463a75 to 0f1a8d6 Compare August 15, 2018 08:13

ruflin reviewed Aug 15, 2018

View reviewed changes

houndci-bot reviewed Aug 15, 2018

View reviewed changes

libbeat/tests/compose/runner.go Show resolved Hide resolved

jsoriano added 8 commits August 15, 2018 17:08

Typo

57e11ea

Use more explicit healthcheck for redis

d7fe343

Keep a redis service in the main docker-compose file

d71d210

Redis test use compose provided hosts

938ba32

Run beat container in host network mode

4301a8d

Run redis python tests with two versions

a572279

Merge remote-tracking branch 'origin/master' into compose-test-runner

180e31f

Add compose runner for kafka

8dd0a98

houndci-bot reviewed Aug 21, 2018

View reviewed changes

Use docker compose 2.1 version

38cc081

jsoriano requested a review from a team as a code owner January 19, 2019 20:59

jsoriano added 2 commits January 20, 2019 02:15

Update versions of the stack

fae8cda

Fix redis key test

ffe2062

jsoriano force-pushed the compose-test-runner branch from 7f274cf to ffe2062 Compare January 20, 2019 13:00

jsoriano mentioned this pull request Feb 27, 2019

Not possible to run integration tests on OS X #10953

Closed

jsoriano added 2 commits March 5, 2019 21:56

Merge remote-tracking branch 'origin/master' into compose-test-runner

dfd3e0d

Merge remote-tracking branch 'origin/master' into compose-test-runner

86cc04b

jsoriano mentioned this pull request Mar 6, 2019

[Metricbeat] Migrate Ceph cluster_status to use ReporterV2 interface #10993

Merged

Merge remote-tracking branch 'origin/master' into compose-test-runner

90a9e0f

houndci-bot reviewed Mar 6, 2019

View reviewed changes

jsoriano added 2 commits March 6, 2019 18:34

Move couchbase DSN helper to mtest

64dfd65

Fix mysql tests

2c9cd97

ruflin reviewed Mar 7, 2019

View reviewed changes

ruflin mentioned this pull request Mar 7, 2019

[Metricbeat] Make integration tests more generic for modules #11131

Closed

jsoriano mentioned this pull request Mar 14, 2019

Skip password test in metricbeat redis module #11243

Merged

jsoriano mentioned this pull request Mar 21, 2019

Cherry-pick #11243 to 7.0: Skip password test in metricbeat redis module #11367

Merged

jsoriano mentioned this pull request Apr 24, 2019

Add helper to get hosts on metricbeat integration tests #11925

Merged

jsoriano mentioned this pull request May 9, 2019

[metricbeat] Expand metricbeat dev guide for testing #12105

Merged

cachedout removed the request for review from a team June 26, 2019 10:21

jsoriano mentioned this pull request Jul 15, 2019

Improvements in metricbeat modules integration tests #12909

Closed

6 tasks

This was referenced Jul 23, 2019

Add test runner for Metricbeat integration tests #13039

Closed

Add helper to get hosts on Go Metricbeat integration tests #13055

Merged

Fail earlier on compose errors #13110

Closed

Release lock of docker compose file if possible #13111

Merged

jsoriano closed this Sep 2, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP/Ideas] Config matrices for integration tests #7957

[WIP/Ideas] Config matrices for integration tests #7957

jsoriano commented Aug 13, 2018 •

edited

Loading

houndci-bot Aug 13, 2018

houndci-bot Aug 13, 2018

andrewkroh Aug 13, 2018

jsoriano Nov 22, 2018

ruflin Nov 22, 2018

jsoriano Nov 23, 2018

ruflin commented Aug 14, 2018

houndci-bot Aug 14, 2018

houndci-bot Aug 14, 2018

houndci-bot Aug 14, 2018

houndci-bot Aug 14, 2018

ruflin left a comment

ruflin Aug 15, 2018

ruflin Aug 15, 2018

jsoriano Aug 15, 2018

ruflin Aug 15, 2018

jsoriano Aug 15, 2018 •

edited

Loading

houndci-bot Aug 21, 2018

houndci-bot Aug 21, 2018

houndci-bot Mar 6, 2019

ruflin Mar 7, 2019 •

edited

Loading

jsoriano commented Sep 2, 2019

[WIP/Ideas] Config matrices for integration tests #7957

[WIP/Ideas] Config matrices for integration tests #7957

Conversation

jsoriano commented Aug 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Aug 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano Aug 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin Mar 7, 2019 • edited Loading

Choose a reason for hiding this comment

jsoriano commented Sep 2, 2019

jsoriano commented Aug 13, 2018 •

edited

Loading

jsoriano Aug 15, 2018 •

edited

Loading

ruflin Mar 7, 2019 •

edited

Loading