tests/e2e: introduce e2e framework basics, first test #1801

eduser25 · 2020-10-08T22:05:57Z

Adds first E2E helper framework and functions
Adds First test utilizing the framework
Added README.md

Co-authored-by: Jon Huhn jon.huhn@microsoft.com

Tests [X]

Please answer the following questions with yes/no.
Yes

Does this change contain code from or inspired by another project? If so, did you notify the maintainers and provide attribution?
Yes, relevant sections have been commented with exact reference implementations that were used.

- Adds first E2E helper framework and functions - Adds First test utilizing the framework - Added README.md Co-authored-by: Jon Huhn <jon.huhn@microsoft.com>

nojnhuh

Left a few comments inline on the README, mostly nits and clarifying kind vs. non-kind. I didn't look at the actual code assuming that hasn't changed from what we've been working off of.

Overall I'd say this is ready for review.

tests/e2e/README.md

nojnhuh · 2020-10-09T15:35:30Z

tests/e2e/README.md

+
+The framework exposes an OSM test data structure or handle, which is in effect the interaction mechanism for the test itself.  The test framework takes care to collect and initialize most of the common functionalities a test would expect when deploying on K8s, including but not limited to Kubernetes and SMI clientsets, flag parsing, container registry values, cleanup hooks, etc., and provides accessibility functions through the handle for the test to use at its own discretion.
+
+The hooks for initialization and cleanup are set at Ginkgo's `BeforeEach` at the top level of test execution (between  Ginkgo `Describes`); we henceforth recommend keeping every test in its own `Describe` section, as well as on a separate file for clarity. You can refer to `suite_test.go` for more details.


I would love to hear others' opinions on how best to structure the tests (at least within each top-level Describe block) and name them so they're easy to isolate.

This seems reasonable. The BeforeEach etc can be within the Describe or Context containers, so it seems ok that every e2e test be in its own Describe container.

tests/e2e/README.md

Co-authored-by: Jon Huhn <nojnhuh@users.noreply.github.com>

Co-authored-by: Jon Huhn <jon.huhn@microsoft.com>

It will run separately as an individual gh action later

codecov-io · 2020-10-09T20:06:53Z

Codecov Report

Merging #1801 into main will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1801   +/-   ##
=======================================
  Coverage   59.20%   59.20%           
=======================================
  Files         125      125           
  Lines        5170     5170           
=======================================
  Hits         3061     3061           
  Misses       2106     2106           
  Partials        3        3

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1291e4b...212fb37. Read the comment docs.

SC2006: Use $(..) instead of legacy `..`.

shashankram

Thanks for this change, will test it out.

shashankram · 2020-10-09T17:15:08Z

tests/e2e/README.md

+
+The framework exposes an OSM test data structure or handle, which is in effect the interaction mechanism for the test itself.  The test framework takes care to collect and initialize most of the common functionalities a test would expect when deploying on K8s, including but not limited to Kubernetes and SMI clientsets, flag parsing, container registry values, cleanup hooks, etc., and provides accessibility functions through the handle for the test to use at its own discretion.
+
+The hooks for initialization and cleanup are set at Ginkgo's `BeforeEach` at the top level of test execution (between  Ginkgo `Describes`); we henceforth recommend keeping every test in its own `Describe` section, as well as on a separate file for clarity. You can refer to `suite_test.go` for more details.


This seems reasonable. The BeforeEach etc can be within the Describe or Context containers, so it seems ok that every e2e test be in its own Describe container.

tests/e2e/README.md

I can't just quote though, won't work: https://stackoverflow.com/questions/37931155/correctly-allow-word-splitting-of-command-substitution-in-bash

tests/e2e/README.md

tests/e2e/e2e_pod_client_server_test.go

shashankram

Tested this out, looks good.

ritazh · 2020-10-12T16:30:11Z

tests/e2e/common.go

+	args = append(args, fmt.Sprintf("--enable-grafana=%v", instOpts.deployGrafana))
+	args = append(args, fmt.Sprintf("--deploy-jaeger=%v", instOpts.deployJaeger))
+
+	stdout, stderr, err := td.RunLocal(filepath.FromSlash("../../bin/osm"), args)


Is there a path to also test helm install

Not yet, but we have that scenario documented in our running list: #1714 (comment)

As discussed offline, helm install should be part of the test suite regardless of the scenarios we are testing. Added #1814 to track.

.golangci.yml

tests/e2e/README.md

michelleN

When running this manually, my test passed but I was confused by the output. Would you mind explaining what is happening below and why I'm seeing the REST req failed messages? and what does the nil mean here:Delete for osm-system: <nil>?

> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 0) Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req failed (status: 404) <nil>
> REST req succeeded: 200
> REST req succeeded: 200
> REST req succeeded: 200
> REST req succeeded: 200
> REST req succeeded: 200
STEP: Deleting SMI policies
> REST req failed correctly: Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed correctly: Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed correctly: Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed correctly: Remote exec err: command terminated with exit code 7 | stderr:
> REST req failed correctly: Remote exec err: command terminated with exit code 7 | stderr:
[AfterEach] [Top Level]
  /Users/michellenoorali/go/src/github.com/openservicemesh/osm/tests/e2e/suite_test.go:26
Deleting namespace client
Delete for client: <nil>
Deleting namespace server
Delete for server: <nil>
Deleting namespace osm-system
Delete for osm-system: <nil>
STEP: Waiting for namespaces [client server osm-system] to vanish
Deleting kind cluster: osm-e2e

eduser25 · 2020-10-12T19:30:12Z

I got similar feedback from @shashankram. I think we have a bit of misleading output on this first test, specifically and only because it was the first one, and we enabled a lot more verbosity on it when debugging.

We can totally improve the way it is presented to avoid misleading information and straightforward understanding.

tests/e2e/common_smi.go

eduser25 · 2020-10-12T19:58:02Z

@michelleN This will have to do for now til we get loggers and flags to enable log verbosity for specific parts of the test.

Wait for pods ready in ns [client]...
Finished waiting for NS [client].
STEP: Creating SMI policies
STEP: [WaitForRepeatedSuccess] waiting 1m0s for 5 iterations to succeed
> (client/client -> server.server:80/) HTTP Req failed 0 Remote exec err: command terminated with exit code 7 | stderr: 
> (client/client -> server.server:80/) HTTP Req failed 0 Remote exec err: command terminated with exit code 7 | stderr: 
> (client/client -> server.server:80/) HTTP Req failed 404 <nil>
> (client/client -> server.server:80/) HTTP Req failed 404 <nil>
> (client/client -> server.server:80/) HTTP Req failed 404 <nil>
> (client/client -> server.server:80/) HTTP Req succeeded: 200
> (client/client -> server.server:80/) HTTP Req succeeded: 200
> (client/client -> server.server:80/) HTTP Req succeeded: 200
> (client/client -> server.server:80/) HTTP Req succeeded: 200
> (client/client -> server.server:80/) HTTP Req succeeded: 200
STEP: Deleting SMI policies
STEP: [WaitForRepeatedSuccess] waiting 1m0s for 5 iterations to succeed
> (client/client -> server.server:80/) HTTP Req failed correctly: Remote exec err: command terminated with exit code 7 | stderr: 
> (client/client -> server.server:80/) HTTP Req failed correctly: Remote exec err: command terminated with exit code 7 | stderr: 
> (client/client -> server.server:80/) HTTP Req failed correctly: Remote exec err: command terminated with exit code 7 | stderr: 
> (client/client -> server.server:80/) HTTP Req failed correctly: Remote exec err: command terminated with exit code 7 | stderr: 
> (client/client -> server.server:80/) HTTP Req failed correctly: Remote exec err: command terminated with exit code 7 | stderr: 
[AfterEach] [Top Level]
  /home/eserra/src/osm/tests/e2e/suite_test.go:26
Deleting namespace osm-system
Deleting namespace client
Deleting namespace server
STEP: Waiting for namespaces [osm-system client server] to vanish

…h#1801) * E2E first commit - Adds first E2E helper framework and functions - Adds First test utilizing the framework - Added README.md Co-authored-by: Jon Huhn <jon.huhn@microsoft.com>

eduser25 added area/tests Related to tests wip Work-in-Progress labels Oct 8, 2020

eduser25 linked an issue Oct 8, 2020 that may be closed by this pull request

create an e2e framework for osm #1711

Closed

eduser25 requested review from nojnhuh and a team October 8, 2020 22:09

eduser25 force-pushed the testing-first branch 2 times, most recently from c02943e to aff9b70 Compare October 8, 2020 22:31

E2E first commit

44a5fe2

- Adds first E2E helper framework and functions - Adds First test utilizing the framework - Added README.md Co-authored-by: Jon Huhn <jon.huhn@microsoft.com>

eduser25 force-pushed the testing-first branch from aff9b70 to 44a5fe2 Compare October 9, 2020 00:14

eduser25 linked an issue Oct 9, 2020 that may be closed by this pull request

define and implement a set of e2e test scenarios for OSM #1714

Closed

3 tasks

Update README.md

2dcb893

nojnhuh reviewed Oct 9, 2020

View reviewed changes

eduser25 and others added 6 commits October 9, 2020 11:04

Apply suggestions from code review

60825c8

Co-authored-by: Jon Huhn <nojnhuh@users.noreply.github.com>

Addressing CR, Adding devel_guide section

53783c6

Co-authored-by: Jon Huhn <jon.huhn@microsoft.com>

ignore dot import lint issues for ginkgo/gomega

7426589

Fix .golangci.yml

14e8a93

removing e2e from test coverage

8812ef9

It will run separately as an individual gh action later

Fix dev_guide link

f602174

Fix SC2006

8a3f5c3

SC2006: Use $(..) instead of legacy `..`.

shashankram reviewed Oct 9, 2020

View reviewed changes

eduser25 added 3 commits October 9, 2020 13:36

Fix SC2046: Quote this to prevent word splitting.

a3eb97f

I can't just quote though, won't work: https://stackoverflow.com/questions/37931155/correctly-allow-word-splitting-of-command-substitution-in-bash

Merge branch 'main' into testing-first

f1e8f17

Apply suggestions from code review

4d0da59

eduser25 marked this pull request as ready for review October 9, 2020 21:20

eduser25 removed the wip Work-in-Progress label Oct 9, 2020

eduser25 requested a review from a team October 9, 2020 21:20

shashankram reviewed Oct 9, 2020

View reviewed changes

tests/e2e/README.md Show resolved Hide resolved

shashankram reviewed Oct 9, 2020

View reviewed changes

tests/e2e/e2e_pod_client_server_test.go Outdated Show resolved Hide resolved

shashankram previously approved these changes Oct 9, 2020

View reviewed changes

ritazh reviewed Oct 12, 2020

View reviewed changes

CR

00b8b3a

eduser25 dismissed shashankram’s stale review via 00b8b3a October 12, 2020 18:38

shashankram previously approved these changes Oct 12, 2020

View reviewed changes

nojnhuh mentioned this pull request Oct 12, 2020

Add an e2e test for permissive traffic policy mode #1815

Closed

michelleN reviewed Oct 12, 2020

View reviewed changes

.golangci.yml Show resolved Hide resolved

michelleN suggested changes Oct 12, 2020

View reviewed changes

tests/e2e/README.md Show resolved Hide resolved

tests/e2e/README.md Show resolved Hide resolved

tests/e2e/README.md Outdated Show resolved Hide resolved

add .golangci.yml comment

212fb37

nojnhuh dismissed shashankram’s stale review via 212fb37 October 12, 2020 19:23

michelleN reviewed Oct 12, 2020

View reviewed changes

snehachhabria reviewed Oct 12, 2020

View reviewed changes

tests/e2e/common_smi.go Outdated Show resolved Hide resolved

tests/e2e/common_smi.go Outdated Show resolved Hide resolved

nojnhuh mentioned this pull request Oct 12, 2020

define and implement a set of e2e test scenarios for OSM #1714

Closed

3 tasks

nojnhuh removed a link to an issue Oct 12, 2020

define and implement a set of e2e test scenarios for OSM #1714

Closed

3 tasks

Addressing CRs

5057e9b

shashankram changed the title ~~E2E first commit~~ tests/e2e: introduce e2e testing framework Oct 12, 2020

shashankram approved these changes Oct 12, 2020

View reviewed changes

snehachhabria approved these changes Oct 12, 2020

View reviewed changes

eduser25 changed the title ~~tests/e2e: introduce e2e testing framework~~ tests/e2e: introduces e2e framework basics, first test Oct 12, 2020

eduser25 changed the title ~~tests/e2e: introduces e2e framework basics, first test~~ tests/e2e: introduce e2e framework basics, first test Oct 12, 2020

michelleN approved these changes Oct 13, 2020

View reviewed changes

eduser25 merged commit 8faf189 into openservicemesh:main Oct 13, 2020

eduser25 deleted the testing-first branch October 13, 2020 00:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests/e2e: introduce e2e framework basics, first test #1801

tests/e2e: introduce e2e framework basics, first test #1801

eduser25 commented Oct 8, 2020

nojnhuh left a comment

nojnhuh Oct 9, 2020

shashankram Oct 9, 2020

codecov-io commented Oct 9, 2020 •

edited

Loading

shashankram left a comment

shashankram Oct 9, 2020

shashankram left a comment

ritazh Oct 12, 2020

nojnhuh Oct 12, 2020 •

edited

Loading

ritazh Oct 12, 2020

michelleN left a comment

eduser25 commented Oct 12, 2020

eduser25 commented Oct 12, 2020 •

edited

Loading


		The framework exposes an OSM test data structure or handle, which is in effect the interaction mechanism for the test itself. The test framework takes care to collect and initialize most of the common functionalities a test would expect when deploying on K8s, including but not limited to Kubernetes and SMI clientsets, flag parsing, container registry values, cleanup hooks, etc., and provides accessibility functions through the handle for the test to use at its own discretion.

		The hooks for initialization and cleanup are set at Ginkgo's `BeforeEach` at the top level of test execution (between Ginkgo `Describes`); we henceforth recommend keeping every test in its own `Describe` section, as well as on a separate file for clarity. You can refer to `suite_test.go` for more details.

tests/e2e: introduce e2e framework basics, first test #1801

tests/e2e: introduce e2e framework basics, first test #1801

Conversation

eduser25 commented Oct 8, 2020

nojnhuh left a comment

Choose a reason for hiding this comment

nojnhuh Oct 9, 2020

Choose a reason for hiding this comment

shashankram Oct 9, 2020

Choose a reason for hiding this comment

codecov-io commented Oct 9, 2020 • edited Loading

Codecov Report

shashankram left a comment

Choose a reason for hiding this comment

shashankram Oct 9, 2020

Choose a reason for hiding this comment

shashankram left a comment

Choose a reason for hiding this comment

ritazh Oct 12, 2020

Choose a reason for hiding this comment

nojnhuh Oct 12, 2020 • edited Loading

Choose a reason for hiding this comment

ritazh Oct 12, 2020

Choose a reason for hiding this comment

michelleN left a comment

Choose a reason for hiding this comment

eduser25 commented Oct 12, 2020

eduser25 commented Oct 12, 2020 • edited Loading

codecov-io commented Oct 9, 2020 •

edited

Loading

nojnhuh Oct 12, 2020 •

edited

Loading

eduser25 commented Oct 12, 2020 •

edited

Loading