Simplify integration test workflow fixes #1914 #2354

jaredlockhart · 2020-03-04T20:18:19Z

Okay! So I made some changes to the integration test workflow to hopefully make things simpler to use.

Changes:

Reuse the standard local dev for running the integration tests instead of a separate copy
Change make kill to REALLY kill everything it can
Setup the vnc server to disable headless mode so you can watch the tests running
Had to take out one of the tests because I couldn't think of a good way to get it passing while reusing the local dev instance (@jrbenny35 if you have an idea of how to keep that test around let me know and I'll put it back otherwise I'd rather lose a test to get this workflow working smoothly)

How to test this:

Manually kill/remove all your containers with
docker ps -aq | xargs docker kill
docker ps -aq | xargs docker rm
docker volume ls -q | xargs docker volume rm
Refresh to create a new db
make refresh
Start the regular dev server with either of
make up
make up_detached
Check that the local dev is running
https://localhost/
Run the headless integration tests in a separate shell
make integration_test
Start the vnc server with either of
make integration_vnc_up
make integration_vnc_up_detached
Connect to vnc server with password secret
vnc://localhost:5900
Right click on the linux desktop and go to Applications > Shell > Bash and enter
tox -c tests/integration/
This should run all the tests and you should be able to see the browser running them
Close vnc
Check your local dev is still running and full of dummy integration test experiments nowo
https://localhost/
You should also be able to use @jrbenny35 's instructions to run the tests locally against the same dev instance: Add docs to run integration-tests #2317

So this way you can keep your local dev running the whole time and then hit it with headless/headful(?) integration tests in any order/combination. I think this should make using/maintaining/modifying the integration tests a lot simpler hopefully 🤞

Let me know if it breaks!

jaredlockhart · 2020-03-04T22:01:24Z

Blocked on conflicts with #2316

We should land that one first and then I'll update this

b4handjr

Just ran through this. Only found one thing, everything else works great!

Makefile

jaredlockhart · 2020-03-06T22:31:54Z

Also it just occurred to me I should add those exact instructions to the README or something.

b4handjr

Just some thoughts

b4handjr · 2020-03-12T17:30:55Z

tests/integration/conftest.py

@@ -15,7 +22,7 @@ def capabilities(capabilities):
 @pytest.fixture
 def firefox_options(firefox_options):
    """Set Firefox Options."""
-    firefox_options.headless = True
+    firefox_options.headless = not bool(os.environ.get("DISABLE_HEADLESS"))


We can get rid of this line and just use the env variable MOZ_HEADLESS.

I couldn't get it to work quickly so I filed it as a followup:

#2389

b4handjr · 2020-03-12T17:33:17Z

docker-compose.integration-test.yml

+    networks:
+      - private_nw
+      - public_nw
+  vnc:


I think instead of having 2 services, we could just use the MOZ_HEADLESS env variable. If it is set then the tests would not be viewable through the vnc server.

tests/integration/test_overview.py

Thanks for the review ben! Don't want this to languish too long so I'm gonna land it now.

tiftran

Checked out and tried this out. Looks good to me 👍

@chayim

Bumps [redis](https://github.com/redis/redis-py) from 3.5.3 to 4.5.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/redis/redis-py/releases">redis's releases</a>.</em></p> <blockquote> <h2>4.5.3</h2> <h1>Changes</h1> <p>Update urgency: HIGH: There is a critical bug that may affect a subset of users. Upgrade!</p> <h2>🐛 Bug Fixes</h2> <ul> <li><a href="https://cwe.mitre.org/data/definitions/404.html">CWE-404</a> AsyncIO Race Condition Fix (<a href="https://redirect.github.com/redis/redis-py/issues/2624">#2624</a>, <a href="https://redirect.github.com/redis/redis-py/issues/2579">#2579</a>)</li> </ul> <h2>4.5.2</h2> <h1>Changes</h1> <h2>🚀 New Features</h2> <ul> <li>Introduce AbstractConnection so that UnixDomainSocketConnection can call super().<strong>init</strong> (<a href="https://redirect.github.com/redis/redis-py/issues/2588">#2588</a>)</li> <li>Added queue_class to REDIS_ALLOWED_KEYS (<a href="https://redirect.github.com/redis/redis-py/issues/2577">#2577</a>)</li> <li>Made search document subscriptable (<a href="https://redirect.github.com/redis/redis-py/issues/2615">#2615</a>)</li> <li>Sped up the protocol parsing (<a href="https://redirect.github.com/redis/redis-py/issues/2596">#2596</a>)</li> </ul> <h2>🐛 Bug Fixes</h2> <ul> <li>Fix behaviour of async PythonParser to match RedisParser as for issue <a href="https://redirect.github.com/redis/redis-py/issues/2349">#2349</a> (<a href="https://redirect.github.com/redis/redis-py/issues/2582">#2582</a>)</li> <li>Replace async_timeout by asyncio.timeout (<a href="https://redirect.github.com/redis/redis-py/issues/2602">#2602</a>)</li> <li>Update json().arrindex() default values (<a href="https://redirect.github.com/redis/redis-py/issues/2611">#2611</a>)</li> </ul> <h2>🧰 Maintenance</h2> <ul> <li>Coverage for pypy-3.9 (<a href="https://redirect.github.com/redis/redis-py/issues/2608">#2608</a>)</li> <li>Developer Experience: Adding redis version compatibility details to the README (<a href="https://redirect.github.com/redis/redis-py/issues/2621">#2621</a>)</li> <li>Remove redundant assignment to RedisCluster.nodes_manager. (<a href="https://redirect.github.com/redis/redis-py/issues/2620">#2620</a>)</li> <li>Developer Experience: [types] update return type of smismember to list[int] (<a href="https://redirect.github.com/redis/redis-py/issues/2617">#2617</a>)</li> <li>Developer Experience: [docs] ConnectionPool SSL example (<a href="https://redirect.github.com/redis/redis-py/issues/2605">#2605</a>)</li> <li>Developer Experience: Fixed CredentialsProvider examples (<a href="https://redirect.github.com/redis/redis-py/issues/2587">#2587</a>)</li> <li>Developer Experience: Update README to make pip install copy-pastable on zsh (<a href="https://redirect.github.com/redis/redis-py/issues/2584">#2584</a>)</li> <li>Developer Experience: Fix for <code>lpop</code> and <code>rpop</code> return typing (<a href="https://redirect.github.com/redis/redis-py/issues/2590">#2590</a>)</li> </ul> <h2>Contributors</h2> <p>We'd like to thank all the contributors who worked on this release!</p> <p><a href="https://github.com/CrimsonGlory"><code>@CrimsonGlory</code></a>, <a href="https://github.com/Galtozzy"><code>@Galtozzy</code></a>, <a href="https://github.com/aksinha334"><code>@aksinha334</code></a>, <a href="https://github.com/barshaul"><code>@barshaul</code></a>, <a href="https://github.com/chayim"><code>@chayim</code></a>, <a href="https://github.com/davemcphee"><code>@davemcphee</code></a>, <a href="https://github.com/dvora-h"><code>@dvora-h</code></a>, <a href="https://github.com/kristjanvalur"><code>@kristjanvalur</code></a>, <a href="https://github.com/ryin1"><code>@ryin1</code></a>, <a href="https://github.com/sileht"><code>@sileht</code></a>, <a href="https://github.com/thebarbershop"><code>@thebarbershop</code></a>, <a href="https://github.com/uglide"><code>@uglide</code></a>, <a href="https://github.com/woutdenolf"><code>@woutdenolf</code></a> and <a href="https://github.com/zakaf"><code>@zakaf</code></a></p> <h2>4.5.1</h2> <h1>Changes</h1> <h2>🐛 Bug Fixes</h2> <ul> <li>Fix <a href="https://redirect.github.com/redis/redis-py/issues/2581">#2581</a> <code>UnixDomainSocketConnection</code> object has no attribute <code>_command_packer</code> (<a href="https://redirect.github.com/redis/redis-py/issues/2583">#2583</a>)</li> </ul> <h2>Contributors</h2> <p>We'd like to thank all the contributors who worked on this release!</p>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/redis/redis-py/blob/master/CHANGES">redis's changelog</a>.</em></p> <blockquote> <pre><code>* Allow data to drain from async PythonParser when reading during a disconnect() * Use asyncio.timeout() instead of async_timeout.timeout() for python >= 3.11 ([#2602](redis/redis-py#2602)) * Add test and fix async HiredisParser when reading during a disconnect() ([#2349](redis/redis-py#2349)) * Use hiredis-py pack_command if available. * Support `.unlink()` in ClusterPipeline * Simplify synchronous SocketBuffer state management * Fix string cleanse in Redis Graph * Make PythonParser resumable in case of error ([#2510](redis/redis-py#2510)) * Add `timeout=None` in `SentinelConnectionManager.read_response` * Documentation fix: password protected socket connection ([#2374](redis/redis-py#2374)) * Allow `timeout=None` in `PubSub.get_message()` to wait forever * add `nowait` flag to `asyncio.Connection.disconnect()` * Update README.md links * Fix timezone handling for datetime to unixtime conversions * Fix start_id type for XAUTOCLAIM * Remove verbose logging from cluster.py * Add retry mechanism to async version of Connection * Compare commands case-insensitively in the asyncio command parser * Allow negative `retries` for `Retry` class to retry forever * Add `items` parameter to `hset` signature * Create codeql-analysis.yml ([#1988](redis/redis-py#1988)). Thanks @chayim * Add limited support for Lua scripting with RedisCluster * Implement `.lock()` method on RedisCluster * Fix cursor returned by SCAN for RedisCluster & change default target to PRIMARIES * Fix scan_iter for RedisCluster * Remove verbose logging when initializing ClusterPubSub, ClusterPipeline or RedisCluster * Fix broken connection writer lock-up for asyncio ([#2065](redis/redis-py#2065)) * Fix auth bug when provided with no username ([#2086](redis/redis-py#2086)) * Fix missing ClusterPipeline._lock ([#2189](redis/redis-py#2189)) * Added dynaminc_startup_nodes configuration to RedisCluster * Fix reusing the old nodes' connections when cluster topology refresh is being done * Fix RedisCluster to immediately raise AuthenticationError without a retry * ClusterPipeline Doesn't Handle ConnectionError for Dead Hosts ([#2225](redis/redis-py#2225)) * Remove compatibility code for old versions of Hiredis, drop Packaging dependency * The `deprecated` library is no longer a dependency * Failover handling improvements for RedisCluster and Async RedisCluster ([#2377](redis/redis-py#2377)) * Fixed "cannot pickle '_thread.lock' object" bug ([#2354](redis/redis-py#2354), [#2297](redis/redis-py#2297)) * Added CredentialsProvider class to support password rotation * Enable Lock for asyncio cluster mode * Fix Sentinel.execute_command doesn't execute across the entire sentinel cluster bug ([#2458](redis/redis-py#2458)) * Added a replacement for the default cluster node in the event of failure ([#2463](redis/redis-py#2463)) * Fix for Unhandled exception related to self.host with unix socket ([#2496](redis/redis-py#2496)) </code></pre> <ul> <li>4.1.3 (Feb 8, 2022) <ul> <li>Fix flushdb and flushall (<a href="https://redirect.github.com/redis/redis-py/issues/1926">#1926</a>)</li> <li>Add redis5 and redis4 dockers (<a href="https://redirect.github.com/redis/redis-py/issues/1871">#1871</a>)</li> <li>Change json.clear test multi to be up to date with redisjson (<a href="https://redirect.github.com/redis/redis-py/issues/1922">#1922</a>)</li> <li>Fixing volume for unstable_cluster docker (<a href="https://redirect.github.com/redis/redis-py/issues/1914">#1914</a>)</li> <li>Update changes file with changes since 4.0.0-beta2 (<a href="https://redirect.github.com/redis/redis-py/issues/1915">#1915</a>)</li> </ul> </li> <li>4.1.2 (Jan 27, 2022)</li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/redis/redis-py/commit/66a4d6b2a493dd3a20cc299ab5fef3c14baad965"><code>66a4d6b</code></a> AsyncIO Race Condition Fix (<a href="https://redirect.github.com/redis/redis-py/issues/2641">#2641</a>)</li> <li><a href="https://github.com/redis/redis-py/commit/318b114f4da9846a2a7c150e1fb702e9bebd9fdf"><code>318b114</code></a> Version 4.5.2 (<a href="https://redirect.github.com/redis/redis-py/issues/2627">#2627</a>)</li> <li><a href="https://github.com/redis/redis-py/commit/1b2f408259405d412d7530291902f9e0c8bd34b3"><code>1b2f408</code></a> Fix behaviour of async PythonParser to match RedisParser as for issue <a href="https://redirect.github.com/redis/redis-py/issues/2349">#2349</a> (...</li> <li><a href="https://github.com/redis/redis-py/commit/7d474f90453c7b90bd06c94e0250b618120a599d"><code>7d474f9</code></a> introduce AbstractConnection so that UnixDomainSocketConnection can call supe...</li> <li><a href="https://github.com/redis/redis-py/commit/c87172347584301f453c601c483126e4800257b7"><code>c871723</code></a> pypy-3.9 CI (<a href="https://redirect.github.com/redis/redis-py/issues/2608">#2608</a>)</li> <li><a href="https://github.com/redis/redis-py/commit/d63313bf6080acaf18d61e072c78303adc0d4166"><code>d63313b</code></a> add queue_class to REDIS_ALLOWED_KEYS (<a href="https://redirect.github.com/redis/redis-py/issues/2577">#2577</a>)</li> <li><a href="https://github.com/redis/redis-py/commit/c61eeb2e3b5dff1f01eb1e665f424c7e75354f56"><code>c61eeb2</code></a> Adding supported redis/library details (<a href="https://redirect.github.com/redis/redis-py/issues/2621">#2621</a>)</li> <li><a href="https://github.com/redis/redis-py/commit/25e85e51e57b7aae9eb8fc77cfb0a45a07a501a7"><code>25e85e5</code></a> fix: replace async_timeout by asyncio.timeout (<a href="https://redirect.github.com/redis/redis-py/issues/2602">#2602</a>)</li> <li><a href="https://github.com/redis/redis-py/commit/91ab12a0f1bdf0e433131e1a51578e9fa2f89718"><code>91ab12a</code></a> Remove redundant assignment. (<a href="https://redirect.github.com/redis/redis-py/issues/2620">#2620</a>)</li> <li><a href="https://github.com/redis/redis-py/commit/8bfd492240fd33489a86cd3d353e3ece1fc94c10"><code>8bfd492</code></a> Making search document subscriptable (<a href="https://redirect.github.com/redis/redis-py/issues/2615">#2615</a>)</li> <li>Additional commits viewable in <a href="https://github.com/redis/redis-py/compare/3.5.3...v4.5.3">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=redis&package-manager=pip&previous-version=3.5.3&new-version=4.5.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/mozilla/experimenter/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Yashika Khurana <yashikakhuranayashika@gmail.com>

jaredlockhart requested review from b4handjr, uhlissuh and tiftran March 4, 2020 20:20

jaredlockhart added the Blocked label Mar 4, 2020

b4handjr requested changes Mar 6, 2020

View reviewed changes

Makefile Outdated Show resolved Hide resolved

jaredlockhart added 5 commits March 12, 2020 08:29

Simplify integration test workflow fixes #1914

23df915

Separate kill into two commands for the full stack with normandy/dc

087cedc

Suppress errors from failing to remove volumes

188993f

Try again to suppress volume rm errors

ee4874d

Integration tests need a bugzilla host to pass

ae113ad

jaredlockhart force-pushed the 1914 branch from b499543 to ae113ad Compare March 12, 2020 12:29

Fixed vnc up detached command

72dd270

jaredlockhart removed the Blocked label Mar 12, 2020

jaredlockhart requested a review from b4handjr March 12, 2020 12:45

b4handjr previously requested changes Mar 12, 2020

View reviewed changes

jaredlockhart added 4 commits March 16, 2020 13:55

Merge branch 'master' into 1914

f61731c

Restore and skip failing test

a26d5d3

Merge branch 'master' into 1914

a89f2e7

Update README

89444f0

tiftran approved these changes Mar 17, 2020

View reviewed changes

jaredlockhart merged commit bba8abd into master Mar 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simplify integration test workflow fixes #1914 #2354

Simplify integration test workflow fixes #1914 #2354

jaredlockhart commented Mar 4, 2020 •

edited by tiftran

Loading

jaredlockhart commented Mar 4, 2020 •

edited

Loading

b4handjr left a comment

jaredlockhart commented Mar 6, 2020

b4handjr left a comment

b4handjr Mar 12, 2020

jaredlockhart Mar 16, 2020

b4handjr Mar 12, 2020

tiftran left a comment

Simplify integration test workflow fixes #1914 #2354

Simplify integration test workflow fixes #1914 #2354

Conversation

jaredlockhart commented Mar 4, 2020 • edited by tiftran Loading

jaredlockhart commented Mar 4, 2020 • edited Loading

b4handjr left a comment

Choose a reason for hiding this comment

jaredlockhart commented Mar 6, 2020

b4handjr left a comment

Choose a reason for hiding this comment

b4handjr Mar 12, 2020

Choose a reason for hiding this comment

jaredlockhart Mar 16, 2020

Choose a reason for hiding this comment

b4handjr Mar 12, 2020

Choose a reason for hiding this comment

tiftran left a comment

Choose a reason for hiding this comment

jaredlockhart commented Mar 4, 2020 •

edited by tiftran

Loading

jaredlockhart commented Mar 4, 2020 •

edited

Loading