Skip to content

Conversation

@bicknellr
Copy link
Collaborator

There seems to be a cache somewhere between the server hosting the tests (which runs this plugin) and the browsers we're testing on Sauce Labs. Without this header, this cache serves responses generated for one browser's requests to those of another browser, which may have different compliation settings.

These different browsers which seem to be having their responses mixed up are running in different OSs on different VM instances, so I don't think this cache could possibly be running anywhere on the client VM. Also, I've checked that the requests are in fact reaching the server and responses are being compiled - the response is just different by the time the browser receives it. This makes me think that the culprit is Sauce Labs Proxy, but the docs for Sauce Labs Proxy claim that it no longer caches.1 They also recently deprecated a --no-proxy-caching flag2, so I'm skeptical.

Setting Cache-Control: no-cache can't be used to work around this because @web/test-runner unconditionally installs server middleware3 that almost always sets Cache-Control: public, max-age=31536000 in non-interactive scenarios.4

Fortunately, this unknown cache seems to be responsive to the Vary request header. The plugin specifically uses the User-Agent request header to determine compilation settings, so adding Vary: User-Agent will cause the cache to use fresh responses for any change in the UA.

Footnotes

  1. https://docs.saucelabs.com/secure-connections/sauce-connect/advanced/architecture/#sauce-connect-proxy-client

  2. https://wiki.saucelabs.com/pages/viewpage.action?pageId=48365781

  3. cacheMiddleware is added before ...config.middleware, giving it higher precedence. It also calls await ctx.next() before doing any work, so it always runs last and overwrites any Cache-Control header added by any user
    middleware: https://github.com/modernweb-dev/web/blob/da7901116322fc112ea45413e6be87f3f770f15e/packages/test-runner-core/src/server/TestRunnerServer.ts#L50

  4. https://github.com/modernweb-dev/web/blob/fdcf2e59253c619aaa07938e5710f9db24841709/packages/test-runner-core/src/server/middleware/cacheMiddleware.ts#L12

…on plugin. (...)

There seems to be a cache somewhere between the server hosting the tests (which
runs this plugin) and the browsers we're testing on Sauce Labs. Without this
header, this cache serves responses generated for one browser's requests to
those of another browser, which may have different compliation settings.

These different browsers which seem to be having their responses mixed up are
running in different OSs on different VM instances, so I don't think this cache
could possibly be running anywhere on the client VM. Also, I've checked that the
requests are in fact reaching the server and responses are being compiled - the
response is just different by the time the browser receives it. This makes me
think that the culprit is Sauce Labs Proxy, but the docs for Sauce Labs Proxy
claim that it no longer caches.[^1] They also recently deprecated a
`--no-proxy-caching` flag[^2], so I'm skeptical.

[^1]: <https://docs.saucelabs.com/secure-connections/sauce-connect/advanced/architecture/#sauce-connect-proxy-client>
[^2]: <https://wiki.saucelabs.com/pages/viewpage.action?pageId=48365781>

Setting `Cache-Control: no-cache` can't be used to work around this because
`@web/test-runner` unconditionally installs server middleware[^3] that almost
always sets `Cache-Control: public, max-age=31536000` in non-interactive
scenarios.[^4]

[^3]: `cacheMiddleware` is added before `...config.middleware`, giving it higher
precedence. It also calls `await ctx.next()` _before_ doing any work, so it
always runs last and overwrites any `Cache-Control` header added by any user
middleware: <https://github.com/modernweb-dev/web/blob/da7901116322fc112ea45413e6be87f3f770f15e/packages/test-runner-core/src/server/TestRunnerServer.ts#L50>
[^4]: <https://github.com/modernweb-dev/web/blob/fdcf2e59253c619aaa07938e5710f9db24841709/packages/test-runner-core/src/server/middleware/cacheMiddleware.ts#L12>

Fortunately, this unknown cache seems to be responsive to the `Vary` request
header. The plugin specifically uses the `User-Agent` request header to
determine compilation settings, so adding `Vary: User-Agent` will cause the
cache to use fresh responses for any change in the UA.
@bicknellr bicknellr merged commit 1d4f740 into master Sep 15, 2022
@bicknellr bicknellr deleted the tests-wtr-vary-ua branch September 15, 2022 20:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants