Skip to content

Commit

Permalink
test: set ESM_DISABLE_CACHE to maybe fix CI-vs--r-esm
Browse files Browse the repository at this point in the history
We were seeing random test failures in the zoe unit tests (specifically
test-offerSafety.js) that looked like:

```
  Uncaught exception in test/unitTests/test-offerSafety.js

  /home/runner/work/agoric-sdk/agoric-sdk/packages/weak-store/src/weakStore.js:5
  import { assert, details, q } from '@agoric/assert';
  ^^^^^^

  SyntaxError: Cannot use import statement outside a module

  ✖ test/unitTests/test-offerSafety.js exited with a non-zero exit code: 1
```

or:

```
  Uncaught exception in test/unitTests/test-offerSafety.js

  /home/runner/work/agoric-sdk/agoric-sdk/packages/zoe/test/unitTests/setupBasicMints.js:1
  import { makeIssuerKit } from '@agoric/ertp';

  SyntaxError: Cannot use import statement outside a module

  ✖ test/unitTests/test-offerSafety.js exited with a non-zero exit code: 1
```

under various versions of Node.js. The error suggests that `-r esm` was not
active (the error site tried to import an ESM-style module, and failed
because the import site was CJS not ESM), however error site itself was in a
module, meaning `-r esm` was active up until that moment. This makes no
sense.

The AVA runner has had some amount of built-in support for ESM, in which it
uses Babel to rewrite test files (but perhaps not the code being tested). If
that were in effect, it might explain how `test-offerSafety.js` could be
loaded, but `weakStore.js` could not. It is less likely to explain how
`setupBasicMints.js` could be loaded but `makeIssuerKit` could not, unless
AVA somehow believes that `setupBasicMints.js` is a different category of
file than the ones pulled from other packages.

In any case, I believe we're bypassing that built-in/Babel support, by using
their recommended `-r esm` integration recipe (package.json has
`ava.require=['esm']`). However the AVA "ESM Plan"
(avajs/ava#2293) is worth reading, if only
for our future move-to-native-ESM plans (#527).

So my hunch here is that the `-r esm` module's cache is not safe against
concurrent access, and AVA's parallel test invocation means there are
multiple processes reading and writing to that cache in parallel. Zoe has
more source files than most other packages, which might increase the
opportunity for a cache-corruption bug to show up. This sort of bug might not
show up locally because the files are already in the cache, whereas CI may
not already have them populated.

This patch adds `ESM_DISABLE_CACHE=true` to the environment variables used in
all tests, in an attempt to avoid this hypothetical bug. Stale entries in
this cache has caused us problems before, so most of us have the same setting
in our local shells.

Another potential workaround would be to add `--serial` to the `ava`
invocation, however the Zoe test suite is large enough that we really to want
the parallelism, just to make the tests finish faster.

This patch also increases the Zoe test timeout to 10m, just in case. I
observed a few tests taking 1m30s or 1m40s to complete, and the previous
timeout was 2m, which was too close to the edge.
  • Loading branch information
warner committed Aug 27, 2020
1 parent 45015dc commit ec8c456
Show file tree
Hide file tree
Showing 2 changed files with 69 additions and 2 deletions.
67 changes: 67 additions & 0 deletions .github/workflows/test-all-packages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ on:
branches: [master]
pull_request:

# set ESM_DISABLE_CACHE=true (will be JSON parsed)
jobs:
build:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -110,64 +111,124 @@ jobs:
# run: yarn test
- name: yarn test (acorn-eventual-send)
run: cd packages/acorn-eventual-send && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (agoric-cli)
run: cd packages/agoric-cli && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (assert)
run: cd packages/assert && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (bundle-source)
run: cd packages/bundle-source && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (captp)
run: cd packages/captp && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (dapp-svelte-wallet/api)
run: cd packages/dapp-svelte-wallet/api && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (deployment)
run: cd packages/deployment && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (ERTP)
run: cd packages/ERTP && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (eventual-send)
run: cd packages/eventual-send && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (import-bundle)
run: cd packages/import-bundle && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (import-manager)
run: cd packages/import-manager && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (install-metering-and-ses)
run: cd packages/install-metering-and-ses && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (install-ses)
run: cd packages/install-ses && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (marshal)
run: cd packages/marshal && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (notifier)
run: cd packages/notifier && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (promise-kit)
run: cd packages/promise-kit && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (registrar)
run: cd packages/registrar && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (same-structure)
run: cd packages/same-structure && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (sharing-service)
run: cd packages/sharing-service && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (sparse-ints)
run: cd packages/sparse-ints && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (spawner)
run: cd packages/spawner && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (stat-logger)
run: cd packages/stat-logger && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (store)
run: cd packages/store && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (swing-store-lmdb)
run: cd packages/swing-store-lmdb && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (swing-store-simple)
run: cd packages/swing-store-simple && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (swingset-runner)
run: cd packages/swingset-runner && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (tame-metering)
run: cd packages/tame-metering && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (transform-eventual-send)
run: cd packages/transform-eventual-send && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (transform-metering)
run: cd packages/transform-metering && yarn test
env:
ESM_DISABLE_CACHE: true
- name: yarn test (weak-store)
run: cd packages/weak-store && yarn test
env:
ESM_DISABLE_CACHE: true

##############
# Long-running tests are executed individually.
Expand Down Expand Up @@ -212,6 +273,8 @@ jobs:
${{ runner.os }}-go-
- name: yarn test (cosmic-swingset)
run: cd packages/cosmic-swingset && yarn test
env:
ESM_DISABLE_CACHE: true

test-swingset:
# BEGIN-TEST-BOILERPLATE
Expand Down Expand Up @@ -243,6 +306,8 @@ jobs:
# END-RESTORE-BOILERPLATE
- name: yarn test (SwingSet)
run: cd packages/SwingSet && yarn test
env:
ESM_DISABLE_CACHE: true

test-zoe:
# BEGIN-TEST-BOILERPLATE
Expand Down Expand Up @@ -274,3 +339,5 @@ jobs:
# END-RESTORE-BOILERPLATE
- name: yarn test (zoe)
run: cd packages/zoe && yarn test
env:
ESM_DISABLE_CACHE: true
4 changes: 2 additions & 2 deletions packages/zoe/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
},
"scripts": {
"build": "yarn build-zcfBundle",
"test": "ava",
"test": "ava --verbose",
"build-zcfBundle": "node -r esm scripts/build-zcfBundle.js",
"lint-fix": "yarn lint --fix",
"lint-check": "yarn lint",
Expand Down Expand Up @@ -63,7 +63,7 @@
"ava": {
"files": ["test/**/test-*.js"],
"require": ["esm"],
"timeout": "2m"
"timeout": "10m"
},
"eslintConfig": {
"extends": [
Expand Down

0 comments on commit ec8c456

Please sign in to comment.