test: E2E tests against ipfs-webui HEAD

[lidel]

Parent: ipfs/ipfs-webui#1164
go-ipfs counterpart: ipfs/kubo#6825

This PR adds TravisCI config for running ipfs-webui's end to end tests against js-ipfs from this repo.

It does not increase CI time, as its runs in parallel to other tests, and those take much longer:

Note: It uses aegir test-external which runs tests twice: first time against release version of js-ipfs, and if that is green, then against version from this repo. If not, we exit without error. That way we avoid breaking the js-ipfs build if upstream ipfs-webui is not green.

TODO

• webui needs build before e2e runx
• ensure binary from HEAD is used during second time webui's test is run
  • ipfs-webui switched to ipfsd-ctl v1.0.6 with a fix
• there is false-positive bug that needs to be fixed
  • suspect: aegir test-external returns 0 if test with original js-ipfs fails its by design ;)

cc @autonome @hugomrdias @alanshaw

[lidel]

Getting late, wrote some notes below to pick this up tomorrow.

Using wrong version of js-ipfs

Something felt off. Just to be sure we test the right thing, I updated js-ipfs version in package.json to end with -test and run test-external

$ E2E_IPFSD_TYPE=js npm run test:external -- ipfs-webui https://github.com/ipfs-shipyard/ipfs-webui.git --branch "fix/test-external"

This revealed a false-positive: for some reason the second time npm test is run during aegir text-external it does not switch to HEAD version of js-ipfs.

• Expected behavior would be to use js-ipfs injected via connect-deps to package.json (.connect-deps-cache/ipfs-0.40.0-test*.tgz)

• Instead, the require.resolve('ipfs/src/cli/bin.js') returns /tmp/ipfs-webui-{random}/node_modules/ipfs/src/cli/bin.js which belongs to old js-ipfs (yarn pack used by connect-deps seems to update only the manifest at /tmp/ipfs-webui-{random}/node_modules/ipfs/package.json):

  [test:e2e  ] E2E using js-ipfs/0.40.0 (/tmp/ipfs-webui-{random}/node_modules/ipfs/src/cli/bin.js)
  ...
  [test:e2e  ] E2E using js-ipfs/0.40.0 (/tmp/ipfs-webui-{random}/node_modules/ipfs/src/cli/bin.js)
  

Defining explicit IPFS_JS_EXEC solves the problem:

$ E2E_IPFSD_TYPE=js IPFS_JS_EXEC=$(realpath ./src/cli/bin.js) npm run test:external -- ipfs-webui https://github.com/ipfs-shipyard/ipfs-webui.git --branch "fix/test-external"

However, it comes at a price of executing both npm test runs against binary from IPFS_JS_EXEC :

[test:e2e  ] E2E using js-ipfs/0.40.0-test (/home/lidel/project/js-ipfs/src/cli/bin.js)
...
[test:e2e  ] E2E using js-ipfs/0.40.0-test (/home/lidel/project/js-ipfs/src/cli/bin.js)

False-positive on error during first npm test of test-external:

I've added throw new Error('TODO this is test of e2e error') to src/core/components/id.js to ensure tests fail during ipfs.id, however if first npm test run by test-external fails, the process ends with exit code 0 (success).

This seems to be a feature: we don't want to break js-ipfs build if upstream tests of 3rd party repo are broken already, we want to fail build only if upstream is green but switching to HEAD triggers error.

In short, this should be solved if we find a way to fix the first issue.

[lidel]

ipfs-webui e2e is green now: https://travis-ci.com/ipfs/js-ipfs/jobs/276855690
(thanks to @hugomrdias for the fix in ipfsd-ctl v1.0.6)

@alanshaw this is ready for review

[alanshaw]

@lidel would you mind rebasing so we can double check this branch is still passing?

[lidel]

@alanshaw had to apply small tweaks (ipfs/ipfs-webui@0fc43a8, ipfs/ipfs-webui@e139736) but its green again (works with both old and new API) 👌