🌟 feat: add websocket bootstrapers to the config

[daviddias]

Added the bootstrappers @lgierth created to the Bootstraper list. This is still not 100% working as it requires some updates throughout the dialing code (DNS addrs need to be valid wss/tcp).

• Make the WebSockets transport DNS aware [WIP] feat: dns support for WS libp2p/js-libp2p-websockets#47
• Decide if we want to loosen up CORS rules for bootstrapers 🌟 feat: add websocket bootstrapers to the config #740 (comment)

Assignee: @dryajov

[dryajov]

As far as I understand, this:

Make the WebSockets transport DNS aware

requires something similar to this - libp2p/js-libp2p-webrtc-star#84 here https://github.com/libp2p/js-libp2p-websockets?

[daviddias]

@dryajov exactly :)

[ghost]

Also, you'll probably need an Origin header, unless the browser sets it for you.

[daviddias]

@lgierth I need a green light from you on these.

Note, it is the only way browser nodes will connect to bootstrapers, it would be extremely useful.

[ghost]

Take these:

/dns4/ams-1.bootstrap.libp2p.io/tcp/443/wss/ipfs/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd
/dns4/sfo-1.bootstrap.libp2p.io/tcp/443/wss/ipfs/QmSoLju6m7xTh3DuokvT3886QRYqxAzb1kShaanJgW36yx
/dns4/lon-1.bootstrap.libp2p.io/tcp/443/wss/ipfs/QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3
/dns4/sfo-2.bootstrap.libp2p.io/tcp/443/wss/ipfs/QmSoLnSGccFuZQJzRadHn95W2CrSFmZuTdDWP8HXaHca9z
/dns4/sfo-3.bootstrap.libp2p.io/tcp/443/wss/ipfs/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM
/dns4/sgp-1.bootstrap.libp2p.io/tcp/443/wss/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu
/dns4/nyc-1.bootstrap.libp2p.io/tcp/443/wss/ipfs/QmSoLueR4xBeUbY9WZ9xGUUxunbKWcrNFTDAadQJmocnWm
/dns4/nyc-2.bootstrap.libp2p.io/tcp/443/wss/ipfs/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64

These addresses will be alot slimer once we have /dnsaddr (see go-multiaddr-dns, need to write it up further).

[daviddias]

The above actually don't work because they don't have valid SSL certs.

Tried with the fruits, they work with Node.js, but in the browser I get this:

[daviddias]

Seems that the gateway didn't like it, perhaps the origin? @lgierth how can I help you debug what happens on that node?

[daviddias]

@lgierth I want to have my websockets bootstrappeeeers \o/

[ghost]

On it, on it. Waiting on DNS verification for the cert now

[ghost]

They're all working now \o/

[daviddias]

Also, you'll probably need an Origin header, unless the browser sets it for you.

The browser does that automatically, using where the app has been loaded. Shall we enable websockets dials from anywhere?

[daviddias]

@dryajov merged your websockets PR, looks good.

Here it is still failing though, see:

» jsipfs daemon
Initializing daemon...
Swarm listening on /ip4/127.0.0.1/tcp/4003/ws/ipfs/QmdsZZEuqzAxRJy2NNFrXEeUzmyMkj8mgn4ybmh9Cfa5bH
Swarm listening on /ip4/127.0.0.1/tcp/4002/ipfs/QmdsZZEuqzAxRJy2NNFrXEeUzmyMkj8mgn4ybmh9Cfa5bH
Swarm listening on /ip4/192.168.1.135/tcp/4002/ipfs/QmdsZZEuqzAxRJy2NNFrXEeUzmyMkj8mgn4ybmh9Cfa5bH
/Users/koruza/code/js-ipfs/node_modules/multiaddr/src/codec.js:54
      throw ParseError('invalid address: ' + str)
      ^

Error: Error parsing address: invalid address: /dns4/strawberry.i.ipfs.io/wss/ipfs

[dryajov]

Ah, weird. Let me check it out.

[dryajov]

@diasdavid I see mafmt 2.1.6 being pulled in by most other dependencies. Wonder if thats the issue? Still investigating.

 yarn list|grep mafmt
│  ├─ mafmt@^2.1.6
│  ├─ mafmt@^2.1.6
│  ├─ mafmt@^2.1.6
│  ├─ mafmt@^2.1.6
│  ├─ mafmt@^2.1.6
│  ├─ mafmt@^2.1.6
├─ mafmt@2.1.7

[daviddias]

oh, it might be indeed. It should pick the latest patch though :( I guess we've to bump a minor and update everywhere manually .

[dryajov]

@diasdavid yeah, maybe worth making some of this deps as peerDependencies, so that only the top package pulls it in.

[dryajov]

btw, I can help updating some of this - wanna split them up?

[dignifiedquire]

@diasdavid you can just explicitly bump the version on the packages instead of releasing a minor, so that the minimum is the version you want

[dryajov]

@dignifiedquire not sure if I follow :) What do you mean by

you can just explicitly bump the version on the packages instead of releasing a minor, so that the minimum is the version you want

Just bump and release each dep that pulls in mafmt? I though thats what @diasdavid is doing.

Also, are you guys aware of any tool like https://github.com/lerna/lerna for non monorepos? I've been resisting the idea of monorepos for some time, but I can definitely see a use case for it with the amount of modules in IPFS/libp2p/multformats/ipld.

I don't think that it makes sense to merge all IPFS and friends under one big monorepo, but it be still nice to:

a) manage all related code in github (and other VSCs) groups (pull/push/rebase across multiple repos)
b) update package versions at once across all the repos in the group or groups of repos
c) release all or some packages at once

If you're already doing that I'd love to hear what you're using for it :)

[dignifiedquire]

I mean just changing the dependency in the package.json to "mafmt": "^2.1.7" so that the minimum is the new one

[dignifiedquire]

re lerna, yes this was discussed before (I actually have a monorepo to make linking easier on my local machine) ref ipfs/community#174

[dryajov]

@dignifiedquire ah, you're right. That should work.

Thanks for the monorepo pointer, gonna read that thread.

[dryajov]

@diasdavid actually, the issue is something else, the bug seems to be in railing here - https://github.com/libp2p/js-libp2p-railing/blob/master/src/index.js#L26, the split is being done incorrectly and you end up with the wrong multiaddr - /dns4/strawberry.i.ipfs.io/wss/ipfs, note the dangling /ipfs. We should use decapsulate instead I think, I'll fix and PR in a sec.

[daviddias]

@dryajov awesome, great catch! Makes sense :)

[dryajov]

here is the PR for railing - libp2p/js-libp2p-bootstrap#53

[dryajov]

@diasdavid there is one more issue with websockets. I'm looking into it right now, should be able to fix it quickly.

[dryajov]

here is the PR to address the remaining issues in websockets - libp2p/js-libp2p-websockets#57

[dryajov]

@diasdavid

These two PRs are needed to get things working

• fix: multiaddr parsing  libp2p/js-libp2p-bootstrap#53
• fix: address parsing libp2p/js-libp2p-websockets#57

[daviddias]

awesome work @dryajov ! :)

[daviddias]

Update

• Adjust the nginx configs so that we can dial on websockets
• Fix the transfer-files example (Chrome is not giving FilesList as an array anymore)
• Transfer a file from a browser node to a gateway
• Diagnose why the Websocket connections to the gateways die up so quickly

[daviddias]

bring this up

@lgierth did you had the chance to configure ws timeouts on the bootstrapers?

[daviddias]

I'm seeing an improve on stability of the websocket bootrapers, in fact, I don't see any going down anymore :)

Just pushed the last finishing touches to merge this PR, will merge once CI is green

[daviddias]

There are some failing pubsub tests but it seems they also fail on master, which tells me some dependency was updated which is causing a breaking change.

All tests pass locally and on Circle, which makes it especially hard to debug.

With regards to this PR, all of the respective functionality is sound.