feat: *.localhost subdomain gateway support with http proxy #853

lidel · 2020-03-22T23:22:23Z

This PR adds support for subdomain gateways introduced in go-ipfs v0.5.0-dev (ipfs/kubo#6096), specifically, one running at *.localhost subdomains.

Um.. what?

TL;DR we will change the URL present in the address bar when user loads content from the local node (ipfs-desktop or standalone go-ipfs):

http://127.0.0.1:8080/ipns/en.wikipedia-on-ipfs.org/wiki/

to:

http://en.wikipedia-on-ipfs.org.ipns.localhost:8080/ipns/wiki/

Why?

localhost subdomain gateway provides unique Origin per content root
- this is extremely important, see
  - historical discussion about URI and Origin
    - Tackle identifying origins with (or without?) dweb: paths in-web-browsers#6
    - https://github.com/ipfs/in-web-browsers/issues?q=label%3AOrigin
  - in-depth analysis of how to unify the way we do redirects without compromising website isolation Redirect of CID-in-subdomain and DNSLink websites #667
better UX on DNSLink websites
DNSLink websites get proper Origin isolation per content root (Redirect of CID-in-subdomain and DNSLink websites #667)
we can enable redirect of subdomain gateways (CID as a Subdomain in-web-browsers#89, Redirect of CID-in-subdomain and DNSLink websites #667)

How?

Disclaimer: below is a draft of what may end up a blogpost after go-ipfs 0.5 ships. WIP.

They key challenge was to ensure *.localhost DNS names resolve to 127.0.0.1 on all platforms, specifically Linux distribution which ship with very basic DNS resolver that returns "not found" when trying to resolve localhost subdomains.

Browsers are moving in the direction of hardcoding localhost names to loopback interface, but we are not there yet. We need to solve this problem with tools at hand.

We do that by setting up HTTP Gateway port of local go-ipfs to act as HTTP Proxy. This removes DNS lookup step from the browser, and go-ipfs ships with implicit support for subdomain gateway when request comes with "Host: .ipfs.localhost:8080" or
similar.

We register HTTP proxy using Firefox and Chromium-specific APIs, but the end result is the same. When enables, default gateway uses 'localhost' hostname (subdomain gateway) instead of '127.0.0.1' (path gateway) and every path-pased request gets redirected to subdomain by go-ipfs itself, which decreases complexity on browser extension side.

TODO

This is work in progress:

caveats

match pattern used for registering proxy handler in Firefox uses localhost name without port. This is by design, match patterns do not seem to support ports and are executed on all of them. We need to check port inside of handler, and return direct proxy route if it does not match
proxy situation is temporary and will change over time
- Chromium does not need proxy, they hardcode localhost and *.localhost to loopback IPs
- Firefox needs proxy until https://bugzilla.mozilla.org/show_bug.cgi?id=1220810 lands

Migration to CIDv1 (default base32) https://github.com/ipfs/ipfs/issues/337
CID as a subdomain CID as a Subdomain in-web-browsers#89
Redirect of CID-in-subdomain and DNSLink websites: Redirect of CID-in-subdomain and DNSLink websites #667
Subdomain Gateway Subdomain Gateway kubo#6498
- Act as HTTP PROXY for http://<cidv1b32>.ipfs.localhost: Act as HTTP PROXY for http://<cidv1b32>.ipfs.localhost kubo#5982
- PR: subdomain gateway via http proxy in go-ipfs: feat: gateway subdomains + http proxy mode kubo#6096
Closes Mixed Content Warning when using localhost Gateway #328
Closes Use Web UI version from IPFS Node #736

lidel · 2020-04-02T15:05:48Z

Surfacing some notes on additional fixes that had to be applies to keep subresource redirects working on HTTPS websites:

Firefox 74 does not mark *.localhost subdomains as Secure Context yet
(https://bugzilla.mozilla.org/show_bug.cgi?id=1220810#c23) so we can't redirect
there when we have IPFS resource embedded on HTTPS page (eg. image loaded from
a public gateway) because that would cause mixed-content warning and
subresource would fail to load. Given the fact that localhost/ipfs/* provided
by go-ipfs 0.5+ returns a redirect to *.ipfs.localhost subdomain we need to
check requests for subresources, and manually replace localhost hostname with
127.0.0.1 (IP is hardcoded as Secure Context in Firefox). The need for this
workaround can be revisited when Firefox closes mentioned bug.

Chromium 80 seems to force HTTPS in the final URL (after all redirects) so
https://*.localhost fails. This needs additional research (could be a bug in
Chromium). For now we reuse the same workaround as Firefox.

To unify use of 127.0.0.1 and localhost in address bar (eg. when user opens an
image in a new tab etc) when Subdomain Proxy is enabled we normalize address
bar requests made to the local gateway and replace raw IP with 'localhost'
hostname to take advantage of subdomain redirect provided by go-ipfs >= 0.5

In other news:

Added a bunch of tests, all green now.
In the meantime I noticed Chrome does not require proxy mode because they already hardcode localhost to raw IP. I am considering removing proxy permission from Chromium package as it may trigger yet another manual review at Chrome Web Store.
- Will keep the code commented out in case we need to enable it in the future for some reason.

lidel · 2020-04-03T00:50:14Z

add-on/_locales/en/messages.json

+  "option_useSubdomains_title": {
+    "message": "Use Subdomains",
+    "description": "An option title on the Preferences screen (option_useSubdomains_title)"
+  },
+  "option_useSubdomains_description": {
+    "message": "Isolate content roots from each other by loading them from subdomains at *.localhost and creating a unique Origin for each CID, IPNS or DNSLink record. Requires a local go-ipfs 0.5.0 or later.",
+    "description": "An option description on the Preferences screen (option_useSubdomains_description)"
+  },


On Preferences screen the above labels look like this:

@jessicaschilling @autonome I suspect this could be phrased in a more human way, but this is the best I could do for next Beta release. Suggestions welcome before we ship to Stable.

This is a description of a toggle that lets people to switch to old path-based gateway on 127.0.0.1 if they choose to do so for some reason. (disabling it stops HTTP proxy and swaps Custom Gateway URL to use raw IP)

lidel · 2020-04-03T01:14:36Z

Shipped this in a new Beta: v2.10.0.902. Will play with it during the weekend.

I do not expect big changes apart from tweaking labels mentioned in previous comment.

This adds support for subdomain gateways introduced in go-ipfs v0.5.0, specifically, one running at *.localhost subdomains They key challenge was to ensure *.localhost DNS names resolve to 127.0.0.1 on all platforms. We do that by setting up HTTP Gateway port of local go-ipfs to act as HTTP Proxy. This removes DNS lookup step from the browser, and go-ipfs ships with implicit support for subdomain gateway when request comes with "Host: <cid>.ipfs.localhost:8080" or similar. We register HTTP proxy using Firefox and Chromium-specific APIs, but the end result is the same. When enableid, default gateway uses 'localhost' hostname (subdomain gateway) instead of '127.0.0.1' (path gateway) and every path-pased request gets redirected to subdomain by go-ipfs itself, which decreases complexity on browser extension side. By default, extension will now redirect from public subdomain gateways such as dweb.link: https://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq.ipfs.dweb.link/wiki/ to the local one: http://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq.ipfs.localhost:8080/wiki/ Redirect to gateway logic was path based, this is wip surgical refactor to unify subdomain and path handling, hopefully decreasing maintenance buden going forward This is work in progress. Fixes will be applied in separate commits.

This avoids ibreaking IPFS content paths by converting them to URI params

It should not block the init()

Firefox 74 does not mark *.localhost subdomains as Secure Context yet (https://bugzilla.mozilla.org/show_bug.cgi?id=1220810#c23) so we can't redirect there when we have IPFS resource embedded on HTTPS page (eg. image loaded from a public gateway) because that would cause mixed-content warning and subresource would fail to load. Given the fact that localhost/ipfs/* provided by go-ipfs 0.5+ returns a redirect to *.ipfs.localhost subdomain we need to check requests for subresources, and manually replace 'localhost' hostname with '127.0.0.1' (IP is hardcoded as Secure Context in Firefox). The need for this workaround can be revisited when Firefox closes mentioned bug. Chromium 80 seems to force HTTPS in the final URL (after all redirects) so https://*.localhost fails. This needs additional research (could be a bug in Chromium). For now we reuse the same workaround as Firefox. To unify use of 127.0.0.1 and localhost in address bar (eg. when user opens an image in a new tab etc) when Subdomain Proxy is enabled we normalize address bar requests made to the local gateway and replace raw IP with 'localhost' hostname to take advantage of subdomain redirect provided by go-ipfs >= 0.5

Chromium 80 already hardcodes localhost to loopback IPs, and we don't want to add unnecessary permission as it may cause us fail review at Chrome Web Store. I also renamed the setting as its about overall Subdomain support on localhost, even when HTTP proxy is not used.

lidel · 2020-04-04T00:00:14Z

test/functional/lib/ipfs-request-workarounds.test.js

+    it('should keep the "Origin: null" header ', async function () {
+      // Presence of Origin header is important as it protects API from XSS via sandboxed iframe
+      // NOTE: Chromium <72 was setting this header in requests sent by browser extension,
+      // but they fixed it since then.


Removed unused code related to loading webui from gateway port without setting CORS and added test to ensure Companion does not remove Origin: null

cc ipfs/go-ipfs-cmds#190

This restores the proper way of opening webui in version provided by IPFS node. Context: Before subdomain gateway support was added, we loaded webui from gateway port. Why? API port has a hardcoded list of whitelisted webui versions and it is not possible to load non-whitelisted CID when new webui was released. To enable API access from webui loaded via Gateway port, the Companion extension removed Origin header for requests coming from its background page. This let us avoid the need for manual CORS setup, but was seen in the diff, was pretty complex process. Webui is stable now, so to decrease maintenance burden we move away from that complexity and just load version whitelisted on API port. What if someone wants to run newest webui? They can now load it from webui.ipfs.io.ipns.localhost:8080 (whitelist API access from that specific Origin by appending it to API.HTTPHeaders.Access-Control-Allow-Origin in go-ipfs config) Closes #736

lidel · 2020-04-05T17:08:19Z

After feedback from two beta releases I feel this is stable enough to merge to master and ship beta+stable.

We need this not only for subdomains, but also to ensure POST-only API (ipfs/kubo#7097) scheduled for go-ipfs 0.5 does break DNSLink lookup and ipfs-webui.

lidel · 2020-04-05T22:07:49Z

Released to Beta: v2.11.0.904
Scheduled to ship to stable as v2.11.0

Subdomains are supported out of the box. If you have go-ipfs 0.5.0-dev you should see interesting change in the address bar:

Backward-compatible

Various edge cases are handled properly now in a way that is backward-compatible. Users of local gateways without subdomains won't see any regressions, but will get a nicer address bar with localhost instead of 127.0.0.1.

Subdomain preference

Just in case, it is possible to restore old school path-based gateway via Preferences screen:

It will switch Custom Gateway URL from localhost to 127.0.0.1, disabling the Subdomain redirects made by go-ipfs 0.5.

Examples

There will be a longer write up, for now enjoy some test links:

Seamless CIDv0 → CIDv1 upgrade:
http://localhost:8080/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR
→ http://bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi.ipfs.localhost:8080/
Subdomain gateways are now redirected to localhost one:
https://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq.ipfs.dweb.link
→ http://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq.ipfs.localhost:8080/wiki/
...and finally, DNSLink is probably the coolest one:
https://en.wikipedia-on-ipfs.org/wiki/
→ http://en.wikipedia-on-ipfs.org.ipns.localhost:8080/wiki/

Testing with Docker

Docker makes it easy to test with go-ipfs v0.5.0-dev without changing your local setup.
To start an ephemeral instance run:

$ docker pull ipfs/go-ipfs:master-latest
$ docker run --rm -it --net=host ipfs/go-ipfs:master-latest

lidel added 2 commits March 18, 2020 23:59

chore: web-ext update

51671da

chore: yarn@1.22.4

4bf9d09

This was referenced Mar 22, 2020

feat: support subdomains in isIPFS.url(url) ipfs-shipyard/is-ipfs#32

Merged

fix: default to https when recovering .eth #847

Merged

lidel force-pushed the feat/subdomain-proxy-support branch 3 times, most recently from 43e6494 to 1d922de Compare March 23, 2020 12:31

This was referenced Mar 24, 2020

Act as HTTP PROXY for http://<cidv1b32>.ipfs.localhost ipfs/kubo#5982

Closed

HTTPS Everywhere in EASE mode blocks redirect to http://*.localhost #855

Open

lidel force-pushed the feat/subdomain-proxy-support branch 2 times, most recently from e3ffed5 to f948443 Compare March 27, 2020 14:42

lidel force-pushed the feat/subdomain-proxy-support branch from a4c9dce to bfa59c3 Compare April 3, 2020 00:33

lidel commented Apr 3, 2020

View reviewed changes

This was referenced Apr 3, 2020

feat: default local gateway URLs to 'localhost' #780

Closed

refactor: use Web UI provided by IPFS node #737

Closed

ipfs.dag.get fails with a map error only in IPFS Companion #856

Closed

lidel added 8 commits April 3, 2020 22:48

fix: toggle per website on <fqdn>.ipfs.localhost

e5889f2

fix: proxy only the gateway port

4ce869e

fix: decode content paths with decodeURI

1acf899

This avoids ibreaking IPFS content paths by converting them to URI params

fix: disable http-proxy when extension is inactive

2a6c67e

fix: async setApiStatusUpdateInterval

207fd76

It should not block the init()

lidel force-pushed the feat/subdomain-proxy-support branch from bfa59c3 to eb66dfa Compare April 3, 2020 20:50

This was referenced Apr 3, 2020

Opt-in for loading latest webui from webui.ipfs.io.ipns.localhost #860

Closed

Use Web UI version from IPFS Node #736

Closed

lidel commented Apr 4, 2020

View reviewed changes

lidel force-pushed the feat/subdomain-proxy-support branch 2 times, most recently from a94e46b to c3a5d93 Compare April 5, 2020 15:45

lidel force-pushed the feat/subdomain-proxy-support branch from c3a5d93 to 4a81bf1 Compare April 5, 2020 15:52

lidel marked this pull request as ready for review April 5, 2020 15:52

test: findDNSLinkHostname without match

5bce2a2

lidel force-pushed the feat/subdomain-proxy-support branch from e174d74 to 5bce2a2 Compare April 5, 2020 17:09

lidel merged commit 5cc81be into master Apr 5, 2020

lidel deleted the feat/subdomain-proxy-support branch April 5, 2020 17:16

lidel mentioned this pull request Apr 16, 2020

IPNS: should work with case-insensitive identifiers (Base32) ipfs/kubo#5287

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: *.localhost subdomain gateway support with http proxy #853

feat: *.localhost subdomain gateway support with http proxy #853

lidel commented Mar 22, 2020 •

edited

Loading

lidel commented Apr 2, 2020

lidel Apr 3, 2020

lidel commented Apr 3, 2020

lidel Apr 4, 2020

lidel commented Apr 5, 2020

lidel commented Apr 5, 2020 •

edited

Loading

feat: *.localhost subdomain gateway support with http proxy #853

feat: *.localhost subdomain gateway support with http proxy #853

Conversation

lidel commented Mar 22, 2020 • edited Loading

Um.. what?

Why?

How?

TODO

caveats

Related

lidel commented Apr 2, 2020

lidel Apr 3, 2020

Choose a reason for hiding this comment

lidel commented Apr 3, 2020

lidel Apr 4, 2020

Choose a reason for hiding this comment

lidel commented Apr 5, 2020

lidel commented Apr 5, 2020 • edited Loading

Backward-compatible

Subdomain preference

Examples

Testing with Docker

lidel commented Mar 22, 2020 •

edited

Loading

lidel commented Apr 5, 2020 •

edited

Loading