Act as HTTP PROXY for http://<cidv1b32>.ipfs.localhost

[lidel]

Version information:

0.4.18

Type:

feature

Description:

Problem

CID-in-subdomain creates Origin-based isolation if website is loaded from public gateway:

• http://bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy.ipfs.dweb.link

..however we continue to have a single Origin for all websites loaded from local gateway:

• http://127.0.0.1:8080/ipfs/bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy/

This means use of local gateway decreases some of security guarantees web developers are used to, and removes some incentives to run local go-ipfs (and redirect to it via ipfs-companion).

Solution

What if go-ipfs could act as HTTP PROXY for requests made to http://*.ipfs.*/ and http://*.ipns.*/ ?

Having that, IPFS Companion could automatically set Gateway port as PROXY for requests to http://*.ipfs.localhost, which would create very nice URLs for local gateway, solving the Origin problem for local go-ipfs 👌

Initial look at technical feasibility

My hope is that we could keep changes to minimum and reuse Gateway port to also act as HTTP PROXY for requests to *.ipfs.localhost.

In other words, in addition to regular HTTP GET:

$  curl -v 'http://127.0.0.1:8080/ipfs/bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy/'
> GET /ipfs/bafybeie5gq4jxvzmsym6hjlwxej4rwdoxt7wadqvmmwbqi7r27fclha2va HTTP/1.1
> Host: 127.0.0.1:8080
< HTTP/1.1 200 OK

Gateway port should also respond to PROXY requests:

$ curl -v --proxy 'http://127.0.0.1:8080' 'http://bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy.ipfs.localhost'
> GET http://bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy.ipfs.localhost/ HTTP/1.1
> Host: bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy.ipfs.localhost

AFAIK all HTTP PROXY needs to do is to support HTTP CONNECT Method (but I think some non-https proxies work even without it):

CONNECT bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy.ipfs.localhost:80 HTTP/1.1 
Host: bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy.ipfs.localhost:80 

After looking at golang's net/http, my initial intuition says it should be possible to detect such requests early via something like:

http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    if r.Method == http.MethodConnect {
        handleProxyRequest(w, r)
    } else {
        handleRegularRequest(w, r)
    }
})

Would love to get some feedback on this idea

I feel this is a huge win for both security and UX, and should be done before we switch to CIDv1 in base32 by default.

As a sidenote: this technique opens us a venue for future support of alternative naming systems such as ENS (*.eth.localhost) or Namecoin (*.bit.localhost).

Refs

• Migration to CIDv1 (default base32) https://github.com/ipfs/ipfs/issues/337
  • go-ipfs specific Migration towards base32 CIDv1 #5358
• Why CID in Subdomain? CID as a Subdomain in-web-browsers#89
• Origin problem for local gateway at go-ipfs
  • Redirect of CID-in-subdomain and DNSLink websites: Redirect of CID-in-subdomain and DNSLink websites ipfs-companion#667
  • Vanity hostnames for localhost HTTP gateway: Hostnames for localhost HTTP gateway in-web-browsers#109

[momack2]

@eingenito @Stebalien curious if you've gotten a chance to look at this and the related companion issue: ipfs/ipfs-companion#678

[ntninja]

Just stumbled on this and wanted to point out that you should not use *.local for this as that is used by MDNS. (A device in the local network with name ipfs would conflict with your proposed use of the name for instance.)
Use *.localhost instead, that's what it exists for.

Otherwise great proposal, I love the idea of this. 👍

[da2x]

.localhost is indeed the correct special-use TLD.
https://tools.ietf.org/html/rfc6761
https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml

[Stebalien]

What if go-ipfs could act as HTTP PROXY for requests made to http://*.ipfs.*/ and http://*.ipns.*/ ?

IMO, redirecting all domains with an ipns and/or ipfs subdomain isn't really the best idea. If we're already using a proxy, what if just use ipfs.dweb.link everywhere? That will work with or without the proxy.

[lidel]

IMO, redirecting all domains with an ipns and/or ipfs subdomain isn't really the best idea

@Stebalien Is there a specific reason why you feel this way?
My take is that Companion will do validation before subdomain redirect, just like it does right now for /ipfs/<cid> paths. Redirect will happen only if:

• <cid>.ipfs.* – <cid> is a valid CID
• <fqdn>.ipns.* – the <fqdn> has a valid DNS TXT record with DNSLink (or is a valid PeerID)

go-ipfs would do the same validation on its side, and return HTTP 403 if URL is not valid.

Unfortunately use of *.ipfs.dweb.link for localhost gateway would introduce serious issues:

• sec: http proxy gateway won't be able to support HTTPS for arbitrary domains
  • if we try to ship private cert for *.ipfs.dweb.link with go-ipfs it will get revoked by CA
• ux: how can user know if they use local vs public gateway, if the hostname remains the same?

As noted by @da2x, .localhost is a good-enough choice for local gateway that can work Today.
(in the future we could look into registering Special-Use *.dweb, similar to *.onion)

[Stebalien]

Is there a specific reason why you feel this way?

Well, I'm not all that happy with hijacking all /ipfs/Cid paths either but at least we're redirecting in that case.

However, in the proxy case, this would be a pretty big security issue: I'd be able to run arbitrary javascript on a subdomain of anyone's website. This can be used to mess with cookies (subdomains can set cookies that apply to the root domain).

Really, we need to:

1. Use a specific domain.
2. Add that domain to the "public suffix" list: https://publicsuffix.org/

Alternatively, we could always redirect to Qm.ipfs.localhost (although that doesn't completely fix the super-cookie issue).

---

sec: http proxy gateway won't be able to support HTTPS for arbitrary domains

Hm. Yeah, I forgot HTTP proxies can't do anything about this. What if we redirect? That is, what if the companion redirects https to http (when proxying is enabled)? Unfortunately, that has some UX issues (looks insecure).

ux: how can user know if they use local vs public gateway, if the hostname remains the same?

This is actually why I want this. The current redirect is annoying from a UX standpoint as localhost links won't "just work". If we're worried about telling the user that we're serving the resource from the local IPFS node, we can put something in the urlbar.

[lidel]

@Stebalien I absolutely agree we shouldn't spoof domains of other people. If subdomain passes validation we would redirect to a subdomain under our specific domain before serving the content, just like we redirect paths. (The purpose of HTTP Proxy here is to provide a vanity hostname for local gateway in a way that works across all platforms).

This specific domain should be configurable via ipfs config, to accommodate more than localhost use case. For example, it could enable people to set up their own public gateways capable of supporting CID in subdomain without any additional URL conversion done at Nginx (starting with our dweb.link: ipfs/infra#81).

As for picking a specific domain, .localhost seems to be the best candidate right now, but we are not forced to use it.

Especially if we are unable to convince browser vendors to make .localhost a Securi Context (ipfs/ipfs-companion#667 (comment)) we may just as well choose to go with .dweb.link (keep the hostname of public gateway and just redirect https to http, as you suggested), and/or in parallel work on RFC making .dweb a thing (taking advantage of .onion secure context precedent)

I'm good with either, as long we move in the right direction.

• https://<cid>.ipfs.foobar.tld → http://<cid>.ipfs.localhost
• https://<cid>.ipfs.foobar.tld → http://<cid>.ipfs.dweb.link
• https://<cid>.ipfs.foobar.tld → http://<cid>.ipfs.dweb

ps. We are tracking adding dweb.link to publicsuffix.org in ipfs/infra#83 (comment)

[Stebalien]

So would the companion redirect other domains to, e.g., .ipfs.localhost? SGTM.

This specific domain should be configurable via ipfs config, to accommodate more than localhost use case.

That's basically what I was planning on doing. My objection was to:

What if go-ipfs could act as HTTP PROXY for requests made to http://*.ipfs.*/ and http://*.ipns.*/ ?

(i.e., having go-ipfs act as a blind proxy for these addresses).

[lidel]

So would the companion redirect other domains to, e.g., .ipfs.localhost? SGTM.

Correct. 👍

My objection was to:

What if go-ipfs could act as HTTP PROXY for requests made to http://*.ipfs.*/ and http://*.ipns.*/ ?

(i.e., having go-ipfs act as a blind proxy for these addresses).

It's my fault, I made a mental shortcut while explaining the intent.

Just like you said, we will not do a blind proxy, browser extension will redirect to a specific domain. Proxy will only work for domain(s) specified in config. Everything else requested in proxy mode will get HTTP 400 Bad Request. Regular requests will work without any change.

[Stebalien]

SGTM.

[ilyaigpetrov]

If user is required to configure proxy for his browser then he won't be able to use any other proxy (naked proxy or via a browser extension). E.g., user won't be able to use proxies/extensions aimed for bypassing censorship.

What would you say about another approach -- installing or configuring caching dns server on a client like, e.g., Dnsmasq? Is this approach worth it?

[lidel]

@ilyaigpetrov HTTP proxy does not need to be global: browser extension is able to set proxy only for *.ipfs.localhost (chrome.proxy and browse.proxy APIs). The main benefit here is that browser extension is the only thing you need. Installing local DNS server also does the trick if you don't mind running on :80 or having :8080 in URL, but then value added on top of #6498 is pretty small. Custom DNS is yet another thing to run and setup, pretty invasive and just not practical for regular users.

[ilyaigpetrov]

@lidel, as far as I know proxy setting in Chromium may be set only for one extension, in the settings of your Chromium you will see "Proxy is controlled by <extension name>". You can't set proxy in one extension for a few sites and also set proxy for other sites in another extension. Even if you use PAC-script then it may be controlled only by one extension. Please, let me know if you need a proof of my words and I will make two extensions that try to control proxy settings at the same time for different sites.

I don't like the idea of browser extension because, well, there is possibility that I will want to access some file on ipns without any browser and ipfs mounts may be unavailable because of absence of the fuse library. I will try to come up with some more realistic use case for it later.

And I also want to ask you: what if we will provide user with both approaches: proxy and local dns -- and user will choose what he wants.

[ilyaigpetrov]

@lidel

CONFLICTS: SwitchyOmega will conflict with other extensions trying to control the proxy settings. Such conflicts are caused by the design of the Chrome browser and thus cannot be avoided.

From webstore.

[lidel]

Extensions running in Firefox have access to much better API called browser.proxy.onRequest. Global proxy control in Chromium is unfortunate, but does not fully block this use case: user can set the proxy in the other extension.

Note the goal of proxy mode is to improve UX in scenarios where DNS solution is not feasible.
That is all.

For most users basic support for subdomain gateway (#6498) will be enough, it will just have bit uglier URLs that include custom port. In systems where *.localhost does not resolve out of the box, custom DNS can be set up, just like you suggested.

[lidel]

HTTP proxy support in ipfs-companion (for *.ipfs.localhost) is added in ipfs/ipfs-companion#853