Recognize IPFS gateways on arbitrary websites

[Kubuxu]

Currently looking for dnslink is a way to check if site is hosted on IPFS via gateway.

I propose that we use other TXT entry specifying that this domain should be served under IPFS/NS.

It could be TXT IPFS-Gateway=true. Simpler nature of this entry means that it can have much longer TTL and can be cached for much longer time.

As workaround of lack of TXT record support in browsers we would need API endpoint for checking it.

Also it would be great if we could have agree on some sane default TTL in case of dnslink and this entry are missing thus increasing cache hits even more.

EDIT: Also as dnslink gets support for _dnslink. prefix something similar might be done in case of this also.

[lidel]

We already have a handy DNS query API (provided by go-ipfs) which returns dnslink value:

GET http://localhost:5001/api/v0/dns/ipfs.io

{"Path":"/ipfs/QmSrk8wCZ9oUgSCxJq4TXyJnPBYfoLDpKLGn7hLPNzauVX"}

and for site without dnslink in TXT record:

GET http://localhost:5001/api/v0/dns/google.com

{"Message":"could not resolve name.","Code":0}

Quick idea/brain dump:

Perhaps if TXT record contains a httpmirror keyword the response could additionally include something like {"Path": "(..)", "HTTPMirror": true}:

ipfs.io. 76 IN TXT "httpmirror;dnslink=/ipfs/QmSrk8wCZ9oUgSCxJq4TXyJnPBYfoLDpKLGn7hLPNzauVX"

It would be (kinda) reverse of dnslink: indicate that queried host provides real IPFS resources under /ipfs/ and /ipns/ which can be safely loaded from IPFS instead of HTTP.

[Kubuxu]

Point of my request is to provide some tag that could have very long TTL and could be cached. TTL of dnslink is limited as you need update it if you want to change the site (is use 60s TTL on most my dnslinks). Latency of DNS resolution can be anything from 100ms to +1s.

As decision if site is to be shown normally or redirected to IPFS should be taken quickly it would be best if we could achieve high cache hit rate for this operation.

Also it would allow sites that host gateways for themselves (or use ipfs.io under the hood) to say that it is safe to redirect it to other gateway.

[lidel]

Ok, how about a separate TXT record with long TTL?

ipfs.pics. 246696 IN TXT "ipfsgateway=public"

Related idea: this new TXT record could additionally provide a hint if this is a public IPFS gateway or if it is restricted to a limited subset of whitelisted resources.

Use case for restricted gateway: a person/organization do not want to be worried about someone else uploading undesired content to ipfs and linking to it using company.com hostname (I assume this kind of gateway would return HTTP 404 for all non-whitelisted IPFS resources).

DNS hint would simply indicate that although some resources can be loaded from IPFS, it should not be used as a public gateway.

[Kubuxu]

I like the idea of restricted.
The hint would be placed along side dnslink record so in the root of selected record or _dnslink. sub-record.

Also do we want, and how, to cover sites like http://ipfs.pics that are gateways but the root site has to be accessed directly? (It might also happen that they won't work if we redirect gateway traffic to local gateway as for CORS and other magic).

EDIT: If site defines dnslink and ipfsgateway=public, we should be safe to redirect it fully to local gateway. There are still edge cases (local storage, cookies for API calls to original site), but I think that defining both of those variables should make site safe to move into local gateway.

EDIT2: Sites that want to stay under their own domain should just not define ipfsgateway but there might be better solution.

EDIT3: There are really three cases:

1. Site that wants to fully move to local gateway.
2. Site that wants its root to stay on remote host but /ipfs/ and /ipns can be moved to local gateway.
3. Site that is gateway but wants to fully stay under its own domain. (XHR, cookies, localStorage and so on).

[jbenet]

I propose that we use other TXT entry specifying that this domain should be served under IPFS/NS.

i don't understand this-- what does this mean?

As decision if site is to be shown normally or redirected to IPFS should be taken quickly it would be best if we could achieve high cache hit rate for this operation.

Why isn't this figured out from the dnslink itself? If you have a dnslink with an /ip{fn}s link, you can use ipfs to resolve it just fine. Browsers today are unaware of ipfs so they need to use a gateway. but custom apps don't.

---

Related idea: this new TXT record could additionally provide a hint if this is a public IPFS gateway or if it is restricted to a limited subset of whitelisted resources.

Why should this be in DNS and not under a route in the gateway, such as /version ?

---

Site that wants to fully move to local gateway.

All /ipfs and /ipns links at a gatway should be able to move to local gateway, always. If we want some standard way to check for a gateway (eg /version)

Site that wants its root to stay on remote host but /ipfs/ and /ipns can be moved to local gateway.

Natural consequence of the above.

Site that is gateway but wants to fully stay under its own domain. (XHR, cookies, localStorage and so on).

No if the site has an ipfs gateway and uses ipfs links /ipfs/..., /ipns/... these can always be moved to a local gateway. XHR/cookies/localStorage are irrelevant here, they SHOULD be handled through the ipfs or ipns root hash as the "suborigin". (once suborigins land -- https://www.chromium.org/developers/design-documents/per-page-suborigins, already implemented)

[Kubuxu]

This DNS record would say if site wants to move under local/client gateway and how.

There are many reasons for including something like this, I will try sum it up here:

1. Currently FF add-on faces problem of how and which sites redirect to local gateway and which not. This decision has to be taken before the HTTP connection start, which means asking IPFS if that site is hosted under IPFS. dnslinks with low TTLs are not suitable for such resolutions, they would cause delays which can be quite significant (depending on many factors). Also non-hit to dnslink is not cached. I've proposed that lack of flag could be also cached so work of the gateway redirector does not slow down the rest of browser. We have to choose IPFS or HTTP and we have to choose it fast because user waits.
2. It would allow sites to work via domain name for add-on users even if backing gateway died.
3. It can be figured out form dnslink but not completely. You can't know from dnslink if site wants to move fully under local gateway or not.
4. Suborgins haven't landed yet (at least not on any FF, I am running alpha stream), and it might take long time to finalize this spec and even longer to be implemented in most browsers.
5. XHR/Origin makes difference if site wants to call home or other non-IPFS service. We want to allow smooth transition and not say: "You have to forget about what you already done and do it again this way". (Case 2)
6. ipfs.pics cannot be moved under local gateway because it calls home, nor can any other site that wants to use localStorage (as there are no Suborgins yet). (Case 2)
7. If site stays under its domain name it can't execute XHR requests to local gateway, they are just blocked (we will be investigating that, Sites under public gateways list can't make XHR request ipfs-companion#45). (Case 3)
8. Some day situation might be reverse and sites will ask not to be redirected to IPFS but to stay under HTTP.

[jbenet]

1. Currently FF add-on faces problem of how and which sites redirect to local gateway and which not. This decision has to be taken before the HTTP connection start, which means asking IPFS if that site is hosted under IPFS. dnslinks with low TTLs are not suitable for such resolutions, they would cause delays which can be quite significant (depending on many factors). Also non-hit to dnslink is not cached. I've proposed that lack of flag could be also cached so work of the gateway redirector does not slow down the rest of browser. We have to choose IPFS or HTTP and we have to choose it fast because user waits.

Do you mean domains like foo.com that setup a dnslink dns TXT record and an A record to our gateway?

for normal sites, i'd just assume http until the A record points to known ipfs gateways. but i get that it may not work as well.

this could be something you opt to do for the extension only but is not meant to stick around or be broader. I'm very wary of explosion of asking users to setup a bunch of records.

1. It would allow sites to work via domain name for add-on users even if backing gateway died.

Could the use of the backing gateway be detected without pre-checking dns yourslef? (i.e. use the result of the A or CNAME records the browser gets on its own)

1. It can be figured out form dnslink but not completely. You can't know from dnslink if site wants to move fully under local gateway or not.

right

1. Suborgins haven't landed yet (at least not on any FF, I am running alpha stream), and it might take long time to finalize this spec and even longer to be implemented in most browsers.

right (but keep the scope in mind when creating tools + standard practices)

1. XHR/Origin makes difference if site wants to call home or other non-IPFS service. We want to allow smooth transition and not say: "You have to forget about what you already done and do it again this way". (Case 2)

which site wants to call to which home. If a site at a /ipfs/... object is trying to hit /... some other non-ipfs path, that's a website creator error.

1. ipfs.pics cannot be moved under local gateway because it calls home, nor can any other site that wants to use localStorage (as there are no Suborgins yet). (Case 2)

I don't think it does. There's a misunderstanding here:

• ipfs.pics/ipfs/... is resolved by ipfs gateway. can be local.
• ipfs.pics/ipns/... is resolved by ipfs gateway. can be local.
• ipfs.pics/* (everything else) IS NOT ipfs content and SHOULD NOT be resolved by ipfs. this is a regular webapp, not an ipfs webapp (AFAIK).

1. If site stays under its domain name it can't execute XHR requests to local gateway, they are just blocked (we will be investigating that, Sites under public gateways list can't make XHR request ipfs-companion#45). (Case 3)

True. this is one reson in support of gateway having friendly CORS (ipfs.io has it).

1. Some day situation might be reverse and sites will ask not to be redirected to IPFS but to stay under HTTP.

I'm not convinced yet. So far, I think this can be figured out from the otherwise available information. I do not want to make lots of tools have to follow this until i'm sure it's necessary.

[Kubuxu]

Do you mean domains like foo.com that setup a dnslink dns TXT record and an A record to our gateway?

I mean that site example.com has dnslink but its own A records of gateway. We can't run dnslink lookup on every site visited by user, every time he refreshes it but we want to redirect all off them to local gateway if it is possible.

True. this is one reson in support of gateway having friendly CORS (ipfs.io has it).

The XHR problem is with site staying under its own domain but under the ipfs.io A records when add-on redirects /ipfs/... to local gateway.

ipfs.pics/ipfs/... is resolved by ipfs gateway. can be local.
ipfs.pics/ipns/... is resolved by ipfs gateway. can be local.

Problem is also that you can't redirect all /ipfs/... to local gateway. Example: https://github.com/ipfs/notes/. FF add-on has option to redirect all /ipfs/... calls to local gateway but it breaks other sites. That is why site could specify it want to be regular webapp but wants to have /ipfs/... and /ipns/... redirected to local gateway which currently isn't possible and makes site like ipfs.pics almost not distributed (unless we explicitly specify it in the add-on configuration which can't be done for all sites).

[jbenet]

@Kubuxu curious, how would an entry in /etc/hosts mapping ipfs.io to localhost help here? (port may still be a problem :/)

also, @lidel do extensions have access to the address bar in firefox?

[Kubuxu]

Port still would be a problem (you can't bind 80 from normal user) but redirecting ipfs.io isn't an issue (it is already done, currently work is in making the bar show: fs:/ipfs/...).

Problem is with discovering all of those sites that might be using ipfs.ip or other gateways under the hood (it was said in an other thread that detection of IPFS after first HTTP response is too late).

[lidel]

@jbenet:

do extensions have access to the address bar in firefox?

Generally Firefox is in the process of “refactoring” addon ecosystem. The only “future-proof” SDK does not provide high level API for accessing this part of GUI (we can only add a toolbar button etc).

It is possible to access all GUI elements via low level APIs called XUL/XPCOM but it will be deprecated in 12-18 months and I would avoid any new work that requires it.

In future there will be a new SDK called WebExtensions and I plan to migrate the addon to it as soon as it is mature enough (not supported by release FF yet). It is a mess at the moment :)

---

Anyway, back to the topic ⏫

To put the problem and idea in simpler words:

• What is already available is dnslink, which is an indicator that entire website can be loaded from IPFS
• What is missing, is a middle-ground:

  • Some sites may use IPFS only as a repository for some of its assets.
    Those assets are usually loaded by reverse-proxy (Nginx/Apache) from IPFS gateway running in the backend.
    For example http://bin.kubuxu.ovh loads snippet content from http://bin.kubuxu.ovh/ipfs/QmRqoVq1G9pL7hyLB2EGi5mndRTQv7tKfFXPRUsSbpHqfK

    So the general idea is that we could opportunistically load those assets from IPFS.

    The question: how to safely detect an asset that can be loaded from IPFS?

    • We could use regex and redirect everything that starts with /ipfs/Qm. There is the problem of false-positives: we could make the regex quite specific, but what if the hash algorithm changes in future? I imagine Qm would change to something else, yes?
    • We could have additional DNS hint which indicates that everything under /ipfs/ can be safely loaded from IPFS. (this proposal)
    • Is there a third way?

The question: how to safely detect an asset that can be loaded from IPFS?

• We could use regex and redirect everything that starts with /ipfs/Qm. There is the problem of false-positives: we could make the regex quite specific, but what if the hash algorithm changes in future? I imagine Qm would change to something else, yes?
• We could have additional DNS hint which indicates that everything under /ipfs/ can be safely loaded from IPFS. (this proposal)
• Is there a third way?

I agree with jbenet that we can infer this info from what's already there. Not from the dnslink record (which is a totally different use case, imo), but from the URL of these assets. You mention github.com/ipfs/foobar and /ipfs/Qm as an example of how this is problematic, but maybe we can make /ipfs and /ipns path validation part of js-multiaddr, and use that in the addon? For /ipfs it must be a multihash, for /ipns it must be a multihash or domain name (see jbenet/go-is-domain). Whether the /ipfs or /ipns path can be resolved is a whole other issue of course, and you can only ever reliably say if it does resolve.

maybe we can make /ipfs and /ipns path validation part of js-multiaddr, and use that in the addon?

cc @diasdavid