DNS query api

[dylanPowers]

For the web browser extensions sake, I realized it would be nice if there was a dns query api available. This way, when a request for http://ipfs.git.sexy/ comes by, a dns lookup could be made to see if an ipfs txt record exists. If it does exist, the extensions could redirect to /ipns/ipfs.git.sexy/ otherwise the request would complete as normal.

Currently, I can forcibly attempt the request to /ipns/ipfs.git.sexy/ and see if it fails or not, but that's a rather messy way to go about it.

This was originally discussed in ipfs/ipfs-companion#16 (comment)

[lidel]

Summary of the discussion in ipfs/ipfs-companion#16:

For seamless IPNS support add-ons need a simple JSON API that accepts
HTTP GET /dns/ipfs.git.sexy requests (where ipfs.git.sexy is FQDN).

A) If TXT record with IPFS hash for the domain is present in DNS, return HTTP 200 with JSON body:

{ 
  ipns: "/ipns/ipfs.git.sexy/", 
  ipfs: "/ipfs/QmVyS3iAy7mvDA2HqQWm2aqZDcGDH3bCRLFkEutfBWNBqN/" 
}

B) If there is no TXT record, return HTTP 404 (Not Found)

[dylanPowers]

I was playing around with this and I actually like the idea of just attempting the request to /ipns/ipfs.git.sexy and seeing if it 404's or not (currently it's 400ing which needs fixing). My major hesitation with it was that it would be slow, but as long as the gateway is always quick to return a response header and I'm prudent in caching requests, things should be fine. My only real concern is if we hit a "context deadline exceeded" (aka this request took way too long so it probably doesn't exist) where a header won't be immediately sent because the server doesn't know the response code to send. Of course in that instance, the user would have similar troubles if they just typed in /ipns/ so I'm fine with that.

[lidel]

I am bit sceptical about “just attempting the request to /ipns/ipfs.git.sexy and seeing if it 404's or not”. Even if all we do is a HTTP HEAD request it would be too slow to do for each request done by browser.

We need a separate, fast DNS query API provided by a local go-ipfs instance plus a shallow cache in browser add-on (so that we don't sent DNS query for each browser request, only for the first one per domain) that expires after some time (15minutes? 4hours?).

(Apologies if I misunderstood your intent)

[dylanPowers]

I've gone ahead and implemented this in my chrome extension https://github.com/dylanPowers/ipfs-chrome-extension/releases/tag/v0.4.0. I tested it out on my Chromebook, which is about the lowest powered computer you can get, and even with the ipfs gateway set up remotely, it works great with minimal impact to browser performance. I'll have to play with it for awhile to see if I notice any performance hiccups.

[jbenet]

@dylanPowers does that mean that you hit /ipns/<domain> for every single domain you browse?

[dylanPowers]

@jbenet Yep

[jbenet]

that's very bad.

• if your extension takes off, it's potentially a huge amount of requests to the gateway, each of which triggers DHT lookups which are 90%+ guaranteed not to be there, which are by far the most expensive. this is not ok the gateways will not cache ipns lookups because we want to give freshness. a way this could work is with a different service that coalesces requests among users, and caches aggressively. (we could expose this on the same nodes, but it would be a different route altogether)
• it's also a huge breach of privacy on the user. this allows us to spy on what domains users with your extension visit a map[ipaddr + user agent]->domain could be easily established. and, since the gateway is http for now, it basically allows anyone on the path to do so too. please take privacy more seriously. if your extension takes off, people can be put at risk.

Do extensions let you do DNS lookups? if so, that's the best way to go.

A second best, is to use a DNS proxy or an API / service (on https) that understands these requests need to be coalesced and cached (similar to the service @lidel suggests).

[whyrusleeping]

I could write a quick http->dns server with caching. Shouldnt take more than five minutes

[jbenet]

@whyrusleeping 👍

[lidel]

In Firefox SDK add-ons can only see A records. I could not find API for reading TXT from DNS. I even looked at add-ons that operate on non-A records such as one for DNSSEC validation and it turns out they try to use libraries installed in OS, and if missing ship custom binary plugin for each OS.

So it seems that having DNS TXT resolution built-in / provided by local HTTP2IPFS gateway is the best way, as it is system-agnostic and can be use across all browsers (even UserScripts could use it).
(Any kind of third-party service for lookup over the internet will leak private information about user's browsing habits.)

[dylanPowers]

@jbenet

if your extension takes off, it's potentially a huge amount of requests to the gateway, each of which triggers DHT lookups which are 90%+ guaranteed not to be there, which are by far the most expensive. this is not ok the gateways will not cache ipns lookups because we want to give freshness. a way this could work is with a different service that coalesces requests among users, and caches aggressively. (we could expose this on the same nodes, but it would be a different route altogether)

I think you're confused that I'm using gateway.ipfs.io for ipns lookups? All requests only go through the local daemon. Currently I have caching set up in the extension, which is better than the API doing caching. In fact, I don't want the API to do any caching. To be on the fast path, the extension needs to reduce the number of HTTP requests it's making and when it does make a request it needs to really count.

each of which triggers DHT lookups which are 90%+ guaranteed not to be there, which are by far the most expensive

If there is no DNS record, why would it make a DHT lookup?

it's also a huge breach of privacy on the user. this allows us to spy on what domains users with your extension visit a map[ipaddr + user agent]->domain could be easily established. and, since the gateway is http for now, it basically allows anyone on the path to do so too. please take privacy more seriously. if your extension takes off, people can be put at risk.

DNS queries are not going to external servers. All requests are made to the local daemon which is under the user's control. The only way requests are sent to external servers is if the user explicitly sets up the extension to do that.

Do extensions let you do DNS lookups? if so, that's the best way to go.

Hah, I thoroughly looked into this. In the Chrome ecosystem, Chrome Apps are the only thing that can make generic socket connections. Google doesn't trust extension developers quite yet.

---

I still need to look deeper into browser dns caching techniques. In an extension I'm awfully limited on what I can do, and it will never come close to what the browser is natively doing, thus the performance will never be at the level it could be. With that said, I'm quite happy with the potential, although I temporarily took down the release because there's a couple of edge cases in regards to performance that I need to take care of.

[lidel]

@dylanPowers I feel I need a quick clarification regarding this:

I was playing around with this and I actually like the idea of just attempting the request to /ipns/ipfs.git.sexy and seeing if it 404's or not (currently it's 400ing which needs fixing).

So my understanding of this flow is this. I go tohttp://foo.bar. Browser add-on makes HTTP request to /ipns/foo.bar at a local gateway and gets a response:

A) 404 -- nothing happens, regular HTTP GET is sent to foo.bar.
B) 200 -- what happens now? Are you providing already received content to user, or are you making a duplicate request to the gateway? Or was the first request was just a HTTP HEAD and the second one is HTTP GET?

[dylanPowers]

You've got it right.

B) 200 -- what happens now? Are you providing already received content to user, or are you making a duplicate request to the gateway? Or was the first request was just a HTTP HEAD and the second one is HTTP GET?

I wish Chrome would allow me to not waste a request, but I just do a redirect. The flow works like so:

I tried an HTTP HEAD request, but the daemon isn't set up to handle those so I'm stuck doing HTTP GET.

[jbenet]

@dylanPowers

I think you're confused that I'm using gateway.ipfs.io for ipns lookups? All requests only go through the local daemon.

Ah-- you're totally right. i assumed this would go to the global gateway in case you didnt have a local gateway. If it doesn't that's much better. still a bit sketch, but threat-model ok.

It's important that this does not fall-back to the global gateway though! please make note of the concerns in your codebases

If there is no DNS record, why would it make a DHT lookup?

Right again! I misunderstood the setup here. Sorry for jumping to conclusions!

[lidel]

@dylanPowers
Thanks for clarification. I agree that with cache it will not impact browsing performance much, but due to this wasted HTTP GET it feels like a temporary hack, not something we should rely on as a default.

I think the way to go is to have cache in add-on (with keys expiring after 15min or 4h) AND a light DNS TXT resolver provided by go-ipfs (as described in #1054 (comment)) as a default.

If local gateway does not provide DNS resolver (eg. alternative IPFS implementation), we could gracefully fallback to the flow with 'wasted HTTP GET'.

Does it make sense?

[dylanPowers]

FYI, I've recently found the ipfs dns api. Somehow I missed it.... http://localhost:5001/api/v0/dns/<domain>