Provide HTTP (or WS) API to support browser extensions

[RangerMauve]

As per dat-ecosystem-archive/dat-desktop#518

The IPFS daemon provides an HTTP API which has enabled them to build a Chrome/Firefox extension, ipfs-companion, which allows users to view ipfs:// urls directly in their preferred browser and provides a window.ipfs global to allow web apps to access the local IPFS node instead of bootstrapping one within the browser.

Having an HTTP API for dat will allow us to do the same thing with the goal of providing the features in the beaker browser as an extension one can add to Chrome or Firefox (redirectng dat urls, DatArchive API).

[soyuka]

Hi!

If I understand correctly, this has nothing to do with "serving dat over HTTP" but is rather about "enabling an HTTP API on top of dat"?

I wouldn't see why we could not add an http daemon on top of the current tcp one :). It should be fairly easy to reuse the code as the tcp one only has 50 lines.

[RangerMauve]

Yes, the goal is to be able to make a client that connects over HTTP which provides the browser with the same interface as the dat JS API.

Would a PR using websockets be welcome?

[soyuka]

Okay I get it.

Do you think that we can force end-users (ie API consumers) to use protocol-buffers instead of JSON?

I mean currently the tcp daemon does:

function onconnection (socket) {
  socket.pipe(
    new Transform({
      async transform (chunk, enc, cb) {
        try {
          this.push(await onmessage(chunk))
          cb()
        } catch (err) {
          cb(err)
        }
      }
    })
  ).pipe(socket)
}

And this code can remain unchanged by using WebSockets and slightly modified for HTTP by using req.pipe(transform).pipe(res) if we can use protocol buffers (not sure about HTTP and the Accept header there haha).

Then, last point before getting in the code is to choose to do:

1. A daemon that can launch HTTP, TCP, WS (one or many)
2. One daemon per protocol

WDYT?

[RangerMauve]

Do you think that we can force end-users (ie API consumers) to use protocol-buffers instead of JSON?

Since this library already has the protobuf serialization / deserialization logic in JS, I think it would be very easy to publish that as an NPM package so that both the daemon and potential client implementations can rely on it.
If using protobufs for this is really easy, then I doubt anybody would complain about it.

slightly modified for HTTP

I think that just having websockets will be enough to support any environments that can use HTTP.

A daemon that can launch HTTP, TCP, WS / one daemon per protocol

Personally, I think having a single daemon that allows you to configure protocols would be ideal.

Alternately, why not get rid of TCP and just have WS? It behaves very similarly with the added benefit of working in browsers.

[soyuka]

Yes why not just use WS it'd allow every sort of use case (cli, desktop, browser stuff etc.).
We need to add one configuration hook to determine the listen argument which can be one of:

• port
• socket

I want to keep the socket as default because it allows us to not check if a port is available.

If you want to propose a patch, feel free to do so!

[soyuka]

Oh I almost forgot, we need to expose lib/proto.js somehow so that clients can use the encode/decode functions.

[RangerMauve]

I'd suggest publishing a new module called dat-daemon-protocol which just has the proto.js content as it's export.

Will the cli need to be changed to support WS?

I could do PRs, but I'll have to wait until this weekend to be able to do a proper job with tests since I'm kinda busy during the week. :D

[soyuka]

Np then I'll do it!

Will the cli need to be changed to support WS?

A bit but not much in fact.

I'd suggest publishing a new module called dat-daemon-protocol which just has the proto.js content as it's export.

Why not but it's easier to maintain here especially since the daemon is in its early states :p.

[RangerMauve]

easier to maintain here

True, I think it'll be safe to just have module authors require the proto.js file directly from this project for now.

Really excited for this! I've been crazy about IPFS the past while because they had pubsub and first class browser support, but the more I think about it the more I appreciate how Dat has a few but very powerful features instead of being super low level.

[soyuka]

Mhh one more though, because websockets would need a new dependency, I'll:

• keep tcp as default
• allow wss through configuration

I'm not sure if HTTP has any added value but we can still add it later!

[RangerMauve]

Fair enough. How will people enable websockets? Will it be a configuration change and a global install of ws?

[soyuka]

Mhh I don't like global modules... I'm thinking about a dat-daemon-ws module that'd reuse the code here but just add websockets on top!

[RangerMauve]

Yeah, that's pretty elegant.