thoughts on http* API

[idiomatic]

For many of us, our first exposure to Node.js was the trivial HTTP server example. It succinctly demonstrated the API conventions one would experience throughout the standard library.

How would you like the Deno HTTP* API to look?

NODEISH

Analagous to readFileSync() and Node.js.

import { createServer, createTLSServer, createSecureServer } from "deno";
const options = {key: KEY, cert: CERT};
const s = createServer((req, res) => { });
const stls = createTLSServer(options, (req, res) => { });
const s2 = createSecureServer(options, (req, res) => { });
s.listen(PORT, ADDRESS);
stls.listen(PORT_TLS, ADDRESS);
s2.listen(PORT_HTTP2, ADDRESS);

MANYLISTENS

Distinct listen() methods.

import { createServer } from "deno";
const options = {key: KEY, cert: CERT};
const s = createServer(options, (req, res) => { });
s.listen(PORT, ADDRESS);
s.listenTLS(PORT_TLS, ADDRESS);
s.listen2(PORT_HTTP2, ADDRESS);

MANYCREATES

Explicit constructors and flat namespace.

import { createHTTPServer, createTLSServer, createHTTP2Server } from "deno";
const options = {key: KEY, cert: CERT};
const s = createHTTPServer((req, res) => { });
const stls = createTLSServer(options, (req, res) => { });
const s2 = createHTTP2Server(options, (req, res) => { });
s.listen(PORT, ADDRESS);
stls.listen(PORT_TLS, ADDRESS);
s2.listen(PORT_HTTP2, ADDRESS);

NAMESPACES

Node.js-like namespaces to disambiguate createServer().

import { http, https, http2 } from "deno";
const options = {key: KEY, cert: CERT};
const s = http.createServer((req, res) => { });
const stls = https.createServer(options, (req, res) => { });
const s2 = http2.createServer(options, (req, res) => { });
s.listen(PORT, ADDRESS);
stls.listen(PORT_TLS, ADDRESS);
s2.listen(PORT_HTTP2, ADDRESS);

CREATEOPT

createServer() attribute.

import { createServer } from "deno";
const options = {key: KEY, cert: CERT};
const s = createServer((req, res) => { });
const stls = createServer({...options, protocol: "https"}, (req, res) => { });
const s2 = createServer({...options, protocol: "http/2"}, (req, res) => { });
s.listen(PORT, ADDRESS)
stls.listen(PORT_TLS, ADDRESS)
s2.listen(PORT_HTTP2, ADDRESS)

LISTENARG

listen() method argument.

import { createServer } from "deno";
const options = {key: KEY, cert: CERT};
const s = createServer(options, (req, res) => { });
s.listen(PORT, ADDRESS, "http");
s.listen({protocol: "https"}, PORT_TLS, ADDRESS);
s.listen({protocol: "http/2"}, PORT_HTTP2, ADDRESS);

LISTENMAP

listen() port map

import { createServer } from "deno";
const options = {key: KEY, cert: CERT};
const s = createServer(options, (req, res) => { });
const portMap = {80:"http", 443:["https", "http/2"]};
s.listen(portMap, ADDRESS);

Legacy Node.js

const http = require('http');
const https = require('https');
const http2 = require('http2');
const s = http.createServer((req, res) => { });
s.listen(PORT, ADDRESS);
const options = {key: KEY, cert: CERT};
const stls = https.createServer(options, (req, res) => { });
stls.listen(PORT_TLS, ADDRESS);
const s2 = http2.createSecureServer(options, (req, res) => { });
s2.listen(PORT_HTTP2, ADDRESS);

Go

package main

import "net/http"

func main() {
    s := &http.Server{Addr:"ADDRESS:PORT"}
    s.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { })
    go func() {
        s.ListenAndServe()
    }()

    stls := &http.Server{Addr:"ADDRESS:PORT_TLS"}
    stls.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { })
    go func() {
        stls.ListenAndServeTLS()
    }()

    // wait forever
    select{ }
}

[matiasinsaurralde]

I've been hacking with a prospective http module, I really like the Nodeish one 😄

[Janpot]

I'd love to see some sort of inverse fetch. With a bit of ServiceWorker mixed in.

import { serve } from 'http'

serve(80, async ({ request }) => {
  const body = JSON.stringify({ path: request.url.pathname });
  return new Response(body, {
    status: 200,
    headers: new Headers({
      'content-type': 'application/json'
    })
  });
});

I'd love to be able to build a fetch -> ServiceWorker -> server chain where every part is built on the same API paradigm.

Edit:

This kind of API also allows some nice middlewares patterns:

function log (handler) {
  return async ({ request, ...props }) => {
    const start = Date.now();
    const response = await handler({ request, ...props });
    const duration = start - Date.now();
    const { method, url } = request;
    const { status } = response;
    console.log(`[${start}] ${method} ${url} ${status} (${duration}ms)`)
    return response;
  };
}

serve(80, log(({ request }) => {
  // ...
}));

variations:

serveTls(443, cert, key, ({ request }) => ...)
serve(new Socket(...), ({ request }) => ...)

idk, just sketching some ideas...

[Jusys]

why not "MANYLISTENS" or "LISTENMAP"? For me they're shortest and cleanest - for newcomers with less experience of deno or typescript it would be easier to understand what's happening.

[cmbkla]

I like "MANYLISTENS", with the assumption that I would have a way to access the same instance of s in another file, and would then get events like OnRequest and OnResponse that I could hook into with middleware or a router.

[nicolasparada]

@Janpot Such a nice API 👏👏👏👏👏

[AnwarShahriar]

MANYLISTENS and LISTENMAP FTW

[EliHeller]

@Janpot what about this...

import { Server, Response, Headers, Routes } from 'http';

const routes = new Routes([
    {
        url: '/',
        method: 'GET',
        response: new Response('hello world', {
            status: 200,
            headers: new Headers({
                'content-type': 'application/html'
            })
        })
    },
    {
        url: '/',
        method: 'POST',
        response: new Response({message: 'hello world'}, {
            status: 200,
            headers: new Headers({
                'content-type': 'application/json'
            })
        })
    }
]);

const PORT = 80;

await new Server(PORT, async request => routes.match(request));

console.log(`http server listening on port ${PORT}`);

[Roshan931]

I think MANYLISTENS are clean and understandable for beginners. There is one server and it can listen on different protocols. Maybe just use a different name for http/2, .listen2 is a bit unclear.

What about something like this...

import { createServer } from "deno"

const options = { key: KEY, cert: CERT }
const s = createServer(options, (req, res) => { })

s.listen(PORT, ADDRESS) // or s.http.listen
s.tls.listen(PORT, ADDRESS)
s.http2.listen(PORT, ADDRESS)

[Janpot]

I think http2 is going to require a little more API than just listen

[Janpot]

@EliHeller That's how I imagine you'd write a routing framework on top of it. I guess it all depends on how batteries included this project intends to be.

[EliHeller]

@Janpot either way with your proposed API i think it is more simple to do routing

[quasis]

Node.js requires wrappers even for trivial tasks. Otherwise, the code becomes too complicated. No wonder express.js became so popular.

IMHO anything that will remove the need in wrappers will be excellent. For example:

import {HTTPServer, HTTPResponse} from "deno";

const server = new HTTPServer();

server.get('/:var1/:var2', (request, var1, var2, queries) =>
{
    return HTTPResponse.json({});
});

await server.listen(8080);

[Lizhooh]

Hope similar to Koa.

[asosnovsky]

What about a semi-java semi-express way of

import { Server, Router, Request, Response } from "http";

const router = new Router();
const server = new Server({
    port: 443,
    ssl: "path-to-ssl",
});

router.route("/", (req: Request): Response => {
    if( req.method === Request.GET ) {
        return new Response("hello world");
    }
    return Response.NotFound();
}):

server.route("/", router);
server.listen();

[ccravens]

While not directly addressing the original topic author's question, I thought it important to discuss. I'm of the opinion that a language API should provide a VERY basic, non-opinionated interface to raw services, for example http. It really should only provide the ability to start a VERY lightweight service, create a request, and respond to requests with request/response objects. But it looks like we are discussing how to handle router configuration. This all should be the responsibility of the framework, not the language API.

IMO, this is what the HTTP language API should offer:

• Create a very lightweight HTTP service
• Provide a request callback with a Response object to respond to valid requests, error callback for invalid requests
• Make raw HTTP calls by creating a raw HTTP request object
• Provide a response callback with the Response object

What the deno HTTP API should NOT offer:

• Router configuration (how do you configure routers? Config file? Declarative in the source code? XML? Decorators?) How do you format your routes for get parameters? What should those get parameters looks like (regex)? All of these are used by a variety of frameworks and should be left to the framework
• Managing views
• Managing connections to a database (connection pooling), Models and ORM

By staying non-opinionated and only focusing on the very basics of the service, this gives the open source community the freedom to offer a variety of web frameworks that can use all sorts of different styles and architectures based on the needs / desires of the individual developer.

[NewLunarFire]

Ryan's talk mentionned how he thought the Node library has grown huge and wanted to keep a simple native interface for deno. You could even argue that you might not need an HTTP api but only a network api and build an HTTP library in userspace.

[idiomatic]

@NewLunarFire and @ccravens, I was thinking similarly with this functional-programming mock-up.

LAYERED

/* this bloat could justify offshoring to a third-party module. */
type RequestHandlerResponse = string | Uint8Array | Reader | Response | Promise<string> | Promise<Uint8Array> | Promise<Reader> | Promise<Response>;

/* wrap connections with TLS.  usable for HTTPS or other TLS protocols. */
export function tls(key, cert: string, cx: ConnectionHandler): ConnectionHandler { … }

/* groks HTTP and HTTP/2; could offshore to a third-party module. */
export function http(h: RequestHandler): ConnectionHandler { … }

import { listen, tls, http, open, Response } from "deno";
// import { route, redirect, static, hterror } from "... third-party ...";

const server = http((req) => {
	if (req.path === "/") {
		return "hello world\n";
	else if (req.path[:8] === "/static/" && req.path.index("/.") == -1) {
		return open(req.path[8:], 'r')
	} else {
		return new Response({statusCode: 302, body: "moved", location: "/"})
	}
});
listen(PORT, server);
listen(TLS_PORT, tls(KEY, CERT, server));

or more trivially:

import { listen } from "deno";
import { http } from "... first or third party ...";
listen(8000, http((req) => "hello world\n"))

[idiomatic]

GENERATORSTACK

Contemporary language syntax features rather than callbacks, object-oriented, or declarative patterns.

import { listen, tls, http } from "deno";
for (let req of http(tls(listen(TLS_PORT)))) {
	return "hello world\n"
}

And to merge multiple connection generators under the same protocol transgenerator, give http() a varargs signature.

EDIT

As @agentme explained in depth (and in far kinder terms), GENERATORSTACK is a dumb, not-even-half-baked idea. The author intended to write something other than return but is dissatisfied with the syntactic bloat.

[Macil]

GENERATORSTACK

Unless there's some radical javascript proposals I'm behind the times on, you can't return to a loop and you need async iteration, so I'm going to respond as if your example were the following:

import { listen, tls, http } from "deno";
for await (let connection of http(tls(listen(TLS_PORT)))) {
	connection.respondWith("hello world\n");
}

Something awkward about the strategy is what happens if you need to await a promise while handling a request. Consider the naive approach:

import { listen, tls, http } from "deno";
for await (let connection of http(tls(listen(TLS_PORT)))) {
	const result = await mySlowDatabaseCall();
	connection.respondWith("hello world\n" + result);
}

Several problems:

• Only one request can be handled at a time. If mySlowDatabaseCall() takes one second to resolve, then new requests to the server have to queue up during that time.
• The req object has to wait until respondWith is called before responding. If mySlowDatabaseCall() throws an exception, or if there's a code path that accidentally never calls req.respondWith(), then the server will hold the connection open forever without responding.
• If there's an uncaught exception while processing requests, then the for-loop entirely stops and the server stops responding to requests. There's no way for the http() module to have any built-in error handling (like give a 500 response to the user and then keep serving requests).

You could address these issues by making req.respondWith() take a promise:

import { listen, tls, http } from "deno";
for await (let connection of http(tls(listen(TLS_PORT)))) {
	connection.respondWith((async () => {
		const result = await mySlowDatabaseCall();
		return "hello world\n" + result;
	})());
}

But that's pretty verbose and I don't think it has any advantages over a more classic async callback strategy, which could look like this to use the same composition strategy you were going for:

import { listen, tls, http } from "deno";
http(tls(listen(TLS_PORT))).use(async connection => {
	const result = await mySlowDatabaseCall();
	return "hello world\n" + result;
});

[CanRau]

some thoughts (might be mentioned already?!)

I like koas middlewares, as mentioned in the 2. comment #199 (comment) by @matiasinsaurralde
but I also really like the "less hidden magic" approach of Zeits micro

[nicolasparada]

I'm in favor of a service worker like API:

server.addEventListener('request', ev => {
  ev.request
  ev.respondWith(new Response())
})

[trylovetom]

Hope api similar to Micro JS. To being lean and explicit.
vercel/micro#8 (comment)

[MichaelJGW]

I have no idea what I'm doing but I thought it would be fun to pitch in. This is what I got.

What do we need to do?

• Declare what the server is
• Give the server what it needs
• respond to the users of the server

// Declare what the server is
// Give the server what it needs

const server = new HttpServer ({
  // Anything the connection needs goes here.
  // If it was a Socket or a HttpsServer this would change accordingly.
  address: 'localhost',
  port: 8080
})

// HttpServer emits events like GET, POST, PUT, ...

// respond to the users of the http server
server.addEventListener('GET', (state) => {
  // As this is a HttpServer the state of the event has request and response.
  // If it was something like a Socket the state of the event can be different.
  state.response('Hello World')
})

With this going from Sockets, HTTPS, HTTP the only changes from this pattern.

• Config given to the server.
• Event emitted
• State Structure

Without comments

import {HttpServer} from "deno"

const server = new HttpServer ({
  address: 'localhost',
  port: 8080
})

server.addEventListener('GET', (state) => {
  state.response('Hello World')
})

[Qard]

I like the idea of just layering async iterators to apply protocol logic in isolation. It's easy to layer the Node.js-like abstractions on top of that. For example, http.createServer could naively be implemented like this:

async function createServer (protocol, handler) {
  for await (let request of protocol) {
    const response = handler(request)
    request.send(response)
  }
}

class Request {
  constructor (socket, { method, path }) {
    this.socket = socket
    this.method = method
    this.path = path
  }

  async send (iter) {
    for await (let chunk of iter) {
      this.socket.write(chunk)
    }
  }
}
class Response {
  constructor (status, body, headers = {}) {
    this.status = status
    this.body = body
    this.headers = headers
  }

  async *[Symbol.asyncIterator]() {
    const { headers, status } = this
    yield `HTTP/1.1 ${status} ${messageForStatusCode(status)}\n`
    
    for (let header of Object.keys(headers)) {
      yield `${header}: ${headers[header]}\n`
    }

    yield "\n"

    for await (let chunk of this.body) {
      yield chunk
    }
  }
}


async* function http (protocol) {
  for await (const socket of protocol) {
    const head = await readHead(socket)
    yield new Request(socket, head)
  }
}

createServer(http(tls(tcp(PORT))), async request => {
  const body = async* () => {
    yield 'hello '
    yield 'world\n'
  }()

  return new Response(200, body)
})

[kevinkassimo]

For those who are interested, Ryan is implementing http as an importable module here and is using async iterator to accept requests.

[bartlomieju]

HTTP module is available in standard library.

CC @ry

[ry]

Move HTTP API discussions to https://github.com/denoland/deno_std