http server in the standard library

[andrewrk]

See #2007 for http client. This issue used to be for both.

http server in the standard library will be helpful for writing tests of the http client at the very least. We can re-evaluate what code does or does not live in std lib closer to 1.0.0; right now it's just helpful to have more test coverage for the language.

[ansarizafar]

Check this
Lwan is an opensource high-performance & scalable web server https://lwan.ws/

[bheads]

I can understand putting in an http client for the package manager but an http server? Building a secure http server is a big project. And if its in the std lib people will use it and it will have to be supported. But an officially support http server reference project used by zig to host zig would be awesome, but I would avoid including it in the std lib.

I would also avoid including C projects directly in the std lib. One mistake in D was including Curl in the std lib, now they have to deal with Curl versioning and distributing curl libs with the application. Would be better having an official package for these kitchen sink items, then when we want to end support they can be forked off into the community (or left abandoned in git) without it affecting the std lib.

[costincaraivan]

Agreed. Include a HTTP client and then everything else should come from community packages. If the community converges on a set of very popular, high quality libraries, then you pull those in and even then, for a language such as Zig you'd probably want them just "outside" of the stdlib, in set of libraries maintained by the language developers yet separate (a sort of recommended and supported libraries, but not necessarily with the same commitment for backwards compatibility as the stdlib, which generally is super strict).

The main reason is that you want to avoid the Python or Java situation where you have a ton of libraries which are either bad or completely obsolete but can't be removed.

[andrewrk]

Right now the standard library exists for 2 reasons:

• To test out the language and make sure language decisions are good
• To eventually become the standard library we will ship with zig 1.0.0

So an HTTP server / client would certainly help for the first case. We can evaluate whether it belongs in std when we do the stdlib audit before 1.0.0.

[costincaraivan]

Ah, ok. But don't forget that as more people start to use libraries, the backlash caused by removing something, even before 1.0, increases. I'd at least mark things clearly as "experimental" or such to clarify a bit, to minimize outcry.

[renatoathaydes]

I would suggest adding to the standard library a low-level http library based on RFC-7230, which defined simply the HTTP message format, basically, not semantics (e.g. meaning of HTTP verbs, headers etc. which is defined in other RFCs). This is simple and can be the foundation of higher level libraries, including http clients and servers, while still being useful on its own to accomplish the goals mentioned by @andrewrk.

I wrote a low-level HTTP library in Java (which I am thinking of translating to zig for the async support and speed) and it's not a lot of work. Higher level APIs to work with HTTP are hard to get right and most people have different preferences on what they should look like (which probably explains most languages having several different options - which is probably a good thing), hence those are better left to third-party libraries, IMHO.

[binary132]

HTTP/2?

---

Separately,

This may be an unpopular position, but I have noticed that Go suffers from very weak a) websocket, b) terminal, and c) GUI packages. I realize there is a vast amount of work represented in that short sentence, but there was also an enormous amount of community interest in it, and a likewise enormous amount of community fragmentation around different half-baked solutions to these problems.

Rust has a similar fragmented set of community solutions. Everyone wants to write a parser, everyone wants to do their own async i/o, etc.

Perhaps the way to do this is to support a canonical set of packages, sort of a second-tier set of official repos that are still outside of the stdlib, but that will always be the go-to for Zig for that problem. Thinking out loud here.

[Wulfklaue]

If it helps - here are some languages that have HTTP either in core or standard:

https://crystal-lang.org/

Has default HTTP server build in.

https://dlang.org/

Relies on external Vibe.D for HTTP support.

Rust has a similar fragmented set of community solutions. Everyone wants to write a parser, everyone wants to do their own async i/o, etc.

Its a issue that all Languages share. Any language that has a package manager tends to fragment the community because everybody wants "their" code to be popular or they disagree with the authors of other packages or ...

Look at Go ... Sometimes 20 packages that do the same thing, just in different ways, different supports ( not counting the clones with minimal changes ). Npm ... same.

Languages need to have a tighter controle over there eco system. No Raft Protocol package? Somebody can write one, that follows the language library standard. That becomes a official standard library package.

Somebody want to improve it? Commit to the official version.
Its not maintained anymore... Allow for a new version "2.0" that is backwards compatible.

Keep tight control over the resources. What makes PHP so easy? Not just the language but also the fact that all official extensions are fast, secure and well documented. Its in general the stuff outside the standard library that has issues over time. Hell, you can use 10 year old library code and it will mostly still work.

Unlike languages that rely a lot on external 3th party plugins, where your old code need to use unmaintained code or it does not work or ... and then requires rewrites or whatever issues.

[ansarizafar]

Is there any progress on this issue?

[andrewrk]

Yep, lots of progress:

• cancel semantics for coroutines are now sound and thread-safe
• the stage2 compiler is now a successful proof-of-concept of zig's async/await syntax integrating perfectly into linux's epoll, macos's kqueue, and windows's IOCP.
• we now have these building blocks:
  • std.event.Channel
  • std.event.Lock
  • std.event.RwLock
  • std.event.Future
  • std.event.Group
• in implementing the file system watcher, I discovered a much better API for a tcp server, using a Channel for incoming sockets rather than an async fn handler. This solves the 1 catch @panic("TODO") that I had in the status quo tcp server implementation.
• I'm working on introduce std.event.fs for async file system functions #1294 which is a nearly complete PR for async/await file system API
• @kristate is working on [WIP] UDP Networking #1331 which implements some async/await networking API

[ansarizafar]

Thanks for the update

[kristate]

Once #1271 lands, first it will be great to get some HTTP action. We could even be faster to market than the rust guys with coroutine async.

[u007]

is context in place yet? i think context is important to allow cancellation or io interruption during call to make http cancellable. go context is a good example. i think it should be part of the std http package as well

[daurnimator]

I'm the author of lua-http

I think the networking side of this issue needs to wait for #1778. However there is a small amount of work that can be done before then including e.g. adding a headers object.

I agree with the above assesment(s) that the standard library should only contain low level interfaces. High level HTTP interfaces are far too subjective and will be a source of churn.

I advocate for following the newer HTTP specs, which e.g. consider the method to simply be a header field with name :method. This manages to really simplify things so that a HTTP connection has many streams which has a number of header or body blocks.

Re: debate about including a http server. I should tell the history of lua-http: I originally wrote some code to play around with the http2 spec while it was being written to shake out bugs. Originally I attempted to just write a client, but I found existing servers were terrible for debugging and didn't have good error messages; considering that http2 clients and servers aren't that different, I wrote in support for both. Once I had http2 client and server, it wasn't a huge leap to add a basic http1 client. However to complete this then required writing a http1 server which is quite the large job. But then that was helpful in writing http1 tests...... So after a lot of effort, lua-http then supported http2 and http1/1.1, both client and server.

[ghost]

@andrewrk would it be acceptable to break this into 2 issues?

personally i have more need for a client than a server

that way we can close one without the other if need be

if acceptable we should rename this to server and i can make a new one for client -
or something

[andrewrk]

@cup sure thing. I broke http client out into #2007

[daurnimator]

@andrewrk as I mentioned above, I don't think the implementation for http client and http server should really be separated: you need one to sufficiently test the other. I would keep them as one issue.

[ghost]

you need one to sufficiently test the other

@daurnimator no, no you dont. if a client is implemented first, it could be tested against anything here pretty much

https://rosettacode.org/wiki/Hello_world/Web_server

why would it need to be tested again a Zig server? If Zig server is implemented first, its even easier:

curl something-being-served-by-zig

finally, i dont think either of the issue should be venue for debate - andrew is very busy and unless you are offering to do some of the heavy lifting (IE Zig code writing) i dont think its fair to people who might want the client or server without the other to have to wait for both because you think its a good idea.

[adontz]

And if its in the std lib people will use it and it will have to be supported.

Yes they will. As a workaround, built is HTTP server can be explicitly marked as for tests-only (like put it in std.internal.not_public.for_tests_only.you_have_being_warned), and also make it so bad, nobody will ever be able to use it in production. For instance it may explicitly not support hosting of static files or returning replies larger than one kilobyte. It must be terrible for a greater good.

Also, as a side note, under Windows it's more convenient to use HTTP Server API, and not socket's base server. HTTP Server API already supports TLS, HTTP/2, WebSockets. It includes HTTP parser, allows multiple applications to listen on the same port, but different paths, integrates with NTLM and Kerberos authentication. Also it is implemented in kernel and is fast as hell. Any comparable custom solution will be slower, less feature rich and orders of magnitude more complex.

[leaanthony]

Piping up here as a Zig lurker and Go enthusiast: There are 2 really great thing about having a standard http server in the standard library:

1. It's your go-to for creating web based applications - be that servers, microservices, APIs, etc... In just a few lines you can get something up and running, in a way other people will understand.
2. You have default structures to build upon. This is powerful as this means that as supplementary libraries grow around the standard server, they can inter-operate. Good examples here are middleware and routers. The vast majority of supplementary libraries in Go work on top of the standard interface of (request, response) error. This means you are much less likely to have to rework code should you change 'framework' and I believe this is a big win for everyone.

I'm really looking forward to seeing where Zig goes. It's got great potential.

[iamgreaser]

TL;DR do an HTTP/1.1 server now, do TLS later, and anything outside of that can be deferred to libraries, although I'm not sure how one would handle websockets or WebRTC. Don't bother with HTTP/2 or HTTP/3, because you have dedicated web servers for that.

In The Real World™, you tend to proxy your middleware through a dedicated HTTP server such as nginx, and the more serious setups have cache servers and load balancers.

The server will be fine as an HTTP/1.1 server. If someone wants an HTTP/2 or HTTP/3 server, then that's the responsibility of e.g. nginx... but if someone insists on having that handled in something written in Zig then that can happen outside of the standard library.

TLS for the server is not strictly necessary but considering that it's likely that an HTTP client would need TLS support, and the bulk of the difficulties of certificate checking and whatnot are typically handled by the client rather than the server, I suspect that the server would end up getting TLS support because "we might as well have it and we've done the hard part anyway".

As for websockets and WebRTC... that's something I lack expertise in, so I'd suggest asking someone how it all works.

Also, as a side note, under Windows it's more convenient to use HTTP Server API, and not socket's base server. HTTP Server API already supports TLS, HTTP/2, WebSockets.

The problem with that is it's a Windows-only thing, and in general if someone's going to be running an HTTP server they're going to be running it on Linux. I don't see that ever being used in Zig purely because it's so Windows-specific.

Whereas if you target the Linux case, then Windows will gain whatever support Linux has barring any Win32 API jankiness that hasn't been ironed out at the time.

[watzon]

Especially in the beginning here I completely agree @iamgreaser. TLS support should eventually be added to the client and server, but it's not needed for either to begin with. The biggest need in the immediate future is for a client that Just Works so it can be used for the package manager, and that's unlikely to need TLS support.

Andrew also mentioned during his last stream that, while things like a HTTP server will be integrated into the standard library to begin with, anything that doesn't actually serve a purpose as a part of the compiler/package manager/etc will probably be relegated to its own community managed repository come v1.0.

So for anyone worried about "a HTTP server doesn't belong in the standard library", don't worry, it's not a forever thing.

[LaPingvino]

I think a good first target is having a webserver that is fully acceptable behind a reverse proxy and/or having a fastcgi library. There are very good webservers out there that most people use anyway and you would use those with either of fastcgi or reverse proxy (e.g. I would use Caddy). If someone decided to write something like a Caddy/Apache/Nginx replacement, that would warrant something more capable, but that is not like anything 90% of web projects that would use the webserver would need... having a good http client is way more important as a first target.

[damaon]

I would suggest to make HTTP server implementations outside stdlib and instead focus on common application interface like Rack in Ruby - https://github.com/rack/rack/blob/master/SPEC.rdoc

This makes is easy for people to switch to more efficient servers without reworking apps and is great win for community.

NodeJs suffers a little due to lack of such common abstraction, see for example https://github.com/fastify/fastify-express

[lobre]

Why not having both the interface and a first robust implementation of it in std lib?

That's what go does and it is such a breeze to work with. That's also what made its success. You can so quickly develop a small app and have a production ready http server without having to include any external dep.

And if people want fancier implementations or new middlewares, they can implement the interface themselves or pull a dependency.

[rishavs]

Having an interface and a default server will be my suggestion as well. Crystal and Nim, and most other young languages seem to follow this path.

[jbsolomon]

FWIW, Go's HTTP server types are not very abstract. Its abstraction is mostly at two levels: http.Handler, which the package user implements, and which is responsible for setting headers and writing the response body, and net.Conn, which has a default TCP/TLS implementation that almost nobody reimplements, and which is orthogonal to the HTTP server implementation.

[dch]

This is a thoroughly bikeshed worthy issue which ultimately will require "somebody else" to write the code for it, so I'm loath to throw my 2c in.

That said, I have seen first-hand the confusion and complexities when dealing with languages that don't have a "standard interface" for classic servers: there is an ever-increasing proliferation of half-finished almost-complete mutually incompatible libraries, with a large hidden iceberg of production-breaking problems.

Anybody writing a generic server will then choose one of those, and then slowly cut themselves on the remaining sharp edges as they unexpectedly run into them. Ultimately, users and library contributors suffer again and again because of this.

For example, I'm really excited by zig's native JSON support, and having common interface for HTTP (1.1 & eventually 2) in stdlib would be a huge advantage in gaining traction.

If it's possible within zig "the language" to define pluggable interfaces, and to have within stdlib a compliant HTTP1.1 library that builds on that, that would be awesome. Then perhaps one of the many existing C HTTP client or server libraries can be dropped in. Or the community can extend on, or replace, that stdlib version, with something that's better optimised for specific use cases.

[rishavs]

@andrewrk , sorry about necromancing an old thread, but is there any further thinking on this? If we are to go with both an interface and a default implementation, I think we can pick up the interface spec right away, so that framework maintainers can start following the spec. Having a fixed spec for the interface will be a good idea and will allow us the time to actually implement something along those lines by 0.10.0

[nektro]

there's a lot of other work the core team is doing. if you need something now there's many community options

• https://github.com/Luukdegram/apple_pie
• https://github.com/ducdetronquito/h11
• https://github.com/Vexu/routez
• https://github.com/lithdew/hello
• https://github.com/frmdstryr/zhp
• https://github.com/vrischmann/zig-io_uring-http-server

[andrewrk]

Landed in 2ee3289. Further enhancements and bug reports can be filed separately.

[tobyhinloopen]

I'm trying out Zig and I'm really happy with the HTTP server, thank you!