Integrate w/ HTTP.jl

[quinnj]

HTTP.jl is the consolidated, modernized, and currently actively maintained package for web functionality (uri parsing, content type detection, cookie handling, client requests, basic server functionality). It'd be great to have all the great work @hustf has done w/ this package lately work w/ that. Happy to discuss how to make that as easy as possible.

[hustf]

It's nice to see some activity here! I am using Websockets for private projects, and think it's a great way for unlocking the door between Julia and Javascript powers.

I am thinking it would be a right move to switch for 0.7. As far as I recall now, I couldn't make a 0.7 version because of no response to a httpcommon pr (and some things that were not settled in 0.7 base.write). There are also some seemingly random delays in opening sockets from an unknown browser (Internet Explorer only, perhaps). Before you laugh too much, I am forced to using Internet Explorer for some work.

The old dispersed infrastructure makes sense with dedicated and competent owners, but that is lacking now. A good oportunity for a fresh start. I'd say switching here would require consensus including @MikeInnes, @shashi. Also, tests ought to include a) interfaces which are used in other packages already, b) the SSHL thingie, c) fully working Unix tests.

The second big issue here is including the ideas from Dandelion Websockets. With a good client, Websockets can be an interface between Julia processes, replacing Zmq and a small host of other technologies. But that increases the competence needed for maintaining this package. The current test set covers the common browser's client implementations.

[hgeorgako]

Looking forward to this work. Especially the Websockets (pure Julia) client integration into Websockets and/or HTTP.jl. For finance/trading related work, this is becoming an important technology to support.

[hustf]

If we are to make this change, it would be really really helpful to have the tests running fully on the probably most common os first. I only use windows/ cygwin/ os x, and have not been able to figure out how to launch the browsers on linux. The tests pass with phantomjs headless browser, but that project is slowly dying.

[shashi]

I think this will be good to do since HTTP.jl is well maintained and future-proof!

[EricForgy]

I need this functionality. If someone could outline some rough thoughts on how to proceed, maybe I can give it a try.

[hustf]

That's great! Rough thoughts before looking much at HTTP.jl

• Do this on 0.6 first.
• Keep it on a branch
• Import the functions from HTTP.jl that need to be modified.
• When it works, we can modify HTTP and remove the import.

[EricForgy]

HttpServer.Server explicitly has a websock field https://github.com/JuliaWeb/HttpServer.jl/blob/master/src/HttpServer.jl#L150.

HTTP.Server does not have such an explicit place to stick a websock https://github.com/JuliaWeb/HTTP.jl/blob/master/src/server.jl#L92

Would HTTP.Server need to be modified to include websock?

[EricForgy]

I see HTTP.Server has in::Channel and out::Channel, but I don't think a WebSocket fits this model, does it?

Edit: I take this back. It is probably fine 😅

[hustf]

I'd have to learn more about channels first. Websockets just use sockets provided by HttpServer.

edit: I think you could say each open Websocket is a task initiated by, as it seems to WebSockets.jl, HttpServer. We would want to keep that kind of model at least, where Http takes over HttpServer's role.

[EricForgy]

I'd have to learn more about channels first.

Same here 😅

[hustf]

These things, tasks, channels, sockets, producers, are closely related terms at least. They say to work on one layer of software, you need a basic understanding of the deeper level, really. Hope I can learn something, too. And the IRC apps are a good way to get quick replies.

[hgeorgako]

There's this too: JuliaWeb/HTTP.jl#54

[hustf]

Hm, I actually think it sounds more optimal to keep the websockets protocol in a separate package from Http. There are still going to be changes to the protocol in the future, and not least in the many different implementations of the client protocol out there. We still don't support the full protocol, and we still don't optimize fully for input / output of some buffers / types in Julia 0.7 (from my limited understanding).

For clarity, websockets can not be implemented except as an upgrade to other sockets. HttpServer does that now, and we are switching to Http.
@samoconnor, what's your take?

[samoconnor]

Hi @hustf, as @hgeorgako pointed out, I've implemented a WebSockets client (and test) in my HTTP.jl PR HTTP.jl branch: JuliaWeb/HTTP.jl#135. My motivation for doing this was that HTTP.jl did not have test coverage for the Upgrade: mechanism, and building a simple client for wss://echo.websocket.org seemed like the easiest way to understand how it is supposed to work and test how it interacts with connection pooling and pipelining.

As for division of functionality into packages I would prefer to see more separation rather than less. HTTP.jl currently aggregates code from a number of previous packages, which I think @quinnj would agree could eventually be split out again. Having now spent some time on the internals of HTTP.jl I can see why @quinnj found it necessary to combine the older packages before starting his mammoth refactoring and integration of his pure-julia parser.

My take at this point is that it is essential for HTTP.jl to include higher level protocols so that real-world tests can be implemented. For example my PR branch implements AWS authorisation to enable a test that does lots of concurrent GET/PUT against AWS S3. This enabled me to find intermittent bugs in HTTP.jl, MbedTLS.jl (JuliaLang/MbedTLS.jl#117, JuliaLang/MbedTLS.jl#114) and Base JuliaLang/julia#25314.

The refactoring I've done attempts to separate functionality into modules as much as possible within HTTP.jl to make it easier to eventually split it up into separate packages. e.g.

• URI.jl
• HTTPParser.jl
• HTTPStream.jl
• HTTPServer.jl
• HTTPClient.jl
• HTTPAuth.jl
• HTTPCookies.jl

I think HTTP.jl should try to build up and maintain a test suite that includes stress testing against real-wold servers as well as unit tests. This probably means having a test/REQUIRE dependancy on the relevant client packages after everything has been split up.