Server-sent events explicitly requires --stream

[mivade]

If I have a server-sent event source and try to simply do

http get http://localhost:1729/stream/all

I get output that looks something like this:

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: text/event-stream
Date: Sat, 29 Aug 2015 12:18:00 GMT
Server: TornadoServer/4.2.1
Transfer-Encoding: chunked

followed by none of the streamed data. This can be worked around by setting the --stream flag. However, better behavior would be to automatically enable streaming support when Content-Type is text/event-stream.

[sigmavirus24]

Actually, the best behaviour is to auto-enable --stream when you see Transfer-Encoding: chunked because the length is indeterminate. That said, this is going to be near impossible to do for httpie. An example of what httpie is doing under the covers is

import requests
r = requests.get('http://localhost:1729/stream/all')
print_headers(r)
print_body(r)

What it does with --stream is:

import requests
r = requests.get('http://localhost:1729/stream/all', stream=True)
print_headers(r)
print_body(r)

This means that by the time httpie has received the headers it's too late to begin streaming. httpie would need to perform a HEAD request for every URL to determine whether or not to automatically enable --stream for the user which would be wildly inefficient. While this is certainly a desirable feature, I'm not sure it's practical.

The one way to acommodate this would be for httpie to always pass stream=True to requests and if --stream isn't provided and the headers aren't something that should require streaming, then it reads the whole content into memory (accessing the content attribute will take care of that). Otherwise, it streams appropriately.

I'm not sure if @jkbrzt wants httpie to absorb some logic from it's dependencies though just to perform some magic.

[cbeams]

If a proper fix is not feasible for the reasons @sigmavirus24 details above, perhaps issuing some kind of warning at the console could be an option. Presumably, httpie could notice after the fact that the response's Content-Type: is text/event-stream and then warn the user if they did not explicitly provide --stream.

I ran into this issue and thought I must be doing something wrong on the server side. For example, like @mivade, I was doing something like this:

$ http get http://localhost:8080/interval Accept:text/event-stream
HTTP/1.1 200 OK
Content-Type: text/event-stream
transfer-encoding: chunked


[hang forever]

Then—after considerable time debugging the server side—I tried the same request with curl, and saw that it just worked:

$ curl http://localhost:8080/interval -H Accept:text/event-stream
data:hello 0

data:hello 1

data:hello 2

Only then did I think to search through HTTPie's GitHub issues, at which point I found on this issue. Emitting a warning message as suggested above would save others from going through the same effort in the future. Also, if the user is explicitly specifying an Accept: header with a stream-based media type (as I do in my examples above), that might be a chance for HTTPie to auto-enable --stream.

[sigmavirus24]

If a proper fix is not feasible for the reasons @sigmavirus24 details above, perhaps issuing some kind of warning at the console could be an option. Presumably, httpie could notice after the fact that the response's Content-Type: is text/event-stream and then warn the user if they did not explicitly provide --stream.

So, the underlying transport library, will try to read the entire response before giving control back to HTTPie.

To solve this, the best course of action is for HTTPie to always stream responses unless told not to. This will not incur a performance penalty for HTTPie.

[rwish]

Hi, I'm a newbie to open source projects. I would like to contribute to make HTTPie to automatically recognize server sent events. Could you please provide me with more info and can I work on this issue?

[danieleteti]

Any news on this? I've the same problem... a simple warning would saved hours of debugging.

[sigmavirus24]

To people who want to work on this, it looks like httpie already streams responses by default. The trick is now to find where to add support for auto-enabling --stream when the content type matches one of the content-types that should be streamed.

[jkbrzt]

the best behaviour is to auto-enable --stream when you see Transfer-Encoding: chunked because the length is indeterminate

@sigmavirus24 great suggestion, I'll look into that.

[kamyar]

Hey all,
Came across this issue when I was looking into why httpie does not show the SSE. --stream indeed solved it.
I did some tinkering in httpie code and noticed a potential location it can be handled from the response.
I would like to work on it if it is still relevant and maintainers of httpie are onboard. :)

cc @jakubroztocil