Support for server-sent events (EventSource) #99

adamchel · 2019-01-29T22:01:09Z

Introduction

Server-sent events are a standard W3C-specified and WHATWG-specified feature that supports processing real-time events from a server over an HTTP text streaming protocol.

All modern browsers (except Edge) have supported this feature for years, and for Edge it is a feature request with 6.7k votes.

There is no support in React Native, nor could I find any open issues for it. I believe React Native should have built-in support these types of streaming requests just like it supports fetch and WebSockets.

The Core of It

The introduction covers the gist of the issue and the proposal, but I will also point that while there are existing polyfills for this feature, they are not sufficient, and I think built-in support is the best solution. Below, I've outlined why I think the existing polyfills are insufficient.

Existing Polyfill 1 : eventsource

With 4.9M weekly downloads, this is the definitive EventSource polyfill and it works well on browsers and Node.js, but unfortunately it uses the node standard library, which is not supported by React Native.

[16:03:52] The package at "node_modules/eventsource/lib/eventsource.js" attempted to import the Node standard library module "url". It failed because React Native does not include the Node standard library. Read more at https://docs.expo.io/versions/latest/introduction/faq.html#can-i-use-nodejs-packages-with-expo

Existing Polyfill 2: react-native-event-source

This library specifically supports React Native with its EventSource implementation, but it is based on a polyfill that wraps XmlHttpRequest, which is not optimal for network streaming.

The polyfill does work, but it doesn't have much usage, and it doesn't play nicely with the default jest configuration for react native (see jordanbyron/react-native-event-source#14). I shouldn't have to run through a lot of hoops to test code using such a basic networking feature.

Existing Polyfill 3: react-native-eventsource

A new version of this polyfill hasn't been released in two years, it appears to have only 2 weekly NPM downloads, and it uses native modules, which makes it infeasible to use in simple Expo-based applications.

Discussion points

EventSource is a standard web feature that is supported well by most browser environments, and well supported by a polyfill in the Node.js environment. As it is a text streaming protocol, it is also relatively straightforward to implement. For this reason I believe it should be a first-class citizen in React Native networking just like fetch and WebSockets. See Networking in the React Native docs.

The text was updated successfully, but these errors were encountered:

elicwhite · 2019-01-30T03:36:16Z

The main question I always have for proposals like this is why does this feature need to be in core vs a 3rd party library?

kelset · 2019-01-30T09:48:10Z

(for reference: #83)

adamchel · 2019-01-30T16:16:16Z

I see in the other discussion referenced that @cpojer says that the "current focus is to reduce the surface area instead of adding to it."

I understand this concern when it comes to non-standard libraries or features that don't exist by default in other JavaScript environments. Redux for instance is an example of a library I would completely agree should remain as a 3rd party library and be out of core.

However,EventSource as I have pointed out is a standard web feature specified by W3C and WHATWG, is well documented by MDN, and is natively supported by browsers.

React Native includes the fetch and WebSocket APIs in core, and your docs even reference the MDN documentation: https://facebook.github.io/react-native/docs/network, (though React Native doesn't support the Streams API for fetch, which would make building a third-party library for EventSource a lot easier).

I'd like to hear from @TheSavior and @cpojer on why they believe EventSource is different from fetch and WebSockets in that it doesn't deserve to exist in the core library.

In my opinion, when JavaScript environments like React Native choose to exclude standard features like this, it further fragments the already messy JavaScript ecosystem, making it a lot more difficult to build universal JavaScript libraries that work across platforms.

I work on a third-party library that intends to support React Native among other platforms (MongoDB Stitch JS SDK), but we've run into many issues along the way with platform-specific bugs and feature gaps, and we've had no choice but to package React Native-specific, Node.js specific, and browser-specific libraries that sit on top of a platform-independent JS codebase that abstracts away features that aren't natively supported by all those environments. Fortunately we have the time and resources to do that, but many independent library developers do not.

If React Native doesn't intend to support features like EventSource, I think it would help library developers a lot if you make your intentions clear on which subset of the standard browser JavaScript ecosystem you intend to support and which you do not.

cpojer · 2019-01-31T09:36:38Z

Hello @adamchel,

thank you for bringing up this issue. I think @TheSavior's question is a good one, and as you pointed out yourself we are actively trying to reduce the surface area of React Native.

In terms of which standards we intent to support, I think it is too early in the lifetime of React Native to settle on any one standard. However, keep in mind that we are not attempting to build a standards compliant web browser, so we will likely never match features in modern browsers exactly. Absent of having a standardization process or spec to follow, I think we should include features that the majority of people will find useful. This is also why fetch and Web Sockets are available in React Native. I don't think at this time we can claim that server sent events are similarly important to a large amount of users.

I do think the best course of action is to either go with eventsource or react-native-event-source. If eventsource expects a url module, maybe we can make that a separate package and support it from React Native? If we were to go with solution 2, I'm pretty sure we can figure out how to make Jest work with it.

Even if we were to consider including this into core at some point in time, it would most likely be an additional plugin that people would have to enable. This is why I'd prefer to help improve the ecosystem around React Native – if some extensions get significant traction, we should consider shipping them by default.

adamchel · 2019-01-31T16:24:56Z

Thanks for your input. I really appreciate the quick responses on this. I totally understand that you don't want to clutter the core React Native library with things that most people won't find useful. I'll offer two more counterpoints but I'm getting the sense that you'd prefer we use and improve a third-party library before you'll consider including this in core 😃.

As for your point on

we should include features that the majority of people will find useful. This is also why fetch and Web Sockets are available in React Native. I don't think at this time we can claim that server sent events are similarly important to a large amount of users.

Server-sent events are more more widely used than you might think. The most popular eventsource polyfill has 4.9M downloads on NPM. Meanwhile, the most popular libraries for fetch have 5.5M downloads for the browser polyfill, and 6.9M for the Node.js library. The most popular WebSocket library for Node.js has 9.9M downloads. Yes, you're right that fetch and WebSocket are more widely used, but eventsource is popular on a similar order of magnitude.

My second point is that as far as I know, there is no native mechanism for simple HTTP streaming in React Native, whether it's via EventSource, or the new fetch Streams API. If there were, then it would be very straightforward to implement EventSource as a third-party library without resorting to a hacky wrapper over XmlHttpRequest.

Are there plans to implement the fetch streaming API in React Native, or is WebSocket going to be the only way to natively do streaming? As this StackOverflow discussion shows, there is value in a simple low-overhead one-way streaming protocol. It makes server implementations a lot simpler, and browsers already support consuming the streams.

elicwhite · 2019-01-31T16:33:26Z

In general I think there are three approaches we could take:

Add this functionality to core
Link a 3rd party module to core by default
Recommend using a 3rd party library in your app

The problem with adding this functionality to core is that we aren’t the experts on this behavior. You really don’t want us to be the gatekeepers for this code. It would move too slowly and not get the attention it deserves. I think one thing you are really asking for in this issue is for Facebook to step in and create a high quality eventstream library because the existing 3rd party ones aren’t good enough. I think it is way less likely that we’d create a good one in core than the community could band together and create a good one themselves from the people who care most about this behavior.

The second approach, linking a well supported 3rd party library by default is more interesting as it would enable the community to iterate freely and then a compatible library would be linked. However, the major downside to this is that it bloats React Native for everyone, including those who don’t use it. We want to decrease the size required by React Native apps and the way to do that is to enable apps to easily add what they need.

This leaves us with the third approach, the one we have today. Use a 3rd party library in your app to add the functionality you need. I think this is likely where we’ll stay on this due to the reasons outlined for the other approaches.

adamchel · 2019-01-31T16:43:32Z

@TheSavior thanks, that makes a lot of sense. My last question then is, what about implementing the Streams API for fetch? (Usage example here)

Network streaming isn't something that you can just easily implement with a third-party library on top of the network primitives that exist in React Native today. You need more low-level access to the networking, which the existing fetch API in React Native doesn't support. To me, that's the biggest thing getting in the way of creating a high-quality eventsource library without resorting to native modules.

dkaminsky · 2019-01-31T16:58:38Z

Agreed with @adamchel, if you don't think it's appropriate for you to provide native/bundled eventsource or equivalent, implementing Streams API for fetch would be the minimum needed for the community (or us) to create a workable solution that doesn't rely on XHR.

If you won't implement the standard, please expose the primitives needed for others to do so.

elicwhite · 2019-01-31T16:59:23Z

I’m not sure what is involved in making that happen but that makes some sense to me.

adamchel · 2019-01-31T17:50:35Z

@TheSavior would you prefer if we opened a new proposal in this repo for supporting a low-level networking API and/or the fetch streams API? Or should I edit this existing one to reflect that?

And I also just wanted to clarify why we believe XHR is not sufficient. XHR doesn't let you clear the incoming streamed response buffer as data comes in (more context here). This means that as data comes in and your application processes it, it can't clear the buffer without starting a new XHR, which defeats the purpose of streaming data. This has implications for mobile devices, where if you have a long-running stream of data, memory use is going to shoot up unless you restart the stream.

The EventSource library react-native-event-source that works on react native currently implements streaming by where the new incoming bytes (or new events) are retrieved by taking the substring of the partially buffered response from the XHR that the EventSource hasn't seen yet (see https://github.com/jordanbyron/react-native-event-source/blob/master/EventSource.js#L68). What ends up happening is that you store a whole lot of buffered response data that is no longer necessary after those bytes are processed. We can implement our library with this non-ideal solution for now, but long-term we'd love to have access to networking primitives that will allow us make a better React Native EventSource library.

adamchel · 2019-01-31T18:32:46Z

As for what needs to be done to make this happen, I'm not too familiar with the React Native source code, but from some light perusing, it seems like the low-level RCTNetworking API (https://github.com/facebook/react-native/tree/master/Libraries/Network) can support streaming, the mechanism for doing so just needs to be exposed.

Interestingly, you implement fetch using the whatwg-fetch polyfill (source here) which just sits on top of your XmlHttpRequest implementation. whatwg-fetch does not support the Streams API because XmlHttpRequest doesn't support streaming. It would probably be a lot of effort on your part to implement the Streams API since you'd actually have to implement all of fetch as well, and I'd understand if you wouldn't want to go down that route.

The low-level RCTNetworking API does seem to support incremental data load:

If Facebook were able to expose a simple request API using RCTNetworking that allowed us to receive data incrementally without keeping the entire response in memory as data comes in (which the XmlHttpRequest implementation does), I think that would be sufficient for us to implement a React Native EventSource library that meets our needs.

elicwhite · 2019-01-31T19:54:06Z

Thanks for the investigation! I’d recommend repurposing this discussion.

I think we are probably open to the idea of exposing some hooks to RCTNetworking but I wouldn’t expect us to have any bandwidth to make that change. The best way to probably make this happen is to investigate the smallest technical changes to core necessary to enable what you need to happen in user space and make sure that is of reasonable scope to be worth sending a PR or if we need to expand the discussion first.

Does that make sense and seem reasonable?

adamchel · 2019-01-31T20:34:25Z

I think that sounds reasonable. When we have the bandwidth (probably sometime in the next few weeks), I can work on producing a PR to create the necessary hooks.

As far as scope goes, I'm imagining two possible approaches:

Implementation of a SimpleStreamRequest JS class in https://github.com/facebook/react-native/tree/master/Libraries/Network that wraps RCTNetworking to support HTTP requests that receive a chunked stream of data instead of a single response body. This wouldn't be following any existing standards, such as the fetch Streams API (since it would be significantly more work), but I think it would simple enough for us to build on top of, and it could also be useful for anyone building applications or libraries that want to make use of simple one-way HTTP streaming. We could build a third-party EventSource library that would exist on top of this SimpleStreamRequest.
Implementation of an EventSource JS class in https://github.com/facebook/react-native/tree/master/Libraries/Network that wraps RCTNetworking to support the EventSource standard. The advantage of this approach is that we are implementing a standard web feature, and not bloating the React Native core with a non-standard network primitive like SimpleStreamRequest. EventSource is itself a very simple protocol that could probably be written with ~200 lines of code directly wrapping RCTNetworking. The XHR wrapper for EventSource is about 200 LOC: https://github.com/jordanbyron/react-native-event-source/blob/master/EventSource.js

Whatever we end up producing, whether it is SimpleStreamRequest or EventSource, we'd expose it in React Native user land via https://github.com/facebook/react-native/blob/master/Libraries/Core/setUpXHR.js.

I personally would prefer the second approach since it's more standard, but would love to hear your thoughts. We wouldn't want to start any work on this unless we know you'd be open to accepting a PR with the approach we go with.

jhampton · 2019-01-31T22:49:56Z

After looking into implementing full-blown fetch with the Streams API, I agree with you, @adamchel. As covered above, this aligns with a Web API standard (YAY FOR WORKERS AND RNWEB) and deviates from node (YAY FOR STREAMS). There are valid cases for making streams available as a part of core (other I/O, e.g. file streams, geolocation updates), but it's non-trivial for this specific ask.

Given the audience for RN and its platform footprint, I think that EventSource looks good and also appears to be the most straightforward to implement and test quickly (and also has long-term value, avoiding a hack).

Thank you for taking this up!

axemclion · 2019-02-01T03:00:18Z

On the Android side, we should also see if we can use this as a way to be able to switch out OKHTTP.

cpojer · 2019-02-01T11:08:36Z

Thanks for this productive discussion. I think we have reached a consensus that React Native's native code needs to change to support this, and we are open to receiving PRs for that. I will close this issue here given that we agree, but feel free to continue the discussion here until a PR is sent. As @axemclion is pointing out, ideally this will also come with Android support.

I am fine with the second option, even if the first option would be ideal for long term support.

Summary: This PR introduces the `EventSource` web standard as a first-class networking feature in React Native. In the discussion we had in February at react-native-community/discussions-and-proposals#99, cpojer indicated that the RN maintainers would be willing to accept a PR to offer this functionality. The linked discussion goes into detail about why this change must happen in React Native Core as opposed to a community library, but the tl;dr is that `XmlHttpRequest` doesn't let you do streaming in a resource-efficient way, since it holds onto the entire response buffer until the request is complete. When processing a stream that might last for a long time, that's not ideal since there might be a lot of data in that buffer that is now useless to maintain. For more information about EventSource and server-sent events, check out these links: * [EventSource on MDN](https://developer.mozilla.org/en-US/docs/Web/API/EventSource) * [Using server-sent events on MDN](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) * [WHATWG spec for server-sent events](https://html.spec.whatwg.org/multipage/server-sent-events.html) I've tried as best as I can to satisfy the linked specification so that this is as standard as possible. One of the projects I maintain has an ideal use case for this feature. The SDK for MongoDB Stitch (a backend-as-a-service for the MongoDB database) has the ability to open a "change stream" to watch for changes that happen on a database. However, in our JavaScript SDK, this feature depends on `EventSource`, because the backend service implements the one-way streaming protocol with server-sent events. We know there is demand for this feature because users have requested it: mongodb/stitch-js-sdk#209. If this PR will be accepted, I am happy to update the `Networking` documentation at https://facebook.github.io/react-native/docs/network ## Changelog [JavaScript] [Added] Implements the `EventSource` web standard in `Libraries/Networking` [JavaScript] [Added] Exposes the `EventSource` implementation in `Libraries/Core/setUpXHR.js` Pull Request resolved: #25718 Test Plan: To test the `EventSource` implementation, I added a comprehensive set of unit tests that cover the basic functionality, as well as edge cases that are laid out in the spec. See `EventSource-test.js` for the cases that the tests handles. For convenience, I've also included the test descriptions as produced by the `jest` test output here. ``` PASS Libraries/Network/__tests__/EventSource-test.js EventSource ✓ should pass along the correct request parameters (527ms) ✓ should transition readyState correctly for successful requests (4ms) ✓ should call onerror function when server responds with an HTTP error (2ms) ✓ should call onerror on non event-stream responses (1ms) ✓ should call onerror function when request times out (1ms) ✓ should call onerror if connection cannot be established (1ms) ✓ should call onopen function when stream is opened (1ms) ✓ should follow HTTP redirects (2ms) ✓ should call onmessage when receiving an unnamed event (2ms) ✓ should handle events with multiple lines of data (1ms) ✓ should call appropriate handler when receiving a named event (1ms) ✓ should receive multiple events (1ms) ✓ should handle messages sent in separate chunks (1ms) ✓ should forward server-sent errors ✓ should ignore comment lines (1ms) ✓ should properly set lastEventId based on server message (1ms) ✓ should properly set reconnect interval based on server message ✓ should handle messages with non-ASCII characters (1ms) ✓ should properly pass along withCredentials option (3ms) ✓ should properly pass along extra headers (1ms) ✓ should properly pass along configured lastEventId (2ms) ✓ should reconnect gracefully and properly pass lastEventId (9ms) ✓ should stop attempting to reconnect after five failed attempts (2ms) ``` As a manual E2E test, I also added streaming support to the Stitch React Native SDK, and tested it with my React Native EventSource implementation, and confirmed that our `EventSource`-based streaming implementation worked with this `EventSource` implementation. * Source code for E2E app test: https://gist.github.com/adamchel/6db456c1a851ed7dd20b54f6db3a6759 * PR for streaming support on our React Native SDK: mongodb/stitch-js-sdk#294 * Very brief video demonstrating E2E functionality: https://youtu.be/-OoIpkAxmcw Differential Revision: D17283890 Pulled By: cpojer fbshipit-source-id: 0e9e079bdb2d795dd0b6fa8a9a9fa1e840245a51

cpojer · 2019-09-11T12:06:19Z

Update: RCTNetwork is will be exposed from react-native from 0.62 (next next release) onwards. See facebook/react-native#25718

Thanks @adamchel

rhoberman · 2019-12-12T12:25:28Z

@adamchel Thanks for driving this. We're using react-native-event-source and would be keen to switch. Happy to help test.

adamchel · 2020-01-30T17:18:29Z

Hey everyone! I've just published a package that exposes EventSource using the network primitives exposed in RN 0.62 and beyond!

https://www.npmjs.com/package/rn-eventsource
https://github.com/adamchel/rn-eventsource

I tested it with v0.62.0-rc.1 and it seems to work!

kovkev · 2020-05-05T05:42:40Z

I'm having trouble getting this to work at all on my Android device and on my Android emulator. The server does receive the GET request, but the client never receives any messages. I'm not sure how to troubleshoot or debug this at all.

kovkev · 2020-05-06T01:40:52Z

It works on ios and ios emulator, it works in Chrome App on ios and Android. I tried using mitmweb but that didn't show the returned Events.

In the Android Studio Network Traffic Monitoring tool, the returned events do not show up.

tail -f /var/log/nginx/error.log doesn't seem to show any issues.

Here are the differences in the headers sent in the request:

Android
{
"x-forwarded-for":"ip",
"host":"mysite.com",
"connection":"close",
"cache-control":"no-store",
"accept":"text/event-stream",
"user-agent":"okhttp/3.14.1"
"accept-encoding":"gzip",
}

iphone:
{
"x-forwarded-for":"ip",
"host":"mysite.com",
"connection":"close",
"cache-control":"no-store"
"accept":"text/event-stream",
"user-agent":"AppName/1 CFNetwork/1121.2.2 Darwin/19.3.0",
"accept-encoding":"gzip, deflate, br",
"accept-language":"en-us",

}

elicwhite · 2020-05-06T01:54:17Z

@kovkev, I don't think this is the right location for your issue as this was a proposal posted over a year ago and finished. If you are having an issue with RCTNetworking then opening an issue in the main repo would be the right choice. If you are having issues with eventsource, opening an issue in https://github.com/adamchel/rn-eventsource is probably a good choice.

kovkev · 2020-05-06T01:58:23Z

I'm not sure if the issue is with RCTNetworking or with rn-eventsource. My guess is that since this is failing with both rn-eventsource and with simple xhr, that the issue is in RCTNetworking?

adamchel mentioned this issue Jan 29, 2019

sockets client with backend service mongodb/stitch-js-sdk#23

Closed

kelset added the 🗣 Discussion This label identifies an ongoing discussion on a subject label Jan 30, 2019

cpojer closed this as completed Feb 1, 2019

adamchel mentioned this issue Feb 4, 2019

watch method on RemoteMongoCollection in ReactNative mongodb/stitch-js-sdk#209

Closed

kelset mentioned this issue Mar 8, 2019

ArrayBuffer over RN Bridge #107

Closed

adamchel mentioned this issue Jul 18, 2019

Expose RCTNetworking as a public 'Networking' API facebook/react-native#25718

Closed

Schwartz10 mentioned this issue Nov 19, 2019

react native fetch does not support HTTP streaming APIs infinitescroll/react-native-ipfs-http-client#1

Open

tDuy mentioned this issue Dec 28, 2019

should RemoteMongoCollection.watch work when using MongoDB Stitch React Native SDK? mongodb/stitch-js-sdk#348

Closed

snyk-bot mentioned this issue Feb 23, 2020

[Snyk] Upgrade mongodb-stitch-browser-sdk from 4.0.15-0 to 4.9.0 saurabharch/docs-tools#3

Open

kovkev mentioned this issue May 6, 2020

With React-Native, SSE aka EventSource does not receive Events on Android facebook/react-native#28835

Closed

acostalima mentioned this issue Oct 27, 2020

Technical planning and discussion react-native-community/fetch#4

Open

acostalima mentioned this issue Nov 9, 2020

Track: react native support ipfs/js-ipfs#2813

Closed

2 tasks

RayHughes mentioned this issue Nov 30, 2023

Android locks up / does not get receive all initial data (release version) binaryminds/react-native-sse#33

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for server-sent events (EventSource) #99

Support for server-sent events (EventSource) #99

adamchel commented Jan 29, 2019

elicwhite commented Jan 30, 2019

kelset commented Jan 30, 2019

adamchel commented Jan 30, 2019 •

edited

Loading

cpojer commented Jan 31, 2019

adamchel commented Jan 31, 2019 •

edited

Loading

elicwhite commented Jan 31, 2019

adamchel commented Jan 31, 2019

dkaminsky commented Jan 31, 2019

elicwhite commented Jan 31, 2019

adamchel commented Jan 31, 2019

adamchel commented Jan 31, 2019

elicwhite commented Jan 31, 2019

adamchel commented Jan 31, 2019

jhampton commented Jan 31, 2019

axemclion commented Feb 1, 2019

cpojer commented Feb 1, 2019

cpojer commented Sep 11, 2019

rhoberman commented Dec 12, 2019

adamchel commented Jan 30, 2020

kovkev commented May 5, 2020

kovkev commented May 6, 2020 •

edited

Loading

elicwhite commented May 6, 2020

kovkev commented May 6, 2020

Support for server-sent events (EventSource) #99

Support for server-sent events (EventSource) #99

Comments

adamchel commented Jan 29, 2019

Introduction

The Core of It

Existing Polyfill 1 : eventsource

Existing Polyfill 2: react-native-event-source

Existing Polyfill 3: react-native-eventsource

Discussion points

elicwhite commented Jan 30, 2019

kelset commented Jan 30, 2019

adamchel commented Jan 30, 2019 • edited Loading

cpojer commented Jan 31, 2019

adamchel commented Jan 31, 2019 • edited Loading

elicwhite commented Jan 31, 2019

adamchel commented Jan 31, 2019

dkaminsky commented Jan 31, 2019

elicwhite commented Jan 31, 2019

adamchel commented Jan 31, 2019

adamchel commented Jan 31, 2019

elicwhite commented Jan 31, 2019

adamchel commented Jan 31, 2019

jhampton commented Jan 31, 2019

axemclion commented Feb 1, 2019

cpojer commented Feb 1, 2019

cpojer commented Sep 11, 2019

rhoberman commented Dec 12, 2019

adamchel commented Jan 30, 2020

kovkev commented May 5, 2020

kovkev commented May 6, 2020 • edited Loading

elicwhite commented May 6, 2020

kovkev commented May 6, 2020

adamchel commented Jan 30, 2019 •

edited

Loading

adamchel commented Jan 31, 2019 •

edited

Loading

kovkev commented May 6, 2020 •

edited

Loading