Implement response.body as a stream

[pcowgill]

Response has no .body property available which is needed as a getter to expose a ReadableStream of the body contents.

https://developer.mozilla.org/en-US/docs/Web/API/Body/body

[pcowgill]

@mislav @josh @dgraham Would you be open to a PR for this if I can get it implemented in a fork?

Since React Native uses this code to implement fetch, it would be nice if the fetch API in React Native supported streams.

[pcowgill]

I'm also curious to hear what the React Native folks think of this idea @arthuralee @janicduplessis @cpojer

[pcowgill]

Finally, I thought I'd rope in some of the people contributing to the streams standard. @domenic @ricea @MattiasBuelens

Thanks!

[MattiasBuelens]

From this comment on #198, it seems like there is no interest in adding streaming support to this polyfill.

There are other fetch polyfills that do support streaming, such as fetch-readablestream or @stardazed/streams-polyfill. (I also made my own, but it's not very well maintained. 😅) If possible, you may want to consider switching to one of those.

[MattiasBuelens]

Since React Native uses this code to implement fetch, it would be nice if the fetch API in React Native supported streams.

Note that if the underlying platform does not support streaming responses, then the only thing a polyfill can do is create a ReadableStream that contains a single chunk with the final response body. So yes, while you can use a polyfill to add Response.body, it would behave just the same as using Response.arrayBuffer() directly.

If you want actual streaming support in React Native, then React Native would need to implement it itself.

[pcowgill]

@MattiasBuelens Thanks for the prompt and thoughtful response!

There are other fetch polyfills that do support streaming, such as fetch-readablestream

fetch-readablestream passes through res.body from the underlying fetch implementation, so that won't help with having .body.getReader() be defined, will it?

If you want actual streaming support in React Native, then React Native would need to implement it itself.

Since React Native uses this package as its implementation of fetch, wouldn't landing it here address that the next time React Native core upgrades the dep? Or is there something lower level that would need to change?

[MattiasBuelens]

fetch-readablestream passes through res.body from the underlying fetch implementation, so that won't help with having .body.getReader() be defined, will it?

It only does that if the underlying fetch implementation actually supports Response.body. Otherwise, it will use either XHR with response type moz-chunked-arraybuffer (for "true" streaming support on older Firefox versions), or XHR with responseText (for "pseudo-streaming" support on all browsers, but with fairly bad performance due to string concatenation). Check the source code for more details.

Since React Native uses this package as its implementation of fetch, wouldn't landing it here address that the next time React Native core upgrades the dep? Or is there something lower level that would need to change?

A fetch polyfill is merely a wrapper around XMLHttpRequest. It can't "magically" add streaming support: the underlying platform needs to provide some low-level primitive that can be used to stream a response (such as moz-chunked-arraybuffer or responseText). React Native only implements XMLHttpRequest, so if that does not have any way to do streaming, then a fetch polyfill built on top of that won't be able to do streaming either.

If React Native wanted to support streaming, they would need to add something to their XMLHttpRequest implementation that a fetch polyfill can pick up. However, since there is no standardized API for streaming in XHR, this would require a non-standard extension to their XHR API, which is unlikely to be integrated into a general-purpose fetch polyfill like this one.

The "proper" way would be for React Native to implement fetch themselves on top of native APIs, rather than using a fetch polyfill built on top of XHR.

That said, I had a quick look at the source code for their XHR, and it would seem like they should support "pseudo-streaming" with responseText. So perhaps fetch-readablestream could automatically use that to polyfill Response.body when run inside React Native? It wouldn't be very efficient though, I think. 🤷‍♂

[cpojer]

Could this be added separate from fetch as an additional plugin similar to the then/promise module? See https://github.com/then/promise

Then people could do something like:

require('fetch');
require('fetch/body');

where the latter would patch the getter into the former. What do you think?

[hugomrdias]

Note that if the underlying platform does not support streaming responses, then the only thing a polyfill can do is create a ReadableStream that contains a single chunk with the final response body. So yes, while you can use a polyfill to add Response.body, it would behave just the same as using Response.arrayBuffer() directly.

This would be enough for a first step.

@cpojer do you think something like the above could be added to react native? if yes whats the best approach? PR this repo? make a new package patching whatwg-fetch like you said ? directly to react-native network code?

[mislav]

This polyfill was originally intended to be used in web browsers that had no support for the fetch standard. Our target is basically pre-2015 web browsers, and every feature that we add to the polyfill had to be reasonably implementable using the capabilities of those browsers, which was mostly reliant on building on top of XMLHttpRequest.

React Native runs under JavaScriptCore, which I have no familiarity with, but I would like to assume that it's a more modern environment than a pre-2015 web browser. I don't think that including this polyfill in React Native was a great call, since we never targeted this environment, and if React Native chose to maintain their own implementation (or use another implementation that targets JavaScriptCore), I think implementing more modern features such as streaming would be more feasible.

TL;DR; we likely won't implement streaming here because we don't want to find out what what would be involved in making this functionality work seamlessly in older browsers.

[pcowgill]

So it sounds like in the long run we need React Native core to implement fetch, and in the short run the React Native core team ought to fork this repo so we can make PRs against the fork without fear of breaking older browsers?

[pcowgill]

It appears that fetch-readablestream won't work well if you're using the fetch API via ky, because ky assumes response.clone() will be defined, and it's not when using fetch-readablestream in a React Native env.

[pcowgill]

@stardazed/streams-polyfill has clone() implemented, and it seems to be working

[cpojer]

fetch is not implemented at the VM level, it's implemented by the host platform (browsers). React Native uses this polyfill directly on top of XMLHttpRequest which itself is implemented using Networking (android implementation, ios implementation), a JS module that exposes native networking functionality for each platform.

It is likely that React Native's networking stack will need changes to support newer features and even better would be native support for fetch, but we are not currently working on that. Is the question here whether we'd accept contributions? If so, yes, definitely! Especially if it unblocks progress for this repo, if people are still interested in maintaining this polyfill.

[hugomrdias]

@cpojer from the comments above it's gonna be hard to add more stuff to this repo.
I'll move this discussion to react-native repo.

[pcowgill]

@hugomrdias @cpojer React Native GitHub issues tend to get closed to comments (totally understandable, but perhaps not ideal for a long-term topic like this). Maybe if we could set up a fork of this repo under the facebook GitHub org with a more liberal issue-closing policy, that would be better? Thanks!

[pcowgill]

@hugomrdias @cpojer React Native GitHub issues tend to get closed to comments (totally understandable, but perhaps not ideal for a long-term topic like this). Maybe if we could set up a fork of this repo under the facebook GitHub org with a more liberal issue-closing policy, that would be better? Thanks!

@cpojer What do you think about this idea? Thanks!

[pcowgill]

Although this issue is closed, it's still an open question where to resume the conversation. So please keep commenting enabled for this issue. Thanks!

[qalqi]

In react-native body.constructor.name == 'Blob' gives true where as Blob.prototype.isPrototypeOf(body) gives false for blobs.

I am able to fix it as

  else if (body && body.constructor && body.constructor.name == 'Blob') {
        this._bodyText = body = body.text()
}

Add this condition to https://github.com/github/fetch/blob/7232090c04e1ddefb806910bbd0a756bc8aac2f0/fetch.js#L229

[pcowgill]

@qalqi With that change, response.body is defined?

[qalqi]

Yep.. Able to receive a response with ipfs 0.41.0 in react-native with this change

[qalqi]

Here is an another alternative approach it seems.. facebook/react-native#25701 (comment)

[pcowgill]

Yep.. Able to receive a response with ipfs 0.41.0 in react-native with this change

@qalqi Just to confirm, the goal of your change is to get response.body to be defined as a getter for a ReadableStream, is that right?

[qalqi]

Yeah. It’s resolves body.text() blob stream and sends as response. With this change, some ipfs 0.41.0 functions work.

[pcowgill]

Yeah. It’s resolves body.text() blob stream and sends as response. With this change, some ipfs 0.41.0 functions work.

@qalqi Thanks for getting back to me!

A) When using the go-ipfs daemon, the js-ipfs daemon, or either?

B) You mean when using the daemon from (A) via ipfs-http-client, right? The current ipfs-http-client version is 42.0.0 and the current js ipfs version is 0.41.2

[pcowgill]

@qalqi Do you remember specifically which functions are working for you? Thanks!

[qalqi]

A) go-ipfs daemon and js-ipfs daemon must be same
B) Yeah I am using ipfs-http-client with version 42.0.0

All get requests are working with ipfs-http-client after changing fetch logic as mentioned above.

Put requests are all failing with error Current request is not a multipart request

Here is react-native sandbox with working get requests https://github.com/qalqi/react-native-ipfs-http-client/tree/primary/rn-nodeify

[pcowgill]

@qalqi Thanks for getting back to me! I'll respond to you over in this new issue on the react-native-community/fetch repo react-native-community#3