Blobs #11103

philikon · 2016-11-24T00:55:20Z

Here at Silk Labs we've had a Blob implementation for React Native for a while. We want to contribute it back upstream. This issue discusses the API and implementation and tracks progress.

Motivation

Like in the browser, Blobs are opaque lumps of (usually binary) data that can be read into TypedArrays or strings, but don't have to be. Concretely in React Native that means that the binary data lives in native and not in JS, unless you explicitly request it. Why is this useful? Say you're receiving some data via XHR, WebSocket, or WebRTC, or some custom method that you've implemented and you want to display it in an <Image> view. Or you want to send it along through some other means without looking paying the cost of transferring it to the JS thread and back.

We already have something somewhat similar to this for images on iOS (RCTImageStoreManager). This would be a generalization of that.

API

The web has an API for Blobs. It's not particularly pretty or anything, but it's kind of a standard, so I think we should use that, or at least a pretty good subset.

Creating

Blobs typically get created when JS is receiving data, e.g. via XHR, WebSocket, etc. I am planning on making

xhr.responseType = 'blob';

and

websocket.binaryType = 'blob';

work so that when you specify those settings, the XHR and WebSocket responses will return Blob objects and not transfer the data to the JS thread. This works exactly like on the web.

Mutating

A subset of mutation operations will be supported:

let combinedBlob = new Blob([existingBlob, anotherBlob]);
let halfBlob = blob.slice(0, blob.size / 2);

I'm not planning on supporting creating a blob in JS from JS data (e.g. a string or TypedArray), though it should be easily implementable.

Consuming

Like in the browser, you'll be able to map Blobs to URIs that you can then use in things that eat URIs, e.g. the view:

let url = URL.createObjectURL(blob);

Also like in the browser, you'll be able to read the blob into a TypedArray or string. I am planning on simply implementing the web's FileReader API.

Freeing

The following method is part of the Blob standard, but is currently not implemented by browsers. However, because we can't automatically garbage collect our blobs like browsers can, it will be necessary to provide a manual way of freeing blobs, so that consumers can make use of it when appropriate:

blob.close();

Here's a complete example of how consuming blobs would work: https://gist.github.com/philikon/4592efd153673f3e64dec353046af7a8

Implementation

Proposal

React Native already has a loose concept of a "blob", in other words, a lump of data that's not directly accessible by JS. The iOS (RCTImageStoreManager) mentioned above is an example.

XHR and most notably the <Image> tag already have a way to express a reference to a blob:

{uri: "some://uri/that/gets/resolved"}

How does some://uri/that/gets/resolved get resolved? On iOS, it gets resolved through a RCTURLRequestHandler. On Android, it gets resolved through a ContentProvider.

For instance, you can upload binary data provided by a system resource in the following way:

const xhr = new XMLHttpRequest();
xhr.open(...);
xhr.send({uri: "some://big/blob/of/binary/data"})

As long as some://big/blob/of/binary/data is resolvable by a RCTURLRequestHandler or ContentProvider, this will Just Work(tm) already.

On this basis, I'm proposing to implement Blobs as simply another way of URI-addressable binary data. Our implementation does exactly that.

Progress

Prep work:

Make XMLHttpRequest and XMLHttpRequest.upload proper EventTargets so that we can receive progress events even when we're not receiving data to JS, because we're receiving it into a blob (~~Make XMLHttpRequest and XMLHttpRequest.upload proper EventTargets #7017~~)
Add responseType as a concept to RCTNetworking, so that we can add 'blob' later (~~Add responseType as a concept to RCTNetworking, send binary data as base64 #8324~~)

Implementation work, most likely by porting silk-react-native-blobs:

Provide native blob storage with NativeModule API, and JS implementation of Blob class
Provide URI integration (RCTURLRequestHandler on iOS, ContentProvider on Android)

Future implementation work:

XHR integration: allow xhr.responseType = 'blob' and xhr.send(blob)
WebSocket integration: allow websocket.binaryType = 'blob' and websocket.send(blob)
Implement FileReader API for reading blobs into JS e.g. as TypedArrays. This can just be a thin wrapper around XHR, yay!

The text was updated successfully, but these errors were encountered:

ide · 2016-11-24T01:04:58Z

Can you coordinate with @satya164, he has a Blob implementation almost ready I believe.

philikon · 2016-11-24T01:09:31Z

Yep, that's the reason for this issue :), so that we can coordinate. I believe he's uplifting our Blob implementation.

satya164 · 2016-11-24T15:23:00Z

Hey! Yeah, so I've been working to port changes from @philikon's implementation to RN master.

I've only worked on Android part so far.

My focus was to first get it working with firebase so people could upload files to firebase. This has slightly different requirements since we need to create a blob locally from a file before uploading.

So I think in addition to the proposed APIs, we need two more non-standard methods,

Blob.fromURI(uri: string, options: { type: string }): Promise<Blob>
File.fromURI(uri: string, options: { type: string }): Promise<File>

I just copied @philikon's implementation for the blob module and with minor changes, and writing some code to support XHR I've been able to get the following working,

Create blobs when data is received via XHR
Create blobs from a local file URI
Upload blobs to a server (e.g. - firebase) via XHR

I couldn't get URL.createObjectURL working from the current implementation, looking into it right now.

I've not yet tested the WebSocket implementation yet, but since that was already implemented I guess shouldn't take much time to get it working.

So things I need to do are,

Get content provider working
Get the WebSocket implementation working

What I'd like to explore is, right now when you create a blob from URI, I read it to memory as bytes. But we could probably just store the uri and read the bytes when needed (we'd need get the bytes length though, so we'd need to read it twice!)

I also want to know if there's a full test suite available to test my implementation against.

philikon · 2016-11-26T04:49:19Z

So I think in addition to the proposed APIs, we need two more non-standard methods,

Blob.fromURI(uri: string, options: { type: string }): Promise
File.fromURI(uri: string, options: { type: string }): Promise

I don't understand the need for File. What use cases does File solve that aren't covered by Blob?
What specific use cases does Blob.fromURI(...) solve? If I have a URI, why wouldn't I make an informal {uri: ...} blob object? If there's an RCTURLRequestHandler / ContentProvider for the URI in question, it'll already work with XHR! The only thing that we gain with Blob support is concatenation and slicing, but as you say, that's really hard to do if we don't at least know the size of each part. I'm not familiar with Firebase, but I'd like to hear more about why a generic URI-based Blob is a good idea. For now it seems pretty hard to implement Blob semantics around any lump of data that's not already stored in memory. Maybe a different API that also happens to expose uri for XHR-compatibility would be better suited for Firebase integration?
It also sounds like you might be thinking about this backwards. If people want to create blobs from some sort of datasource, the datasource should return Blob objects, e.g. someFirebaseAPI.get(options).then((blob) => ...). Or would Blob.fromURI() be an API that's meant only for React Native modules, much like the non-standard Blob.create() I introduced?

satya164 · 2016-11-26T05:27:33Z

I don't understand the need for File. What use cases does File solve that aren't covered by Blob

File implements the File API from the spec. It's just a thin wrapper around blob.

What specific use cases does Blob.fromURI(...) solve? If I have a URI, why wouldn't I make an informal {uri: ...} blob object? If there's an RCTURLRequestHandler / ContentProvider for the URI in question, it'll already work with XHR! The only thing that we gain with Blob support is concatenation and slicing, but as you say, that's really hard to do if we don't at least know the size of each part. I'm not familiar with Firebase, but I'd like to hear more about why a generic URI-based Blob is a good idea. For now it seems pretty hard to implement Blob semantics around any lump of data that's not already stored in memory. Maybe a different API that also happens to expose uri for XHR-compatibility would be better suited for Firebase integration?

Libraries like firebase expect a blob or file object for uploading. We cannot pass a { uri: string } object to firebase. So allowing to create a file/blob object from an URI is the only way people can use firebase for now. I don't like it much, but I don't think there are any options.

Firebase API looks like this,

const ref = firebase.storage().ref().child('image.jpg');
const snapshot = await ref.put(blob);

It also sounds like you might be thinking about this backwards. If people want to create blobs from some sort of datasource, the datasource should return Blob objects, e.g. someFirebaseAPI.get(options).then((blob) => ...). Or would Blob.fromURI() be an API that's meant only for React Native modules, much like the non-standard Blob.create() I introduced?

This is so that we'll be able to pass blob objects to libraries which don't accept uris. I think in general this should be discouraged though, and libraries can internally use this API if they need.

philikon · 2016-11-26T05:43:59Z

File implements the File API from the spec. It's just a thin wrapper around blob.

Meh ok. Seems unnecessary, unless there's demand from 3rd party libraries out there. (Perhaps Firebase?)

Libraries like firebase expect a blob or file object for uploading.

Ok, so then just do

const response = await fetch(uri);
const blob = await response.blob();
someFirebaseAPI.consume(blob);

Or in XHR spelling:

const xhr = new XMLHttpRequest();
xhr.open('GET', uri);
xhr.responseType = 'blob';
xhr.onload = () => {
  const blob = xhr.response;
  someFirebaseAPI.consume(blob);
};
xhr.send();

Either variant will load the data from the URI into a Blob that can then be passed to Firebase. Though, I still don't quite understand what Firebase is going to do with it? Upload it to the server?

satya164 · 2016-11-26T06:03:49Z

Meh ok. Seems unnecessary

It's very tiny amount code to polyfill a web API, which makes sure libs which need File constructor will work.

Either variant will load the data from the URI into a Blob that can then be passed to Firebase.

I didn't know I can do fetch from local URI. Or are you proposing to implement this API?

Though, I still don't quite understand what Firebase is going to do with it? Upload it to the server?

Yes

philikon · 2016-11-26T06:08:48Z

I didn't know I can do fetch from local URI.

Yep, should already work!

(* Definitely on iOS. No idea about Android. If XHR can't currently download from content:// URIs, then we should add that support.)

philikon · 2016-11-26T06:09:13Z

It's very tiny amount code to polyfill a web API, which makes sure libs which need File constructor will work.

Sure, ok.

brentvatne · 2016-11-27T01:05:41Z

It also sounds like you might be thinking about this backwards. If people want to create blobs from some sort of datasource, the datasource should return Blob objects, e.g. someFirebaseAPI.get(options).then((blob) => ...). Or would Blob.fromURI() be an API that's meant only for React Native modules, much like the non-standard Blob.create() I introduced?

I had the same thought, the issue that @satya164 pointed out to me is that if the datasource API (let's say ImagePicker which lets you pick an image from your camera roll or take a photo) returns a Blob every time then people who are not interested in the Blob but rather in other other values like the URI might just forget to close the Blob. We could make it an optional parameter on those APIs:

let imageBlob = ImagePicker.launchImageLibraryAsync({
  allowsEditing: false,
  aspect: [1, 2],
  responseType: 'blob',
})

Thoughts @satya164 @philikon?

(the ImagePicker I refer to here is the one in Exponent - see docs if you're curious)

ide · 2016-11-27T09:56:03Z

Another approach would be to write a separate module that creates a Blob from a URI -- BlobLoader.loadFromURIAsync(uri). This would keep the function out of the Blob/FileSystem namespace but still let you work with any API that gives you back a local URI.

That said, one thing I like about libraries that return Blobs (like Brent's ImagePicker example) is that you either get back a valid Blob (assuming it's eagerly loaded) or the call fails. There's no uncertain state halfway where you get back a valid URI but then the file on disk gets deleted before you load the Blob from the URI.

So we might want both -- Blob-aware APIs that give you back Blobs in a more atomic way, and a generic URI-to-Blob loader that works with any URI-aware API.

satya164 · 2016-11-28T03:37:18Z

Another approach would be to write a separate module that creates a Blob from a URI

Separate module for creating blobs from URI sounds good to me

We could make it an optional parameter on those APIs

That could work, it only saves one line of code of code though (not really since you've to specify responseType: 'blob' anyways).

While the blob-aware APIs seem nice to me in a general sense, in case of RN I'm not super happy about them since the user always has to manually free the blob. . One thing I wanted to explore is to lazy-load URI based blobs. But this negates the case @ide mentioned about not having to deal with uncertain state.

hramos · 2017-05-25T20:36:12Z

Closing this issue because it has been inactive for a while. If you think it should still be opened let us know why.

Summary: This is the first PR from a series of PRs grabbou and me will make to add blob support to React Native. The next PR will include blob support for XMLHttpRequest. I'd like to get this merged with minimal changes to preserve the attribution. My next PR can contain bigger changes. Blobs are used to transfer binary data between server and client. Currently React Native lacks a way to deal with binary data. The only thing that comes close is uploading files through a URI. Current workarounds to transfer binary data includes encoding and decoding them to base64 and and transferring them as string, which is not ideal, since it increases the payload size and the whole payload needs to be sent via the bridge every time changes are made. The PR adds a way to deal with blobs via a new native module. The blob is constructed on the native side and the data never needs to pass through the bridge. Currently the only way to create a blob is to receive a blob from the server via websocket. The PR is largely a direct port of https://github.com/silklabs/silk/tree/master/react-native-blobs by philikon into RN (with changes to integrate with RN), and attributed as such. > **Note:** This is a breaking change for all people running iOS without CocoaPods. You will have to manually add `RCTBlob.xcodeproj` to your `Libraries` and then, add it to Build Phases. Just follow the process of manual linking. We'll also need to document this process in the release notes. Related discussion - #11103 - `Image` can't show image when `URL.createObjectURL` is used with large images on Android The websocket integration can be tested via a simple server, ```js const fs = require('fs'); const http = require('http'); const WebSocketServer = require('ws').Server; const wss = new WebSocketServer({ server: http.createServer().listen(7232), }); wss.on('connection', (ws) => { ws.on('message', (d) => { console.log(d); }); ws.send(fs.readFileSync('./some-file')); }); ``` Then on the client, ```js var ws = new WebSocket('ws://localhost:7232'); ws.binaryType = 'blob'; ws.onerror = (error) => { console.error(error); }; ws.onmessage = (e) => { console.log(e.data); ws.send(e.data); }; ``` cc brentvatne ide Closes #11417 Reviewed By: sahrens Differential Revision: D5188484 Pulled By: javache fbshipit-source-id: 6afcbc4d19aa7a27b0dc9d52701ba400e7d7e98f

Summary: This is the first PR from a series of PRs grabbou and me will make to add blob support to React Native. The next PR will include blob support for XMLHttpRequest. I'd like to get this merged with minimal changes to preserve the attribution. My next PR can contain bigger changes. Blobs are used to transfer binary data between server and client. Currently React Native lacks a way to deal with binary data. The only thing that comes close is uploading files through a URI. Current workarounds to transfer binary data includes encoding and decoding them to base64 and and transferring them as string, which is not ideal, since it increases the payload size and the whole payload needs to be sent via the bridge every time changes are made. The PR adds a way to deal with blobs via a new native module. The blob is constructed on the native side and the data never needs to pass through the bridge. Currently the only way to create a blob is to receive a blob from the server via websocket. The PR is largely a direct port of https://github.com/silklabs/silk/tree/master/react-native-blobs by philikon into RN (with changes to integrate with RN), and attributed as such. > **Note:** This is a breaking change for all people running iOS without CocoaPods. You will have to manually add `RCTBlob.xcodeproj` to your `Libraries` and then, add it to Build Phases. Just follow the process of manual linking. We'll also need to document this process in the release notes. Related discussion - facebook/react-native#11103 - `Image` can't show image when `URL.createObjectURL` is used with large images on Android The websocket integration can be tested via a simple server, ```js const fs = require('fs'); const http = require('http'); const WebSocketServer = require('ws').Server; const wss = new WebSocketServer({ server: http.createServer().listen(7232), }); wss.on('connection', (ws) => { ws.on('message', (d) => { console.log(d); }); ws.send(fs.readFileSync('./some-file')); }); ``` Then on the client, ```js var ws = new WebSocket('ws://localhost:7232'); ws.binaryType = 'blob'; ws.onerror = (error) => { console.error(error); }; ws.onmessage = (e) => { console.log(e.data); ws.send(e.data); }; ``` cc brentvatne ide Closes facebook/react-native#11417 Reviewed By: sahrens Differential Revision: D5188484 Pulled By: javache fbshipit-source-id: 6afcbc4d19aa7a27b0dc9d52701ba400e7d7e98f

austinksmith · 2017-10-03T15:06:21Z

@hramos Please reopen , blob support is very critical to my current issue. #16034 (comment)

satya164 · 2017-10-03T15:16:44Z

Theres no point in re-opening. There's already a WIP PR implementing it.

austinksmith · 2017-10-03T15:20:01Z

Can you post that here and on the ticket I've opened? ETA on completion etc.?

satya164 · 2017-10-03T15:28:09Z

There's no ETA. #11573

austinksmith · 2017-10-03T17:57:59Z

So how can we get this issue prioritized, it appears that PR has been in limbo for almost a year

satya164 · 2017-10-03T19:09:04Z

Only way is to help me with the iOS code and then wait for review.

austinksmith · 2017-10-06T14:07:47Z

@satya164 What is needed on IOS side?

pradeep250677 · 2018-01-16T05:55:29Z

@satya164 could you share me with Andriod code? Binary data communication API between JSC and Native using RN bridge is very important. I am the only concern for Andriod.

satya164 · 2018-01-16T06:57:24Z

@pradeep250677 you can check the pr linked above.

brentvatne mentioned this issue Dec 2, 2016

[fetch] Does fetch with blob() marshal data across the bridge? #854

Closed

satya164 mentioned this issue Dec 20, 2016

[BREAKING on iOS] Add blob implementation with WebSocket integration #11417

Closed

hramos closed this as completed May 25, 2017

hramos added the Icebox label May 26, 2017

facebook locked as resolved and limited conversation to collaborators Jul 19, 2018

react-native-bot added the Resolution: Locked This issue was locked by the bot. label Jul 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Blobs #11103

Blobs #11103

philikon commented Nov 24, 2016 •

edited

Loading

ide commented Nov 24, 2016

philikon commented Nov 24, 2016

satya164 commented Nov 24, 2016 •

edited

Loading

philikon commented Nov 26, 2016

satya164 commented Nov 26, 2016 •

edited

Loading

philikon commented Nov 26, 2016 •

edited

Loading

satya164 commented Nov 26, 2016 •

edited

Loading

philikon commented Nov 26, 2016 •

edited

Loading

philikon commented Nov 26, 2016

brentvatne commented Nov 27, 2016 •

edited

Loading

ide commented Nov 27, 2016

satya164 commented Nov 28, 2016

hramos commented May 25, 2017

austinksmith commented Oct 3, 2017

satya164 commented Oct 3, 2017

austinksmith commented Oct 3, 2017

satya164 commented Oct 3, 2017

austinksmith commented Oct 3, 2017

satya164 commented Oct 3, 2017

austinksmith commented Oct 6, 2017

pradeep250677 commented Jan 16, 2018

satya164 commented Jan 16, 2018

Blobs #11103

Blobs #11103

Comments

philikon commented Nov 24, 2016 • edited Loading

Motivation

API

Creating

Mutating

Consuming

Freeing

Implementation

Proposal

Progress

ide commented Nov 24, 2016

philikon commented Nov 24, 2016

satya164 commented Nov 24, 2016 • edited Loading

philikon commented Nov 26, 2016

satya164 commented Nov 26, 2016 • edited Loading

philikon commented Nov 26, 2016 • edited Loading

satya164 commented Nov 26, 2016 • edited Loading

philikon commented Nov 26, 2016 • edited Loading

philikon commented Nov 26, 2016

brentvatne commented Nov 27, 2016 • edited Loading

ide commented Nov 27, 2016

satya164 commented Nov 28, 2016

hramos commented May 25, 2017

austinksmith commented Oct 3, 2017

satya164 commented Oct 3, 2017

austinksmith commented Oct 3, 2017

satya164 commented Oct 3, 2017

austinksmith commented Oct 3, 2017

satya164 commented Oct 3, 2017

austinksmith commented Oct 6, 2017

pradeep250677 commented Jan 16, 2018

satya164 commented Jan 16, 2018

philikon commented Nov 24, 2016 •

edited

Loading

satya164 commented Nov 24, 2016 •

edited

Loading

satya164 commented Nov 26, 2016 •

edited

Loading

philikon commented Nov 26, 2016 •

edited

Loading

satya164 commented Nov 26, 2016 •

edited

Loading

philikon commented Nov 26, 2016 •

edited

Loading

brentvatne commented Nov 27, 2016 •

edited

Loading