Crashed to upload large files ( > 500MB): [Terminating app due to uncaught exception 'NSMallocException', reason: 'Failed to grow buffer'] #146

nikolaytsigvintsev · 2019-03-28T11:43:09Z

Hi.

I'm trying to upload a large file. My application crashes after launching the upload with the error:

Terminating app due to uncaught exception 'NSMallocException', reason: 'Failed to grow buffer'

On a small file size works well.

iPad Pro
Deployment Target : 12.1
React-Native v0.58

Code:

      const upload = new tus.Upload({ uri: 
        `${RNFS.DocumentDirectoryPath}/images/${item.image}`}, {
        endpoint: 'http://192.168.1.5:1080/files/',
        retryDelays: [0, 1000, 3000, 5000, 10000, 15000, 20000],
        resume: true,
        metadata: {
          name: item.image,
          filetype: item.image.substring(item.image.lastIndexOf('.') + 1).toUpperCase(),
          copyright: 30
        },
        onError: (error) => {
          console.log('Error upload:', error)
        },
        onProgress: (uploadedBytes, totalBytes) => {
          console.log(`Progress: ${uploadedBytes / totalBytes}%`)
        },
        onSuccess: () => {
          console.log('Upload URL:', upload.url)
        }
      })
      upload.start()

The text was updated successfully, but these errors were encountered:

Acconut · 2019-03-30T17:07:13Z

That's interesting, thanks for reporting. It seems to me as if tus-js-client is trying to read the entire file in memory before uploading it. This works fine if enough memory is available to cover a smaller file but obviously fails for bigger uploads.

I am not sure how to fix this inside tus-js-client out of my head now but maybe @arturi knows if React Native has a better way for streaming uploads.

@nikolaytsigvintsev: As a workaround you might want to add the chunkSize: 100 * 1024 * 1024 option when you create the tus.Upload instance. This will cause the upload to be split into multiple PATCH requests where at most 100MiB are uploaded per request. Therefore, not more than 100MiB should be necessary. Of course, this is only a workaround for now and we need to find a better solution.

nikolaytsigvintsev · 2019-04-02T07:26:00Z

Hi.

Thanks for the answer. I tried to install chunkSize, but it did not help.
Exactly the same mistake.

  const upload = new tus.Upload({ uri: `${RNFS.DocumentDirectoryPath}/images/${item.image}`}, {
        endpoint: 'http://192.168.1.5:1080/files/',
        retryDelays: [0, 1000, 3000, 5000, 10000, 15000, 20000],
        chunkSize: 100 * 1024 * 1024,
        resume: true,
        metadata: {
          name: item.image,
          filetype: item.image.substring(item.image.lastIndexOf('.') + 1).toUpperCase(),
          copyright: 30
        },
        onError: (error) => {
          console.log('Error upload:', error)
        },
        onProgress: (uploadedBytes, totalBytes) => {
          console.log(`Progress: ${uploadedBytes / totalBytes}`)
        },
        onSuccess: () => {
          console.log('Upload URL:', upload.url)
      })
      upload.start()

Acconut · 2019-04-03T19:41:15Z

Oh, I forgot. tus-js-client apparently tries to load the file into memory before it slices it using the chunkSize option. That explains why setting this parameter does not change anything for your situation. Frankly, I am not sure how to resolve this. I am not experienced enough with React Native to know if there is a way to read a fixed amount of bytes from a file at a specific offset. Not sure if @arturi had time to look into this yet.

In the meantime you might want to look into using https://github.com/vinzscam/react-native-tus-client which is a tus-js-client-like wrapper around the native Android and iOS tus clients. I have never tried them by myself but they should be able to handle bigger files.

nikolaytsigvintsev · 2019-04-03T20:35:46Z

Hi.

Thank you for taking the time to respond.

The tus-js-client documentation says that you can start the upload with the Readable stream option. I think that one of the ways out of this problem is to create a readable stream and send it to the upload function. Only I cannot create a readable stream (react-native) that I can send in the Upload function.
I created a question on Stakoverflow.

Thanks for help.

Acconut · 2019-04-05T20:15:00Z

The tus-js-client documentation says that you can start the upload with the Readable stream option.

We only implemented uploading from readable stream for the browsers and Node.js, so I doubt that this would work without modifications for React Native. We are happy to extend the support for streaming uploads to React Native if you would like to give it a try.

I created a question on Stakoverflow.

That's great, would you mind sharing a link with us?

nikolaytsigvintsev · 2019-04-25T12:32:23Z

Hi.

Thanks for the answer.

I do not know how I can help you. I am new to the react-native. I can try to make a function to upload file by stream, but tell me where to start.

link: https://stackoverflow.com/questions/55496609/tus-js-client-for-react-native-crashed-to-upload-large-files

kvz · 2019-04-25T12:41:07Z

Hi @nikolaytsigvintsev, we're working on this. We'll have a (very basic) example/version online over at the Uppy site, hopefully today.

nikolaytsigvintsev · 2019-04-26T12:54:04Z

Hi @kvz.

I am very glad that your work goes forward.
I noticed that @arturi added an example response-native-expo to the Uppy site (https://github.com/transloadit/uppy/tree/master/examples/react-native-expo). I took the code from the example and compiled it for 'react-native'. I think this is what I need.

Thanks and good luck.

kvz · 2019-05-06T07:59:31Z

Hi @nikolaytsigvintsev, I noticed you said it wasn't working for you in a different thread. it seems we had a typo in there earlier 😱 sad that we're not using TypoScript (pun intended :) ) for examples yet maybe 😹 @ifedapoolarewaju discovered and fixed it here: transloadit/uppy@854654b

We've tested this and confirmed it works on an Android simulator.

Sorry for the long and dangerous road we're putting you on! Would you mind seeing if this fixes it for you too?

Thanks for bearing with us!

nikolaytsigvintsev · 2019-05-08T10:47:29Z

Hi @kvz.

Thanks for the answer.
The program works on the simulator, because there may be more memory on the simulator that real device ?
On real device iPad Pro all works good until size file < 500 - 700MB
I tried upload file 542MB result good, on upload file 2.4G result failed.

Videoreport:
https://www.dropbox.com/s/1z8yy5v85mmy72m/ScreenRecording_05-08-2019%2013-31-57.MP4?dl=0

I'm trying to understand how the tusFileReader function works.
I inserted console.log() statements into the getTusFileReader function and into the slice() function.
The function slice() is not called.
As I understand it, the slice() function works for streaming uploading.

Thanks.

ifedapoolarewaju · 2019-05-08T11:34:52Z

@nikolaytsigvintsev there might be something off with your code. How are you instantiating Tus? Are you passing the tusFileReader into your TusOptions?

nikolaytsigvintsev · 2019-05-08T12:17:00Z

Hi @ifedapoolarewaju

Yes.

const upload = new tus.Upload({uri: ${RNFS.DocumentDirectoryPath}/images/${node.asset.url}}, {
endpoint: 'http://192.168.1.5:1080/files/',
retryDelays: [0, 1000, 3000, 5000],
resume: true,
fileReader: getTusFileReader,
metadata: {
name: node.asset.url,
filetype: node.image.substring(node.image.lastIndexOf('.') + 1).toUpperCase(),
copyright: 30
},
onError: (error) => {
console.log('Error upload:', error)
this.setState({isUploading: false})
},
onSuccess: () => {
console.log('Upload Success:')
this.setState({isUploading: false})
}
})
upload.start()

The getTusFileReader function is called, but slice () in class TusFileReader does not call

ifedapoolarewaju · 2019-05-08T12:20:53Z

what is the content of getTusFileReader ? Also what version of tus-js-client are you using this with?

nikolaytsigvintsev · 2019-05-08T12:29:10Z

version of tus-js-client:
"tus-js-client": "^1.8.0-0"

The getTusFileReader is function in tusFileReader.js:
https://github.com/transloadit/uppy/tree/master/examples/react-native-expo

My code for react-native:

import RNFetchBlob from 'rn-fetch-blob'
import RNFS from 'react-native-fs'
import base64 from 'base64-js'

export default function getTusFileReader(file, chunkSize, cb) {
RNFetchBlob.fs.stat(file.uri)
.then((stats) => {
console.log('Size file = ', stats.size)
cb(null, new TusFileReader(file, stats.size))
})
.catch(cb)
}

class TusFileReader {
constructor(file, size) {
this.file = file
this.size = size
}

slice(start, end, cb) {
console.log('Start slice:')
end = Math.min(end, this.size)
const options = {
length: end - start,
position: start
}
RNFS.read(this.file.uri, options.length, options.position, 'base64').then((data) => {
console.log('Position: ', ${start}:${end} - ${options.position} : ${options.length})
cb(null, base64.toByteArray(data))
}).catch(cb)
}

close() {

}
}

ifedapoolarewaju · 2019-05-08T12:38:01Z

so you are using rn-fetch-blob, not expo-file-system?

nikolaytsigvintsev · 2019-05-08T12:59:08Z

Yes.

I tried to solve my problem in order to upload large 1G-5G files for 'react-native'. I found an example on expo and took a snippet of code.

The example on expo also does not upload large files. I made a video report.
Maybe I'm doing something wrong ?

ifedapoolarewaju · 2019-05-09T21:37:52Z

@nikolaytsigvintsev the slice method exists so that we can read the file in chunks, and not the entire content. But the actual chunk reading is dependent on your implementation of the slice method. In the getTusFileReader example here, we are using Expo.FileSystem which ensures that the files are read in chunks. But from what I see in your code above, you are modifying it to use RNFetchBlob and RNFS, which I cannot guaranty that they will always read files in chunks.

So this error is due to your modification. In fact, based on the error in your logs above:
[Error: tus: cannot fetch file.uri as Blob, make sure the uri is correct and accessible. [object Object]], I can guess it is triggered by this portion of your code:

export default function getTusFileReader(file, chunkSize, cb) {
   RNFetchBlob.fs.stat(file.uri)
     .then((stats) => {
        console.log('Size file = ', stats.size)
        cb(null, new TusFileReader(file, stats.size))
      }).catch(cb)
}

This line particularly RNFetchBlob.fs.stat(file.uri)
so it fails right earlier on. This is why you never see the console logs in the slice method. Cos it never gets there.

nikolaytsigvintsev · 2019-05-10T09:47:44Z

Hi.

I partially solved the problem.

I added chunkSize, and everything began work, the file began to upload in chunks (version tus-js-client 1.8.0).
I made a separate thread for the upload process, where I am waiting for the upload to finish.

But all the same there is an error: Thread 47: EXC_RESOURCE RESOURCE_TYPE_MEMORY (limit = 1400 MB, unused = 0x0)

Position: start=981467136 : end=989855744 : chunk = 8388608

File size = 1.2GB

Somewhere there is a memory leak ?

nikolaytsigvintsev · 2019-05-10T14:08:22Z

Hi.

An apparent memory leak occurs with each call function cb(null, base64.toByteArray(data))
I think the problem will be in xhr objects or xhr.onload = function () {} ( file tus-js-client/lib.es5/upload.js ), but I could be wrong

Thanks.

ifedapoolarewaju · 2019-06-07T07:16:49Z

I added chunkSize, and everything began work

@nikolaytsigvintsev ah, I didn't realize that you weren't setting chunkSize. Yes, the chunkSize is required for it to work.

An apparent memory leak occurs with each call function cb(null, base64.toByteArray(data))

How can this memory leak be reproduced? Or maybe you could also share how you arrived at the theory?

nikolaytsigvintsev · 2019-06-07T09:33:08Z

Hi @ifedapoolarewaju

I am very glad that you took the time.

A memory leak occurs with each fragment. When debugging, I can see that there is a leak in the CFString module. I now do not know who is to blame or tus-js-client or React-Native

Thanks .

nikolaytsigvintsev · 2019-10-18T08:57:55Z

Hello, guys !

I was forced to return to the problem of uploading large files for REACT-NATIVE, and I found that a memory leak occurred while reading the file. While I dealing with this problem.

itinance/react-native-fs#785

Acconut · 2019-10-22T06:15:20Z

That's interesting, thanks for pointing it out, @nikolaytsigvintsev. If there is indeed a memory leak inside react-native-fs, it will be hard/impossible for tus-js-client to work around this. I am unsure if you will get a good support there as the repository has many open issues, but in all honesty, I don't know what to do about it.

nikolaytsigvintsev · 2019-11-25T14:27:18Z

Hello, guys!

After a long search for a solution to my problem ( answers from react-native developers were not received. facebook/react-native#26972 ) I returned to trying to upload files using tus-js-client without the chunk method (without fileReader: getTusFileReader parameter).
I was uploading many files of 40MB each.
But a memory leak and application crash were also detected. The application works while there is a memory.

On platform react-native:
video report:
https://www.dropbox.com/s/d7y596b5ba0wx9x/Screen%20Recording%202019-11-25%20at%2016.08.53.mov?dl=0
The expo application was installed for react-native and uploading files of 100MB each (
https://github.com/tus/tus-js-client/tree/master/demos/reactnative) and it was launched on iPad Pro
video report:

uploading and crash: https://www.dropbox.com/s/m2m4xajr7x6gh28/ScreenRecording_11-25-2019%2014-49-21.MP4?dl=0
logs server expo: https://www.dropbox.com/s/uuet13yfnmp386r/Screen%20Shot%202019-11-25%20at%2014.57.15.png?dl=0
log tus server: https://www.dropbox.com/s/kd93wg83fqh1w3f/Screen%20Recording%202019-11-25%20at%2012.57.30.mov?dl=0

Please tell me how can I solve this problem?

Thanks!

Acconut · 2019-12-02T14:44:41Z

I returned to trying to upload files using tus-js-client without the chunk method (without fileReader: getTusFileReader parameter).

I am wondering, why did you decide against the fileReader option? I thought that previously worked for you?

nikolaytsigvintsev · 2019-12-04T08:47:58Z

Hi Marius!
Thanks for the answer.

No. The uploading files doesn't work with fileReader: getTusFileReader parameter. I am reading files by means RNFS (react-native) by chunks and occur the memory leak. I have written about it on 10 May, 10 Jun, 18 Oct. I made a lot of tests and sent the description problem to the developers react-native and react-native-fs:

Memory leak in Native Modules with RCT_EXPORT_METHOD facebook/react-native#26972
[BUGS:] Memory leak and application crash on reading a large file itinance/react-native-fs#785
[BUG:] Memory leak and application crash RNFetchBlob.fs.readStream joltup/rn-fetch-blob#474

But the answer no.

I decided to turn off fileReader: getTusFileReader parameter and try to upload small files but faced the same problem. I have thought that I have mistakes in my code. Then I have downloaded the example your code (https://github.com/tus/tus-js-client/tree/master/demos/reactnative) but the same problem.

I showed it on video.
I'm at a loss.
Has anyone already had such a problem?

Thanks

Acconut · 2019-12-04T16:10:53Z

No. The uploading files doesn't work with fileReader: getTusFileReader parameter.

Ok, that's unfortunate to hear. I was at the believe that this feature should solve the problem

Has anyone already had such a problem?

No me personally. And I also don't have the time or working environment to reproduce this problem. Maybe @ifedapoolarewaju could have a look at this since he was previously involved?

jahn-brito · 2020-08-07T14:23:38Z

Hey, any news regarding this problem?

nikolaytsigvintsev · 2020-08-10T08:00:08Z

Hey, any news regarding this problem?

Hi, Jahn-Brito

Unfortunately, we have postponed this question for now

jahn-brito · 2020-08-11T14:04:46Z

@nikolaytsigvintsev I managed to upload it to Vimeo with large files.
Implementation:

_upload = new Upload(file, {
      uploadUrl: video.url,
      retryDelays: [0, 1000, 3000, 5000],
      chunkSize: 15 * 1024 * 1024,
      fileReader: new TusFileReader(),
      uploadSize: video.size,
      metadata: {
        filename: video.name || video.filename,
        filetype: video.mime
      },
      onError: (error) => onError(video.hashId, error),
      onProgress: (bytesAccepted, bytesTotal) => onProgress(video.hashId, bytesAccepted, bytesTotal),
      onSuccess: () => onSuccess(video.hashId)
    });

    _upload.start();

import RNFS from 'react-native-fs'
import base64 from 'base64-js'

export default class TusFileReader {
  openFile(input, chunkSize){
    return RNFS.stat(input.uri)
     .then((stats) => new FileSource(input, stats.size))
     .catch((err) => console.tron.log('erro', err))
  }
}

class FileSource {
  constructor(file, size){
    this._file = file
    this._size = size
  }

  async slice(start, end){
    end = Math.min(end, this._size)
    const length = end - start;
    const position = start;

    const data = await RNFS.read(this._file.uri, length, position, 'base64')
    const value = base64.toByteArray(data)

    return Promise.resolve({ value });
  }

  close(){}
}

sources:

nikolaytsigvintsev · 2020-08-12T07:23:38Z

Hi, Jahn-Brito !

Many thanks for the help! I will definitely try and report the result.

divyanshverma · 2020-08-16T00:43:26Z

Hey, any news on how did this work using the File Reader ?
I am currently stuck with as well. My setup is :
Ionic Capacitor, Angular
On android device the tus client does not do anything after the first HEAD request however when I run this in browser everthing works ok.

I am thinking it might be due to the way tus client reads the file on android device and perhaps having a Filereader will fix the issue.

Any help appreciated.

Thanks

Acconut · 2020-08-21T08:41:16Z

@divyanshverma This issue is about React Native. Does Ionic Capacitor use React Native under the hood? If not, please open a new issue with more details about the problems that you are seeing.

hellochirag · 2021-09-29T16:59:39Z

@nikolaytsigvintsev I managed to upload it to Vimeo with large files. Implementation:

_upload = new Upload(file, {
      uploadUrl: video.url,
      retryDelays: [0, 1000, 3000, 5000],
      chunkSize: 15 * 1024 * 1024,
      fileReader: new TusFileReader(),
      uploadSize: video.size,
      metadata: {
        filename: video.name || video.filename,
        filetype: video.mime
      },
      onError: (error) => onError(video.hashId, error),
      onProgress: (bytesAccepted, bytesTotal) => onProgress(video.hashId, bytesAccepted, bytesTotal),
      onSuccess: () => onSuccess(video.hashId)
    });

    _upload.start();

import RNFS from 'react-native-fs'
import base64 from 'base64-js'

export default class TusFileReader {
  openFile(input, chunkSize){
    return RNFS.stat(input.uri)
     .then((stats) => new FileSource(input, stats.size))
     .catch((err) => console.tron.log('erro', err))
  }
}

class FileSource {
  constructor(file, size){
    this._file = file
    this._size = size
  }

  async slice(start, end){
    end = Math.min(end, this._size)
    const length = end - start;
    const position = start;

    const data = await RNFS.read(this._file.uri, length, position, 'base64')
    const value = base64.toByteArray(data)

    return Promise.resolve({ value });
  }

  close(){}
}

sources:

it is not working for me i have implemented the same thing and still getting this error

*** Terminating app due to uncaught exception 'NSMallocException', reason: 'Failed to grow buffer'
*** First throw call stack

adrienfloor · 2021-09-30T10:02:47Z

@nikolaytsigvintsev I managed to upload it to Vimeo with large files. Implementation:

_upload = new Upload(file, {
      uploadUrl: video.url,
      retryDelays: [0, 1000, 3000, 5000],
      chunkSize: 15 * 1024 * 1024,
      fileReader: new TusFileReader(),
      uploadSize: video.size,
      metadata: {
        filename: video.name || video.filename,
        filetype: video.mime
      },
      onError: (error) => onError(video.hashId, error),
      onProgress: (bytesAccepted, bytesTotal) => onProgress(video.hashId, bytesAccepted, bytesTotal),
      onSuccess: () => onSuccess(video.hashId)
    });

    _upload.start();

import RNFS from 'react-native-fs'
import base64 from 'base64-js'

export default class TusFileReader {
  openFile(input, chunkSize){
    return RNFS.stat(input.uri)
     .then((stats) => new FileSource(input, stats.size))
     .catch((err) => console.tron.log('erro', err))
  }
}

class FileSource {
  constructor(file, size){
    this._file = file
    this._size = size
  }

  async slice(start, end){
    end = Math.min(end, this._size)
    const length = end - start;
    const position = start;

    const data = await RNFS.read(this._file.uri, length, position, 'base64')
    const value = base64.toByteArray(data)

    return Promise.resolve({ value });
  }

  close(){}
}

sources:

it is not working for me i have implemented the same thing and still getting this error

*** Terminating app due to uncaught exception 'NSMallocException', reason: 'Failed to grow buffer' *** First throw call stack

Hey guys, same for me here !
@nikolaytsigvintsev did you manage to figure something out ?
Thanks a lot

nikolaytsigvintsev · 2021-09-30T11:32:12Z

@nikolaytsigvintsev I managed to upload it to Vimeo with large files. Implementation:

_upload = new Upload(file, {
      uploadUrl: video.url,
      retryDelays: [0, 1000, 3000, 5000],
      chunkSize: 15 * 1024 * 1024,
      fileReader: new TusFileReader(),
      uploadSize: video.size,
      metadata: {
        filename: video.name || video.filename,
        filetype: video.mime
      },
      onError: (error) => onError(video.hashId, error),
      onProgress: (bytesAccepted, bytesTotal) => onProgress(video.hashId, bytesAccepted, bytesTotal),
      onSuccess: () => onSuccess(video.hashId)
    });

    _upload.start();

import RNFS from 'react-native-fs'
import base64 from 'base64-js'

export default class TusFileReader {
  openFile(input, chunkSize){
    return RNFS.stat(input.uri)
     .then((stats) => new FileSource(input, stats.size))
     .catch((err) => console.tron.log('erro', err))
  }
}

class FileSource {
  constructor(file, size){
    this._file = file
    this._size = size
  }

  async slice(start, end){
    end = Math.min(end, this._size)
    const length = end - start;
    const position = start;

    const data = await RNFS.read(this._file.uri, length, position, 'base64')
    const value = base64.toByteArray(data)

    return Promise.resolve({ value });
  }

  close(){}
}

sources:

it is not working for me i have implemented the same thing and still getting this error
*** Terminating app due to uncaught exception 'NSMallocException', reason: 'Failed to grow buffer' *** First throw call stack

Hey guys, same for me here ! @nikolaytsigvintsev did you manage to figure something out ? Thanks a lot

Hi @adrienfloor

Unfortunately, we have postponed this question for now.
I am currently using the react-native-tus-client module.

But @jahn-brito said he can transfer large files?

hellochirag · 2021-10-01T04:31:02Z

react-native-tus-client

@nikolaytsigvintsev react-native-tus-client is also throwing the same error, it not allowed the 500 MB + video from devices

nikolaytsigvintsev · 2021-11-05T14:25:34Z

@nikolaytsigvintsev I managed to upload it to Vimeo with large files. Implementation:

_upload = new Upload(file, {
      uploadUrl: video.url,
      retryDelays: [0, 1000, 3000, 5000],
      chunkSize: 15 * 1024 * 1024,
      fileReader: new TusFileReader(),
      uploadSize: video.size,
      metadata: {
        filename: video.name || video.filename,
        filetype: video.mime
      },
      onError: (error) => onError(video.hashId, error),
      onProgress: (bytesAccepted, bytesTotal) => onProgress(video.hashId, bytesAccepted, bytesTotal),
      onSuccess: () => onSuccess(video.hashId)
    });

    _upload.start();

import RNFS from 'react-native-fs'
import base64 from 'base64-js'

export default class TusFileReader {
  openFile(input, chunkSize){
    return RNFS.stat(input.uri)
     .then((stats) => new FileSource(input, stats.size))
     .catch((err) => console.tron.log('erro', err))
  }
}

class FileSource {
  constructor(file, size){
    this._file = file
    this._size = size
  }

  async slice(start, end){
    end = Math.min(end, this._size)
    const length = end - start;
    const position = start;

    const data = await RNFS.read(this._file.uri, length, position, 'base64')
    const value = base64.toByteArray(data)

    return Promise.resolve({ value });
  }

  close(){}
}

sources:

Hi @jahn-brito!

I had to go back to using tus-js-client to upload large files. I upgraded to 2.3.0 and updated my fileReader function from your example.

Yes ! Yes ! Yes !

I certify that the upload went smoothly and without memory leaks on my Android device.
android: 9.0
react-native: 0.64.2
tus-js-client: 2.3.0
react-native-fs: 2.16.6
file: 2.8 GB.
chunkSize: 8 * 1024 * 1024,

true, when the chunk is more than 10MB, the application crashes. I could not determine the reason. I put 8MB.

==============================================================
I certify that the upload went smoothly and without memory leaks on my Ipad device.
ios: 13.3
react-native: 0.64.2
tus-js-client: 2.3.0
react-native-fs: 2.16.6
file: 1.7 GB.
chunkSize: 8 * 1024 * 1024,

report:
https://www.dropbox.com/s/fhnoa2ya2cyk0vy/Screenshot%202021-11-05%20at%2012.43.50.png?dl=0

@jahn-brito Thanks! Great job!
@kvz,@ifedapoolarewaju Thanks!

Thanks to everyone who participated in the development of this library!

Acconut · 2021-11-08T13:13:16Z

I certify that the upload went smoothly and without memory leaks on my Android device.

Wow, that is amazing news! Congrats 🎉

swizes · 2021-11-21T12:43:19Z

@nikolaytsigvintsev thanks for letting me know about this issue. Did you test if this upload method works while app is in background?

nikolaytsigvintsev · 2021-11-22T08:10:31Z

@nikolaytsigvintsev thanks for letting me know about this issue. Did you test if this upload method works while app is in the background?

Hi @swizes !

Yes. This method works in the background for Android and works with iOS restrictions.

ccharliemagne · 2021-11-24T05:35:09Z

@nikolaytsigvintsev I managed to upload it to Vimeo with large files. Implementation:
_upload = new Upload(file, {
      uploadUrl: video.url,
      retryDelays: [0, 1000, 3000, 5000],
      chunkSize: 15 * 1024 * 1024,
      fileReader: new TusFileReader(),
      uploadSize: video.size,
      metadata: {
        filename: video.name || video.filename,
        filetype: video.mime
      },
      onError: (error) => onError(video.hashId, error),
      onProgress: (bytesAccepted, bytesTotal) => onProgress(video.hashId, bytesAccepted, bytesTotal),
      onSuccess: () => onSuccess(video.hashId)
    });

    _upload.start();
import RNFS from 'react-native-fs'
import base64 from 'base64-js'

export default class TusFileReader {
  openFile(input, chunkSize){
    return RNFS.stat(input.uri)
     .then((stats) => new FileSource(input, stats.size))
     .catch((err) => console.tron.log('erro', err))
  }
}

class FileSource {
  constructor(file, size){
    this._file = file
    this._size = size
  }

  async slice(start, end){
    end = Math.min(end, this._size)
    const length = end - start;
    const position = start;

    const data = await RNFS.read(this._file.uri, length, position, 'base64')
    const value = base64.toByteArray(data)

    return Promise.resolve({ value });
  }

  close(){}
}
sources:

Example Uppy with React-Native

FileReader interface in the tus-js-client documentation
Hi @jahn-brito!

I had to go back to using tus-js-client to upload large files. I upgraded to 2.3.0 and updated my fileReader function from your example.

Yes ! Yes ! Yes !

I certify that the upload went smoothly and without memory leaks on my Android device. android: 9.0 react-native: 0.64.2 tus-js-client: 2.3.0 react-native-fs: 2.16.6 file: 2.8 GB. chunkSize: 8 * 1024 * 1024,

true, when the chunk is more than 10MB, the application crashes. I could not determine the reason. I put 8MB.

============================================================== I certify that the upload went smoothly and without memory leaks on my Ipad device. ios: 13.3 react-native: 0.64.2 tus-js-client: 2.3.0 react-native-fs: 2.16.6 file: 1.7 GB. chunkSize: 8 * 1024 * 1024,

report: https://www.dropbox.com/s/fhnoa2ya2cyk0vy/Screenshot%202021-11-05%20at%2012.43.50.png?dl=0

@jahn-brito Thanks! Great job! @kvz,@ifedapoolarewaju Thanks!

Thanks to everyone who participated in the development of this library!

Hi @nikolaytsigvintsev

Its working but I'm experiencing FPS drops on JS thread up to -2fps. On Android.

(Update) Every time the slice() function in FileSource get triggered it brings down the JS to -2fps#146

chunkSize: 8 * 1024 * 1024

nikolaytsigvintsev · 2021-11-24T09:33:46Z

FPS drops on JS thread

Hi @ccharliemagne!

Yes. I see it too. Apparently, when a packet is formed for transmission, then all the processing power is used. I am struggling with this by reducing the chunk size to 1MB. 1 * 1024 * 1024.

samal-rasmussen · 2022-03-10T12:48:53Z

What I've seen is that it's the base64.toByteArray(data) that takes a long time to process per chunk/slice. RNFS.read and the rest of the slice function implementation is quick.

Acconut · 2022-03-14T13:14:16Z

What I've seen is that it's the base64.toByteArray(data) that takes a long time to process per chunk/slice.

Do you know if there is an replacement for the toByteArray function to speed this up?

samal-rasmussen · 2022-03-22T09:45:21Z

I tried a couple of alternatives, but they were all slow. In the end we just figured out that we could do without the custom FileReader, so I didn't get so far as to profile the running code and see precisely what was slow. But an educated guess would be that it is because converting a base64 encoded string to a byte array in an inherently computationally intensive operation and the js runtime in React Native does not have a jit compiler[0] and only an interpreter. An interpreter will run code muuuuch slower than jit compiled code, and computationally intensive code like this will be hit especially hard by this slowdown.

[0] https://reactnative.dev/docs/javascript-environment

Acconut · 2022-04-09T09:52:28Z

@samal-rasmussen Ah, thank you for the insights. So if I understand correctly, you are just using the FileReader that is built into tus-js-client? I.e. this one:

tus-js-client/lib/browser/fileReader.js

Lines 9 to 19 in c0b5a7d

    
           // In React Native, when user selects a file, instead of a File or Blob, 
        
           // you usually get a file object {} with a uri property that contains 
        
           // a local path to the file. We use XMLHttpRequest to fetch 
        
           // the file blob, before uploading with tus. 
        
           if (isReactNative() && input && typeof input.uri !== 'undefined') { 
        
             return uriToBlob(input.uri) 
        
               .then((blob) => new FileSource(blob)) 
        
               .catch((err) => { 
        
                 throw new Error(`tus: cannot fetch \`file.uri\` as Blob, make sure the uri is correct and accessible. ${err}`) 
        
               }) 
        
           }

malnajdi · 2023-02-14T23:55:46Z

Hey guys, i am trying to upload large file and faced the same issue with the ios and android memory fills up and crash the app. Using React native as well. i switched from axios to your client and i tried to implement the above solution without luck.

I am using Expo Picker -> will yield a uri for me and pass it to new tus.Upload(fileUrl, {}) with options as stated above but i always get undefined is not an object

mifi · 2023-02-15T09:36:41Z

Could you post some more info about the error? Any stack or line which it crashes on?

malnajdi · 2023-02-15T09:43:15Z

[Wed Feb 15 2023 02:21:43.400] LOG [TypeError: undefined is not an object (evaluating 'path.startsWith')]

this is the error i get. I am not sure thought if i need to change my backend to be tus compatible or i can just use it with normal multipart/formdata

mifi · 2023-02-15T13:46:42Z

if you want to use tus-js-client you need to have a tus compatible backend. the official recommended one is https://github.com/tus/tusd

did you try this code? #146 (comment)

malnajdi · 2023-02-15T14:44:52Z

if you want to use tus-js-client you need to have a tus compatible backend. the official recommended one is https://github.com/tus/tusd

did you try this code? #146 (comment)

i will try it with a compatible backend

malnajdi · 2023-02-19T01:28:10Z

getting this error.
tusd: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by tusd)

Acconut · 2023-02-27T13:37:06Z

tusd: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by tusd)

See tus/tusd#909

nikolaytsigvintsev added the bug label Mar 28, 2019

Acconut mentioned this issue Sep 26, 2019

Ideas for major v2.0 release #167

Closed

11 tasks

shinsaku-ta mentioned this issue Nov 11, 2020

When uploading a large video from the iPhone, the resource is not loaded in the middle. #227

Closed

nikolaytsigvintsev closed this as completed Nov 5, 2021

nikolaytsigvintsev mentioned this issue Nov 20, 2021

Error: tus: cannot fetch file.uri as Blob, make sure the uri is correct and accessible. #284

Open

Crashed to upload large files ( > 500MB): [Terminating app due to uncaught exception 'NSMallocException', reason: 'Failed to grow buffer'] #146

Crashed to upload large files ( > 500MB): [Terminating app due to uncaught exception 'NSMallocException', reason: 'Failed to grow buffer'] #146

Comments

nikolaytsigvintsev commented Mar 28, 2019 • edited by Acconut Loading

Acconut commented Mar 30, 2019

nikolaytsigvintsev commented Apr 2, 2019 • edited by Acconut Loading

Acconut commented Apr 3, 2019

nikolaytsigvintsev commented Apr 3, 2019

Acconut commented Apr 5, 2019

nikolaytsigvintsev commented Apr 25, 2019

kvz commented Apr 25, 2019

nikolaytsigvintsev commented Apr 26, 2019 • edited Loading

kvz commented May 6, 2019 • edited Loading

nikolaytsigvintsev commented May 8, 2019 • edited Loading

Videoreport: https://www.dropbox.com/s/1z8yy5v85mmy72m/ScreenRecording_05-08-2019%2013-31-57.MP4?dl=0

ifedapoolarewaju commented May 8, 2019

nikolaytsigvintsev commented May 8, 2019

ifedapoolarewaju commented May 8, 2019

nikolaytsigvintsev commented May 8, 2019

ifedapoolarewaju commented May 8, 2019

nikolaytsigvintsev commented May 8, 2019

ifedapoolarewaju commented May 9, 2019 • edited Loading

nikolaytsigvintsev commented May 10, 2019 • edited Loading

nikolaytsigvintsev commented May 10, 2019 • edited Loading

ifedapoolarewaju commented Jun 7, 2019

nikolaytsigvintsev commented Jun 7, 2019

nikolaytsigvintsev commented Oct 18, 2019

Acconut commented Oct 22, 2019

nikolaytsigvintsev commented Nov 25, 2019 • edited Loading

Acconut commented Dec 2, 2019

nikolaytsigvintsev commented Dec 4, 2019

Acconut commented Dec 4, 2019

jahn-brito commented Aug 7, 2020

nikolaytsigvintsev commented Aug 10, 2020

jahn-brito commented Aug 11, 2020

nikolaytsigvintsev commented Aug 12, 2020

divyanshverma commented Aug 16, 2020

Acconut commented Aug 21, 2020

hellochirag commented Sep 29, 2021

adrienfloor commented Sep 30, 2021

nikolaytsigvintsev commented Sep 30, 2021

hellochirag commented Oct 1, 2021

nikolaytsigvintsev commented Nov 5, 2021 • edited Loading

Acconut commented Nov 8, 2021

swizes commented Nov 21, 2021

nikolaytsigvintsev commented Nov 22, 2021

ccharliemagne commented Nov 24, 2021 • edited Loading

nikolaytsigvintsev commented Nov 24, 2021 • edited Loading

samal-rasmussen commented Mar 10, 2022

Acconut commented Mar 14, 2022

samal-rasmussen commented Mar 22, 2022

Acconut commented Apr 9, 2022

malnajdi commented Feb 14, 2023

mifi commented Feb 15, 2023

malnajdi commented Feb 15, 2023

mifi commented Feb 15, 2023

malnajdi commented Feb 15, 2023

malnajdi commented Feb 19, 2023

Acconut commented Feb 27, 2023

nikolaytsigvintsev commented Mar 28, 2019 •

edited by Acconut

Loading

nikolaytsigvintsev commented Apr 2, 2019 •

edited by Acconut

Loading

nikolaytsigvintsev commented Apr 26, 2019 •

edited

Loading

kvz commented May 6, 2019 •

edited

Loading

nikolaytsigvintsev commented May 8, 2019 •

edited

Loading

Videoreport:
https://www.dropbox.com/s/1z8yy5v85mmy72m/ScreenRecording_05-08-2019%2013-31-57.MP4?dl=0

ifedapoolarewaju commented May 9, 2019 •

edited

Loading

nikolaytsigvintsev commented May 10, 2019 •

edited

Loading

nikolaytsigvintsev commented May 10, 2019 •

edited

Loading

nikolaytsigvintsev commented Nov 25, 2019 •

edited

Loading

nikolaytsigvintsev commented Nov 5, 2021 •

edited

Loading

ccharliemagne commented Nov 24, 2021 •

edited

Loading

nikolaytsigvintsev commented Nov 24, 2021 •

edited

Loading