Problem with S3BotoStorageFile and a file greater than 5Mb

[iceseyes]

My team and I have found a problem in file-like interface for S3botoStorageFile.

If I do something like that:

 storage = S3BotoStorage()
 with storage.open("test.csv", "w") as f:
     df.to_csv(f)

where df is a Pandas' Dataframe, and file is greater than 5Mb, when we run the program from an ec2 instance, program stops over a write instruction.

Debugging we have seen that when file size become 5Mb the test in s3boto.py:159

if self.buffer_size <= self._buffer_file_size:
     self._flush_write_buffer()

is True for every write request, so for every write I have a request to upload to s3.... and after a few of seconds program freezes.

I think that test should be something like that

if (self._write_counter * self.buffer_size) <= self._buffer_file_size:
     self._flush_write_buffer()

Thank you.

[timur-orudzhov]

We've also encountered with that issue and pr, that provided by tommwatson, fixed that issue. In our case, we open local file that weight about 13 mb and write to file by chunks (~ 1mb), that on s3, as a result we have file with weight about 76 mb. How to reproduce:

from django.core.files.storage import default_storage

def read_in_chunks(file_object, chunk_size=1024 * 1000):
    """Lazy function (generator) to read a file piece by piece.
       Default chunk size: 1mb.
    """
    while True:
        data = file_object.read(chunk_size)
        if not data:
            break
        yield data


with open('/path/to/file', 'r+b') as local_file:
    with default_storage.open('path/to/s3_file', mode='wb') as s3_file:
        for chunk in read_in_chunks(local_file):
            s3_file.write(chunk)

Are there chances to merge fix i've mentioned earlier ?