Integration with django-storages.

[shuryhin-oleksandr]

I have the following code in my project:

settings.py:
CHUNKED_UPLOAD_STORAGE_CLASS = 'sstraffic.storage_backends.PublicMediaStorage'

storage_backends.py:

class PublicMediaStorage(S3Boto3Storage):
    location = 'media'
    default_acl = 'public-read'
    file_overwrite = False

I have an error:

[DJANGO] ERROR 2020-10-28 14:42:52,843 log django.request.log_response:228: Internal Server Error: /instructors/create-program/6cebcf3a-ae80-483c-8c40-b4dbf5179858/upload-media-chunks/
Traceback (most recent call last):
  File "/home/alex/.local/share/virtualenvs/sstraffic-kPpfodf3/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/home/alex/.local/share/virtualenvs/sstraffic-kPpfodf3/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/alex/.local/share/virtualenvs/sstraffic-kPpfodf3/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/alex/.local/share/virtualenvs/sstraffic-kPpfodf3/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/alex/.local/share/virtualenvs/sstraffic-kPpfodf3/lib/python3.7/site-packages/django/utils/decorators.py", line 45, in _wrapper
    return bound_method(*args, **kwargs)
  File "/home/alex/.local/share/virtualenvs/sstraffic-kPpfodf3/lib/python3.7/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/home/alex/.local/share/virtualenvs/sstraffic-kPpfodf3/lib/python3.7/site-packages/django/utils/decorators.py", line 45, in _wrapper
    return bound_method(*args, **kwargs)
  File "/home/alex/code/sstraffic/app/instructors/decorators.py", line 18, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/home/alex/.local/share/virtualenvs/sstraffic-kPpfodf3/lib/python3.7/site-packages/django/views/generic/base.py", line 97, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/alex/.local/share/virtualenvs/sstraffic-kPpfodf3/lib/python3.7/site-packages/chunked_upload/views.py", line 100, in post
    return self._post(request, *args, **kwargs)
  File "/home/alex/.local/share/virtualenvs/sstraffic-kPpfodf3/lib/python3.7/site-packages/chunked_upload/views.py", line 221, in _post
    chunked_upload.append_chunk(chunk, chunk_size=chunk_size, save=False)
  File "/home/alex/.local/share/virtualenvs/sstraffic-kPpfodf3/lib/python3.7/site-packages/chunked_upload/models.py", line 65, in append_chunk
    with open(self.file.path, mode='ab') as file_obj:  # mode = append+binary
  File "/home/alex/.local/share/virtualenvs/sstraffic-kPpfodf3/lib/python3.7/site-packages/django/db/models/fields/files.py", line 57, in path
    return self.storage.path(self.name)
  File "/home/alex/.local/share/virtualenvs/sstraffic-kPpfodf3/lib/python3.7/site-packages/django/core/files/storage.py", line 109, in path
    raise NotImplementedError("This backend doesn't support absolute paths.")
NotImplementedError: This backend doesn't support absolute paths.
[DJANGO] ERROR 2020-10-28 14:42:52,846 basehttp django.server.log_message:154: "POST /instructors/create-program/6cebcf3a-ae80-483c-8c40-b4dbf5179858/upload-media-chunks/ HTTP/1.1" 500 26323

I fixed it by adding a function path to my PublicMediaStorage class.

    def path(self, name):
        return name

Is not that too hucky? Will not I break something by this?
It would be nice to have that working out of the box.

[awwester]

Did you get a full working solution for this? I've been overcoming multiple errors trying to get this to work with s3 and django-rest-framework (finally landing on this one), but am still running into some difficulties. Did you end up having to make additional changes other than just adding this path method?

[shuryhin-oleksandr]

No, I just upload my file directly from FE to AWS.

[tngeene]

Did you get a full working solution for this? I've been overcoming multiple errors trying to get this to work with s3 and django-rest-framework (finally landing on this one), but am still running into some difficulties. Did you end up having to make additional changes other than just adding this path method?

I had to override the entire upload function in the chunkedupload model by writing the custom model. In my case, I was uploading to google cloud storage but it should work for s3 as well.
However, we can all agree that this package should be considered somewhat legacy as it hasn't been maintained in years.

class ChunkedUploadFile(ChunkedUpload):
    file = models.FileField(verbose_name=_('file'),
                            max_length=255,
                            upload_to=generate_chunked_filename,
                            null=True)

    def allowed_owners(self):
        return super(ChunkedUploadFile, self).allowed_owners()

    def allowed_owner(self, owner_type, owner_id=None, msg=None):
        return super(ChunkedUploadFile,
                     self).allowed_owner(owner_type, owner_id, msg)

    def append_chunk(self, chunk, chunk_size=None, save=True):
        gcs_file = self.file
        gcs_file.close()
        gcs_file.open(mode='w') #change this if uploading to ab Digital ocean or baremetal server
        for subchunk in chunk.chunks():
            gcs_file.write(subchunk)
        if chunk_size is not None:
            self.offset += chunk_size
        elif hasattr(chunk, 'size'):
            self.offset += chunk.size
        else:
            self.offset = self.file.size
        # clear any cached checksum
        self._checksum = None
        if save:
            self.save()
        self.file.close()

    def get_uploaded_file(self):
        gcs_file = self.file
        gcs_file.close()
        gcs_file.open(mode='rb')
        return UploadedFile(file=self.file,
                            name=self.filename,
                            size=self.file.size)

    @transaction.atomic
    def completed(self,
                  completed_at=timezone.now(),
                  ext=_settings.COMPLETE_EXT):
        if ext != _settings.INCOMPLETE_EXT:
            original_path = self.file.name
            path_ext = original_path.split('.')[-1]
            # remove filename extension
            formatted_file_path = original_path.split('.')[0]
            self.file.name = formatted_file_path + ext
        self.status = self.COMPLETE
        self.completed_at = completed_at
        self.save()
        if ext != _settings.INCOMPLETE_EXT:
            rename_blob(original_path, self.file.name)
            self.save()