Version 5.0 breaks collectstatic when using some storage backends

[machakux]

When using Google Cloud Storage backend via django-storages running ./manage.py collectstatic fails with some ValueError. Based on my analysis it seems the error has something to do with DebugStaticFilesStorage objects from debug toolbar 5. Downgrading to version 4.4.* seems to solve the issue.

The issue can be encounteres when using
django-debug-toolbar==5.0.1
Django==5.1.5
django-storages[google]==1.14.4

A sample error returned by ./manage.py collectstatic ...

  File "./manage.py", line 22, in <module>
    main()
  File "./manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File ".virtualenvs/test/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File ".virtualenvs/test/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File ".virtualenvs/test/lib/python3.10/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File ".virtualenvs/test/lib/python3.10/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
  File ".virtualenvs/test/lib/python3.10/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 209, in handle
    collected = self.collect()
  File ".virtualenvs/test/lib/python3.10/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 135, in collect
    handler(path, prefixed_path, storage)
  File ".virtualenvs/test/lib/python3.10/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 378, in copy_file
    self.storage.save(prefixed_path, source_file)
  File ".virtualenvs/test/lib/python3.10/site-packages/django/core/files/storage/base.py", line 49, in save
    name = self._save(name, content)
  File ".virtualenvs/test/lib/python3.10/site-packages/storages/backends/gcloud.py", line 192, in _save
    file_object = GoogleCloudFile(name, "rw", self)
  File ".virtualenvs/test/lib/python3.10/site-packages/storages/backends/gcloud.py", line 45, in __init__
    self.blob = storage.bucket.get_blob(name, chunk_size=storage.blob_chunk_size)
  File ".virtualenvs/test/lib/python3.10/site-packages/google/cloud/storage/bucket.py", line 1268, in get_blob
    blob.reload(
  File ".virtualenvs/test/lib/python3.10/site-packages/google/cloud/storage/_helpers.py", line 247, in reload
    self.path,
  File ".virtualenvs/test/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 334, in path
    return self.path_helper(self.bucket.path, self.name)
  File ".virtualenvs/test/lib/python3.10/site-packages/google/cloud/storage/bucket.py", line 1173, in path
    raise ValueError("Cannot determine path without bucket name.")
ValueError: Cannot determine path without bucket name.

[matthiask]

The debug toolbar doesn't appear in the traceback at all, only Django and django-storages and the google cloud library.

Maybe changing the toolbar version installed or uninstalled something else, maybe a dependency?

[tim-schilling]

I agree with @matthiask. DebugConfiguredStorage overrides the url method. That doesn't seem to be used here, though I could be wrong.

Based on my analysis it seems the error has something to do with DebugStaticFilesStorage objects from debug toolbar 5

Could you elaborate on this? Maybe we're missing something.

[machakux]

Could you elaborate on this? Maybe we're missing something.

Based on my analysis it seemed like when collectstatic is executed the DebugStaticFilesStorage object is passed to some internal parts of the process (example in GoogleCloudFile initialization) instead of the original storage object. The passed DebugStaticFilesStorage() will contain a bucket attribute which is Bucket instance but without name hence causing google cloud storage utilities to fail in performing some operations.

Whenever I was facing this issue and tried disabling the debug toolbar (without changing any installed python packages), the issue disappeared. And after trying to track down the source of the issue I noticed in some places the passed storage object is a DebugStaticFilesStorage instance and the storage.bucket = <Bucket: None>.

I am not an expert regarding the internals of debug toolbar so I might be missing something. But In general I thought probably the DebugStaticFilesStorage is supposed to somehow wrap or copy the original storage object but something wasn't taken into account.

Thanks

[tim-schilling]

I think we're going to need your help identifying the problem here. If you can create a reproducible example for us that would be ideal.

If that's not possible, it's unlikely we'll get to this. <Bucket: None> seems to indicate the bucket isn't defined which to me, sounds like it's a project configuration issue.

[machakux]

The issue may be reproduced by the following steps

1. start a new django project.
2. install django-storages[google], and django-debug-toolbar
3. configure google cloud storage (including the bucket name) as a storage backend for static files via django-storages
4. configure the debug toolbar
5. run ./manage.py collectstatic

I have managed to reproduce the issue using those steps. I have attached a minimal django project example that is could be used to reproduce the issue.

If the issue can't be reproduced on your end and you strongly think it is a configuration issue feel to close the ticket.

testdjango.zip

[matthiask]

I have an idea what might be going on. We're reinstantiating the static files storage class here, probably losing some attributes in the process:

django-debug-toolbar/debug_toolbar/panels/staticfiles.py

Line 76 in fdd636c

self._wrapped = DebugStaticFilesStorage()

We could (and maybe should?) instead add a mixin to the MRO which wraps the url() method, something like this:

class URLMixin:
    def url(...):
        ...

configured_storage = storages["staticfiles"]
configured_storage.__class__.__bases__.insert(0, URLMixin)

With the appropriate fallbacks and everything. Maybe there are better ways to implement this, but all we want is to see the static files for which URLs are generated.

Alternatively, we would wrap the static files instance with a proxy which forwards everything to the wrapped instance, similar to how django.utils.functional.LazyObject does it, but that seems like much more work.

[matthiask]

I'm doubting my own comment from above because we have been using the way of wrapping the static files storage for a really long time, that hasn't changed recently. So upgrading to 5.0 shouldn't have changed anything.

[matthiask]

@machakux Here's a draft PR. It would be great if you could test this:
#2069

This isn't ready to be merged though, no tests etc.

[machakux]

@matthiask Based on my basic tests, It seems like #2069 is working as expected (i.e. collectstatic doesn't produce the error when using that version).

[matthiask]

@machakux Thanks for testing the change!

[dr-rompecabezas]

I tested @matthiask's changes per #2069 using @machakux's testdjango.zip, including creating a project/bucket in GCP, and confirm the error message is gone.