Skip to content

perf: enable gzip. #2422

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 27, 2025
Merged

perf: enable gzip. #2422

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 84 additions & 0 deletions apps/common/middleware/gzip.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# coding=utf-8
"""
@project: MaxKB
@Author：虎
@file： gzip.py
@date：2025/2/27 10:03
@desc:
"""
from django.utils.cache import patch_vary_headers
from django.utils.deprecation import MiddlewareMixin
from django.utils.regex_helper import _lazy_re_compile
from django.utils.text import compress_sequence, compress_string

re_accepts_gzip = _lazy_re_compile(r"\bgzip\b")


class GZipMiddleware(MiddlewareMixin):
"""
Compress content if the browser allows gzip compression.
Set the Vary header accordingly, so that caches will base their storage
on the Accept-Encoding header.
"""

max_random_bytes = 100

def process_response(self, request, response):
if request.method != 'GET' or request.path.startswith('/api'):
return response
# It's not worth attempting to compress really short responses.
if not response.streaming and len(response.content) < 200:
return response

# Avoid gzipping if we've already got a content-encoding.
if response.has_header("Content-Encoding"):
return response

patch_vary_headers(response, ("Accept-Encoding",))

ae = request.META.get("HTTP_ACCEPT_ENCODING", "")
if not re_accepts_gzip.search(ae):
return response

if response.streaming:
if response.is_async:
# pull to lexical scope to capture fixed reference in case
# streaming_content is set again later.
orignal_iterator = response.streaming_content

Check warning on line 47 in apps/common/middleware/gzip.py

View workflow job for this annotation

GitHub Actions / Spell Check with Typos 

"orignal" should be "original".

async def gzip_wrapper():
async for chunk in orignal_iterator:

Check warning on line 50 in apps/common/middleware/gzip.py

View workflow job for this annotation

GitHub Actions / Spell Check with Typos 

"orignal" should be "original".
yield compress_string(
chunk,
max_random_bytes=self.max_random_bytes,
)

response.streaming_content = gzip_wrapper()
else:
response.streaming_content = compress_sequence(
response.streaming_content,
max_random_bytes=self.max_random_bytes,
)
# Delete the `Content-Length` header for streaming content, because
# we won't know the compressed size until we stream it.
del response.headers["Content-Length"]
else:
# Return the compressed content only if it's actually shorter.
compressed_content = compress_string(
response.content,
max_random_bytes=self.max_random_bytes,
)
if len(compressed_content) >= len(response.content):
return response
response.content = compressed_content
response.headers["Content-Length"] = str(len(response.content))

# If there is a strong ETag, make it weak to fulfill the requirements
# of RFC 9110 Section 8.8.1 while also allowing conditional request
# matches on ETags.
etag = response.get("ETag")
if etag and etag.startswith('"'):
response.headers["ETag"] = "W/" + etag
response.headers["Content-Encoding"] = "gzip"

return response
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After reviewing the provided code snippet, I have identified several areas that could be improved:

  1. Import Order: The import statements are alphabetical, which can sometimes make them harder to read.

  2. Variable Naming: Variable names like re_accepts_gzip, max_random_bytes, etc., follow Django's naming conventions well but should be consistent across similar contexts.

  3. Code Duplication: There is duplicated code between the non-streaming and streaming paths in the process_response method. This can be cleaned up by using a single function for both cases.

  4. Error Handling: While not explicitly mentioned, handling exceptions during compression might improve robustness.

  5. Comments: Some parts of the code could benefit from more detailed comments explaining what each section does.

Here is an optimized version of the code based on these points:

# coding=utf-8
"""
    @project: MaxKB
    @Author：虎
    @file： gzip.py
    @date：2025/2/27 10:03
    @desc:
"""

import zlib

from django.utils.cache import patch_vary_headers
from django.utils.deprecation import MiddlewareMixin
from django.utils.regex_helper import _lazy_re_compile
from django.utils.text import compress_sequence, compress_string


ACCEPTS_GZIP_PATTERN = r"\bgzip\b"

class GZipMiddleware(MiddlewareMixin):
    """
    Compress content if the browser allows gzip compression.
    Set the Vary header accordingly, so that caches will base their storage
    on the Accept-Encoding header.
    """

    MAX_RANDOM_BYTES = 100

    def process_request(self, request):
        self.accepts_gzip = ACCEPTS_GZIP_PATTERN.match(request.META.get('HTTP_ACCEPT_ENCODING', ''))

    def process_response(self, request, response):
        if (
            request.method != "GET" or
            request.path.startswith("/api") or
            not response.streaming and len(response.content) < 200
        ):
            return response

        # Avoid gzipping if we've already got a content-encoding.
        if response.has_header("Content-Encoding"):
            return response

        patch_vary_headers(response, ("Accept-Encoding",))

        if not self.accepts_gzip:
            return response

        self._apply_gzip_compression(response)

        return response

    def _apply_gzip_compression(self, response):
        if response.streaming:
            if response.is_async:
                async_iterable = response.streaming_content
                original_iterator = iter(async_iterable)
                
                async def gzip_wrapper():
                    async for chunk in original_iterator:
                        yield zlib.compress(chunk, level=9)  # Use zlib for better quality
                
                response.streaming_content = gzip_wrapper()
    
            else:
                response.streaming_content = (zlib.compress(chunk, level=9) for chunk in response.streaming_content)
    
                # Adjust Content-Length for streaming with zlib; cannot determine beforehand
                del response.headers["Content-Length"]
        
        else:
            compressed_content = zlib.compress(response.content, level=9)
            
            if len(compressed_content) >= len(response.content):
                return
            
            response.content = compressed_content
            response.headers['Content-Length'] = str(len(response.content))
    
            # Update ETag for conditional requests
            original_etag = response.get('ETag')
            if original_etag and original_etag.startswith('"'):
                adjusted_etag = f"W/{original_etag}"
                response.set_cookie(
                    'etag',
                    adjusted_etag,
                    max_age=None,
                    expires=None,
                    domain=None,
                    path='/', secure=False, http_only=True
                )
                response.delete_cookie('etag')  # Remove old cookie with original ETag
                response.setdefault('ETag', adjusted_etag).replace("\"")

        response.headers['Content-Encoding'] = 'gzip'

Key Changes Made:

  • Imports and Comments: Improved structure and added more descriptive comments.
  • Request Processing Logic: Moved logic into helper functions _apply_gzip_compression.
  • Compression Library: Used zlib instead of custom byte manipulation for better functionality and performance.
  • Conditional Requests: Updated ETag handling to reflect changes made when compressing the content.
  • Streamed Responses: Adjusted processing to handle streamed responses efficiently using iterators with asyncio where necessary.

This refactoring enhances readability and maintainability while ensuring efficient gzip compression for user agents supporting it.

2 changes: 1 addition & 1 deletion apps/smartdoc/settings/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.gzip.GZipMiddleware',
'common.middleware.gzip.GZipMiddleware',
'common.middleware.static_headers_middleware.StaticHeadersMiddleware',
'common.middleware.cross_domain_middleware.CrossDomainMiddleware'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provided code looks generally correct for setting up middleware in Django. However, there are a few potential areas to consider:

  1. Middleware Order: The order of middleware is important. Ensure that CommonMiddleware and related middleware classes come before any custom middleware like gzip, cross-domain headers, etc.

  2. Import Correctly: Make sure you import 'compressionmiddleware' instead of 'django.middleware.gzip.GZipMiddleware'. Assuming that's what you meant, the line should be updated accordingly:

    'compressionmiddleware.middleware.CompressionMiddleware',

  3. Static Headers Middleware: Ensure that static files middleware (django.contrib.staticfiles.middleware.StaticFilesMiddleware) comes after any compression or other middleware that might interfere with it.

  4. Cross-Domain Middleware ensures proper handling of cross-origin resource sharing (CORS) rules. It should typically appear near others involved in response customization.

Here’s an improved version with these considerations addressed:

    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.contrib.staticfiles.middleware.StaticFilesMiddleware', # Static files middleware must come first
    
    'compressionmiddleware.middleware.CompressionMiddleware', # Replace with appropriate gzip library
    
    'common.middleware.gzip.GZipMiddleware',
    'common.middleware.static_headers_middleware.StaticHeadersMiddleware',
    'common.middleware.cross_domain_middleware.CrossDomainMiddleware'

If you're using a specific third-party module for GZIP compression, make sure to replace 'gzipmiddleware.middleware.CompressionMiddleware with the corresponding class from that library, if applicable.

Expand Down
Loading