compression shouldn't compress binary and images by default

[prabirshrestha]

Just like how HOST, PORT and others just works, it would be good if compression() handler was a bit smarter to not compress images and binaries. It had one my site breaking when I deployed but not in localhost.

Might be something like a logger style api that could provide custom condition on whether to compress or not would be good. By default it can compress text, js, css, html, json and others can force compress if they want.

[jbr]

It had one my site breaking when I deployed but not in localhost.

Could you explain how it broke? I'm unaware of any reason that images and binaries can't/shouldn't be compressed. The client would avoid sending Accept-Encoding headers if that were the case. This suggests to me that I have a bug in how trillium-compression compresses binary content, so I'd love to understand more of how it fails. Were you streaming the binary content with trillium-static or sending it as fixed-length content with trillium-static-compiled?

[prabirshrestha]

When running directly in localhost it did work properly. But when I deployed it broke. I'm using caprover which is using nginx as the reverse proxy. Not sure if that changes anything but I'm using their base nginx config. https://github.com/caprover/caprover/blob/master/template/base-nginx-conf.ejs and https://github.com/caprover/caprover/blob/master/template/root-nginx-conf.ejs

I temporarily reverted it. prabirshrestha/rblog@2541667

I was also surprised why compressing images wouldn't work when deploying. I'm wondering if it has something to do with the order?

https://github.com/prabirshrestha/rblog/blob/05a0295afc06acdba504a3636eb82c4560583639/src/app.rs#L18-L19.

I'm currently using compression first and then caching headers. Wondering if it is something to do with ordering here.

[prabirshrestha]

Were you streaming the binary content with trillium-static or sending it as fixed-length content with trillium-static-compiled?

I'm using trillium-static with conn.send_path(path).

[jbr]

I believe this is an interaction with nginx, and if you are unable to modify the nginx configuration, it is best just not to use trillium-compression. I don't think there's a general-purpose need to disable/enable compression for only a subset of requests

[prabirshrestha]

I can modify the nginx config but was surprised why it can't handled properly since nginx by default is not doing any compression at all.

[jbr]

I think this is relevant: https://reinout.vanrees.org/weblog/2015/11/19/nginx-proxy-gzip.html

[prabirshrestha]

Setting proxy_http_version 1.1 on nginx config seems to work. Would be good if it was documented. Thanks for all the help.

[prabirshrestha]

Swapping the order doesn't seem to help.

Log for broken

* Connected to blog.prabir.me (66.94.115.178) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [4026 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=blog.prabir.me
*  start date: Nov 29 01:51:27 2021 GMT
*  expire date: Feb 27 01:51:26 2022 GMT
*  subjectAltName: host "blog.prabir.me" matched cert's "blog.prabir.me"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55ef449969f0)
} [5 bytes data]
> GET /posts/sharing-public-domain-with-tailscale-and-internal-network-with-one-to-one-nat-using-opnsense/opnsense-firewall-enable-nat.png HTTP/2
> Host: blog.prabir.me
> user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:96.0) Gecko/20100101 Firefox/96.0
> accept: image/avif,image/webp,*/*
> accept-language: en-US,en;q=0.5
> accept-encoding: gzip, deflate, br
> connection: keep-alive
> referer: https://blog.prabir.me/posts/sharing-public-domain-with-tailscale-and-internal-network-with-one-to-one-nat-using-opnsense/
> cookie: <----removed--->
> sec-fetch-dest: image
> sec-fetch-mode: no-cors
> sec-fetch-site: same-origin
> pragma: no-cache
> cache-control: no-cache
> te: trailers
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
} [5 bytes data]

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< HTTP/2 200 
< server: nginx
< date: Thu, 23 Dec 2021 00:30:49 GMT
< content-type: image/png
< content-encoding: br
< vary: Accept-Encoding
< x-request-id: FaDZdPPVTx
< last-modified: Thu, 23 Dec 2021 00:21:39 GMT
< etag: W/"1640218899.0-101503"
< 
{ [3867 bytes data]

Here is the broken file. https://www.dropbox.com/s/qbp7izxeu3vljme/opnsense-firewall-enable-nat.png?dl=0

I have disabled compression so my site is back to normal.