DownloadLengthMismatchError because requests automatically decodes gzip files

[dennisvang]

Description of issue or feature request:

If target is a gzip file, served with a Content-Encoding: gzip header, requests will automatically decode the file.

Current behavior:

As a result, the default fetcher for tuf ngclient raises a DownloadLengthMismatchError:

tuf.api.exceptions.DownloadLengthMismatchError: Downloaded x bytes exceeding the maximum allowed length of y

Expected behavior:

From the requests docs:

An important note about using Response.iter_content versus Response.raw. Response.iter_content will automatically decode the gzip and deflate transfer-encodings. Response.raw is a raw stream of bytes – it does not transform the response content. If you really need access to the bytes as they were returned, use Response.raw.

I would expect tuf to access the Reponse.raw instead of Response.iter_content here:

python-tuf/tuf/ngclient/_internal/requests_fetcher.py

Line 99 in b5e952c

for data in response.iter_content(self.chunk_size):

Note that Response.raw is a urllib3.response.HTTPResponse, which also decodes gzip encoded content by default, but requests disables this by setting decode_content=False.

[joshuagl]

Thanks for the detailed report and for proposing a fix @dennisvang. Would you be interested in creating a Pull Request to implement the proposed fix?

[dennisvang]

@joshuagl Yes, I would be happy to create a pull request for this.

Actually I already started working on one, but it's taking a bit longer than anticipated.

[joshuagl]

Brilliant, thank you

[dennisvang]

@joshuagl I believe PR #2048 does the job, although I'm not entirely sure my method of testing is acceptable.

[lukpueh]

This is not a bug in the client but in the server. See #1774 and #2048 (comment) pp. for details.