An error occurred while receiving a compressed request by deflate method

[AlexanderMartynoff]

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/aiohttp/web_protocol.py", line 461, in start
yield from payload.readany()
File "/usr/local/lib/python3.6/site-packages/aiohttp/streams.py", line 558, in readany
return (yield from super().readany())
File "/usr/local/lib/python3.6/site-packages/aiohttp/streams.py", line 305, in readany
raise self._exception

aiohttp.web_protocol.RequestPayloadError: 400, message='Can not decode content-encoding: deflate'

python-3.6
aiohttp-2.0.7

[gsakkis]

I am probably hitting this bug on the following example:

import aiohttp
import asyncio

async def main(path):
    async with aiohttp.ClientSession() as client:
        url = 'http://httpbin.org' + path
        async with client.get(url) as resp:
            assert resp.status == 200
            print(url)
            print(resp.headers)
            print(await resp.json())
            print()


loop = asyncio.get_event_loop()
loop.run_until_complete(main('/gzip'))
loop.run_until_complete(main('/deflate'))

Output:

http://httpbin.org/gzip
<CIMultiDictProxy('Connection': 'keep-alive', 'Server': 'meinheld/0.6.1', 'Date': 'Thu, 15 Jun 2017 19:56:44 GMT', 'Content-Type': 'application/json', 'Content-Length': '194', 'Content-Encoding': 'gzip', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'X-Powered-By': 'Flask', 'X-Processed-Time': '0.00179004669189', 'Via': '1.1 vegur')>
{'gzipped': True, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'close', 'Host': 'httpbin.org', 'User-Agent': 'Python/3.6 aiohttp/2.1.0'}, 'method': 'GET', 'origin': '78.87.217.47'}

http://httpbin.org/deflate
<CIMultiDictProxy('Connection': 'keep-alive', 'Server': 'meinheld/0.6.1', 'Date': 'Thu, 15 Jun 2017 19:56:46 GMT', 'Content-Type': 'application/json', 'Content-Length': '181', 'Content-Encoding': 'deflate', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'X-Powered-By': 'Flask', 'X-Processed-Time': '0.000823020935059', 'Via': '1.1 vegur')>
Traceback (most recent call last):
  File "client.py", line 17, in <module>
    loop.run_until_complete(main('/deflate'))
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 466, in run_until_complete
    return future.result()
  File "client.py", line 11, in main
    print(await resp.json())
  File "/home/gsk/aiohttp/venv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 712, in json
    yield from self.read()
  File "/home/gsk/aiohttp/venv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 673, in read
    self._content = yield from self.content.read()
  File "/home/gsk/aiohttp/venv/lib/python3.6/site-packages/aiohttp/streams.py", line 562, in read
    return (yield from super().read(n))
  File "/home/gsk/aiohttp/venv/lib/python3.6/site-packages/aiohttp/streams.py", line 275, in read
    raise self._exception
aiohttp.client_exceptions.ClientPayloadError: 400, message='Can not decode content-encoding: deflate'

[fafhrd91]

I am not sure what http://httpbin.org/deflate should return,
wget can not decode response and just stores raw data in file, but wget can decode http://httpbin.org/gzip response and stores some json.
requests can decode response. I guess this is bug in httpbin.org and requests

I am not sure what we should do with not decodable responses.

@asvetlov @kxepal

[gsakkis]

Here's what http://httpbin.org/deflate does. Browsers (at least chrome and firefox) have no trouble decoding it.

[fafhrd91]

seems browsers just support buggy servers. ok, we need to support it too

[gsakkis]

What is buggy about it? Or alternatively, how does a correct deflate response look like?

[fafhrd91]

-zlib.MAX_WBITS

[fafhrd91]

wget can not parse response

[fafhrd91]

@asvetlov what do you think? with current implementation wget can read compressed responses but requests can not.

[fafhrd91]

browsers seems can read any

[asvetlov]

If browsers support the case and we can implement it without very ugly hacks -- let's do it.

[fafhrd91]

question is should we do this at all, because browsers support current implementation, only requests doesn't. question about server side

[fafhrd91]

fixed in master

[asvetlov]

question is should we do this at all, because browsers support current implementation, only requests doesn't. question about server side

I don't follow. How it affects sever side?

[fafhrd91]

I fixed DeflateBuffer. But when we compress data with use -zlib.MAX_WBITS, browsers can read response, wget can too, but requests does not. reading part is easy (fixed already) but writing part is tricky.

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.