500s when handling a request with wrong Content-Encoding

[rafbarr]

Long story short

When serving a request with wrong Content-Encoding (e.g. deflate while payload is actually not compressed), I get 500 Internal Server Error instead of a 400 Bad Request.

Expected behaviour

I would expect to get a 400 Bad Request to inform the client they are not requesting correctly.

Actual behaviour

It raises a aiohttp.errors.ContentEncodingError exception. If I capture this exception in the handler, I can change the response. However, after terminating the application I get the following stack trace:

2017-03-10 12:07:47,178 aiohttp.access[24766] INFO: 127.0.0.1 - - [10/Mar/2017:20:07:47 +0000] "POST /recommend HTTP/1.1" 400 16 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"
2017-03-10 12:07:47,178 aiohttp.server[24766] DEBUG: Uncompleted request.
2017-03-10 12:07:47,178 aiohttp.server[24766] DEBUG: Start lingering close timer for 30.0 sec.
^C2017-03-10 12:07:52,232 asyncio[24766] ERROR: Task exception was never retrieved
future: <Task finished coro=<ServerHttpProtocol.start() done, defined at /Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/server.py:225> exception=RuntimeError('Cannot call write() after write_eof()',)>
Traceback (most recent call last):
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/protocol.py", line 389,in feed_data
    chunk = self.zlib.decompress(chunk)
zlib.error: Error -3 while decompressing data: invalid block type

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/server.py", line 297, in start
    yield from payload.readany()
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/streams.py", line 498, in wrapper
    result = yield from func(self, *args, **kw)
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/streams.py", line 575, in readany
    return (yield from super().readany())
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/streams.py", line 285, in readany
    raise self._exception
  File "/Users/rafaelbarreto/Dropbox/Coding/repos/vungle/nostradamus/nostradamus/api_server2/app.py", line 34, in middleware_handler
    return await handler(request)
  File "/Users/rafaelbarreto/Dropbox/Coding/repos/vungle/nostradamus/nostradamus/api_server2/app.py", line 24, in _ping_handler
    data = await request.json(loads=ujson.loads)
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/web_reqrep.py", line 404, in json
    body = yield from self.text()
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/web_reqrep.py", line 392, in text
    bytes_body = yield from self.read()
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/web_reqrep.py", line 382, in read
    chunk = yield from self._payload.readany()
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/streams.py", line 498, in wrapper
    result = yield from func(self, *args, **kw)
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/streams.py", line 575, in readany
    return (yield from super().readany())
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/streams.py", line 285, in readany
    raise self._exception
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/parsers.py", line 188, in set_parser
    next(p)
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/protocol.py", line 316,in __call__
    yield from self.parse_length_payload(out, buf, length)
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/protocol.py", line 362,in parse_length_payload
    out.feed_data(chunk, len(chunk))
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/protocol.py", line 391,in feed_data
    raise errors.ContentEncodingError('deflate')
aiohttp.errors.ContentEncodingError: 400, message='deflate'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/server.py", line 319, in start
    exc.message)
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/server.py", line 371, in handle_error
    response.send_headers()
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/site-packages/aiohttp/protocol.py", line 685,in send_headers
    self.transport.write(headers)
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/asyncio/streams.py", line 294, in write
    self._transport.write(data)
  File "/Users/rafaelbarreto/.miniconda3/envs/nostradamus/lib/python3.6/asyncio/selector_events.py", line 749, in write
    raise RuntimeError('Cannot call write() after write_eof()')
RuntimeError: Cannot call write() after write_eof()

Steps to reproduce

1. Use the following app:

async def _ping_handler(request):
    data = await request.json(loads=ujson.loads)
    resp = web.json_response(data, dumps=ujson.dumps)
    resp.enable_compression()
    return resp

app = web.Application()
app.router.add_post('/ping', _ping_handler)

2. Submit a request to it with a bad Content-Encoding header. For example, Content-Encoding: gzip but an uncompressed payload.

Your environment

I'm using aiohttp==1.3.3 with Python 3.6.

[fafhrd91]

actually this is interesting. question is how to handle this situation.
technically http message is valid but payload is broken, so client can receive response message without exception.

in any case I am going to fix this bug in version 2.0 in https://github.com/aio-libs/aiohttp

[fafhrd91]

fixed in 2.0

[rafbarr]

@fafhrd91, thanks for the quick respond. Is the development moving to https://github.com/aio-libs/aiohttp?

[fafhrd91]

Yes, development is moving to aio-libs. I am planing to release 2.0b early next week

…

Sent from my iPhone

On Mar 11, 2017, at 11:14 PM, Rafael Barreto ***@***.***> wrote: @fafhrd91, thanks for the quick respond. Is the development moving to https://github.com/aio-libs/aiohttp? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[fafhrd91]

fixed in master

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.