Got more than 8190 bytes when reading Header value is too long.

[salmosri]

Hi Team,

When attempting to carry out a move of Spotify -> Google the following error appears which causes the app to hang (but not crash):

Listening on http://0.0.0.0:3132
Please open your browser window to http://localhost:3132
Gathering tracks for playlist Saved Tracks (784)
Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 559, in start
(message, payload) = yield from self._protocol.read()
File "/usr/local/lib/python3.6/site-packages/aiohttp/streams.py", line 509, in read
yield from self._waiter
File "/usr/local/lib/python3.6/site-packages/aiohttp/client_proto.py", line 165, in data_received
messages, upgraded, tail = self._parser.feed_data(data)
File "aiohttp/_http_parser.pyx", line 274, in aiohttp._http_parser.HttpParser.feed_data (aiohttp/_http_parser.c:4364)
File "aiohttp/_http_parser.pyx", line 382, in aiohttp._http_parser.cb_on_header_value (aiohttp/_http_parser.c:6391)
aiohttp.http_exceptions.LineTooLong: 400, message='Got more than 8190 bytes when reading Header value is too long.'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/aiohttp/web_protocol.py", line 422, in start
resp = yield from self._request_handler(request)
File "/usr/local/lib/python3.6/site-packages/aiohttp/web.py", line 306, in _handle
resp = yield from handler(request)
File "/usr/local/lib/python3.6/site-packages/pyportify/middlewares.py", line 18, in index_handler
ret = yield from handler(request)
File "/usr/local/lib/python3.6/site-packages/pyportify/app.py", line 117, in transfer_start
yield from transfer_playlists(request, s, g, lists)
File "/usr/local/lib/python3.6/site-packages/pyportify/app.py", line 169, in transfer_playlists
done = yield from asyncio.gather(*tasks)
File "/usr/local/lib/python3.6/site-packages/pyportify/app.py", line 242, in search_gm_track
track = yield from g.find_best_track(search_query)
File "/usr/local/lib/python3.6/site-packages/pyportify/google.py", line 56, in find_best_track
data = yield from self.search_all_access(search_query)
File "/usr/local/lib/python3.6/site-packages/pyportify/google.py", line 48, in search_all_access
'ct': '1,2,3,4,6,7,8,9',
File "/usr/local/lib/python3.6/site-packages/pyportify/google.py", line 113, in _http_get
params=merged_params,
File "/usr/local/lib/python3.6/site-packages/aiohttp/helpers.py", line 97, in iter
ret = yield from self._coro
File "/usr/local/lib/python3.6/site-packages/aiohttp/client.py", line 241, in _request
yield from resp.start(conn, read_until_eof)
File "/usr/local/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 564, in start
message=exc.message, headers=exc.headers) from exc
aiohttp.client_exceptions.ClientResponseError: 400, message='Got more than 8190 bytes when reading Header value is too long.'

==============

This issue exists while using the install method of brew or downloading the dmg and running the pyportify-copyall script.

I am using MacOS Sierra 10.12.6

Let me know if there is anything i can do to resolve this or provide more information!

Cheers,
Shadi

[rckclmbr]

It looks like aiohttp limits header length to 8190 in their response parser. I don't see any easy way in their API to change this length, since this is hidden behind a few layers of abstraction. I'll dig into this and see what I can do.

There's a C and python parser, but they both have this param.

https://github.com/aio-libs/aiohttp/blob/e7c9390111932dd6dbb642170d7a0da1876271ec/aiohttp/http_parser.py#L59

[robhazan]

I'm encountering the same error on Ubuntu. Any debugging information I can provide to help get this resolved? Anything I can change to avoid this error?

[robhazan]

@rckclmbr - friendly ping to see if there's anything that can be done about this. Any workarounds?

[rckclmbr]

@robhazan there's not much I can do about this short of moving away from aiohttp, since aiohttp doesn't support what we need it to. We need to wait until aio-libs/aiohttp#2304 is resolved (they set the milestone for 3.1) then we can work around this.

The other approach is to figure out why the header is exceeding 8190 -- it's not all users, and I can't troubleshoot it since it's not happening to me. Maybe there's something we can do to make google not have such a large header, if we could find what it actually is.

The hackish solution is to manually edit aiohttp to allow a longer header. I don't think it's a good idea to make a release with that though, for a variety of reasons...

[dIgor93]

Hello!
I found some solution for same problem. Pass the handler_args parameter when creating app. It will pass to parser.

web.Application(handler_args={'max_field_size': 16380})

[magnus919]

Running into this years later with Home Assistant.

[danelec-hpm]

Even 2024, with all the tokens and suff we have in headers today, who was crazy enough to set an 8 kb limit. Did we not learn from Bill Gates "640kb should be enough for everyone" resulting in the Gate A20 in the Bios Keyboard to be used to access High Memory above 640kb.

[TVidori]

I had the same issue. I found the solution on: https://stackoverflow.com/questions/69239895/aiohttp-got-more-than-8190-bytes-xxxx-when-reading-header-value-is-too-long
In short, increase the max_field_size argument of the aiohttp.ClientSession. For instance, to double it:

session = aiohttp.ClientSession(
    max_field_size=8190 * 2,
)