Exception 'LookupError: unknown encoding:' when response contains content-type not handled by requests.utils.get_encoding_from_headers

[MiltiadisKoutsokeras]

Prerequisites

• I am using the latest version of Locust
• I am reporting a bug, not asking a question

Description

When the tested HTTP server returns a Response header of content-type that is not handled by the external library method requests.utils.get_encoding_from_headers properly (for example content-type: application/problem+json) the calling code sets encoding to empty string which results in a bytes to string encoding error here. The thrown exception stack trace:

Traceback (most recent call last):
  File "/GitHub-Issues/locustio/locust/UnexpectedResponseHeaderContentType/.venv/lib/python3.9/site-packages/locust/user/task.py", line 343, in run
    self.execute_next_task()
  File "/GitHub-Issues/locustio/locust/UnexpectedResponseHeaderContentType/.venv/lib/python3.9/site-packages/locust/user/task.py", line 374, in execute_next_task
    self.execute_task(self._task_queue.pop(0))
  File "/GitHub-Issues/locustio/locust/UnexpectedResponseHeaderContentType/.venv/lib/python3.9/site-packages/locust/user/task.py", line 495, in execute_task
    task(self.user)
  File "/GitHub-Issues/locustio/locust/UnexpectedResponseHeaderContentType/locustfile.py", line 25, in reproduce_issue
    with self.rest(http_method, url=endpoint,
  File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "/GitHub-Issues/locustio/locust/UnexpectedResponseHeaderContentType/.venv/lib/python3.9/site-packages/locust/contrib/fasthttp.py", line 382, in rest
    if resp.text is None:
  File "/GitHub-Issues/locustio/locust/UnexpectedResponseHeaderContentType/.venv/lib/python3.9/site-packages/locust/contrib/fasthttp.py", line 467, in text
    return str(self.content, self.encoding, errors="replace")
LookupError: unknown encoding:

Command line

locust --web-host 127.0.0.1 --web-port 9999 -f locustfile.py

Locustfile contents

"""Minimal file to reproduce issue of unexpected Response header 'content-type'
"""

import logging

from locust import FastHttpUser, between, task


class UnexpectedResponseHeaderContentType(FastHttpUser):
    """Minimal problem reproduce class.
    """

    wait_time = between(0.5, 3)
    api_key = None

    @task
    def reproduce_issue(self):
        """Task to reproduce the issue.
        """

        http_method = 'GET'
        endpoint = '/api'

        result = None
        with self.rest(http_method, url=endpoint,
                       name='Task to reproduce the issue') as resp:
            if resp.js is None:
                return None

            result = resp.js

        logging.info('Result: %s', result)

Python version

3.9.2

Locust version

2.19.0

Operating system

Debian 11

[cyberw]

Apparently requests has a fallback to using chardet https://github.com/psf/requests/blob/0b4d494192de489701d3a2e32acef8fb5d3f042e/src/requests/models.py#L908 Maybe you could implement this? Or maybe we should just assume utf-8 or something...

[MiltiadisKoutsokeras]

I will provide a PR when I find the time. The proper solution would be of course to also handle properly the possible response headers in requests package also.

[MiltiadisKoutsokeras]

PR created: #2484

[MiltiadisKoutsokeras]

Another PR which uses detection instead of default utf-8: #2485

Choose the one you prefer.

[cyberw]

Fixed

[prince-melvin]

Hi,

When the content-type returned in the response is application/octet-stream I am getting the following stacktrace

Traceback (most recent call last):
  File "C:\Users\princemelvin\AppData\Local\Programs\Python\Python312\Lib\site-packages\locust\user\task.py", line 343, in run
    self.execute_next_task()
  File "C:\Users\princemelvin\AppData\Local\Programs\Python\Python312\Lib\site-packages\locust\user\task.py", line 376, in execute_next_task
    self.execute_task(self._task_queue.pop(0))
  File "C:\Users\princemelvin\AppData\Local\Programs\Python\Python312\Lib\site-packages\locust\user\task.py", line 497, in execute_task
    task(self.user)
  File "D:\Comp_Benchmark\LoadTest\download_512mb_to_file_4mb_1task_fullfile.py", line 49, in t
    with self.rest("GET", url = blob_url_sas, headers = {"x-ms-range" : f"bytes={runningOffset}-{runningOffset + length}", "Accept" : "application/octet-stream"}) as response:
  File "C:\Users\princemelvin\AppData\Local\Programs\Python\Python312\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\princemelvin\AppData\Local\Programs\Python\Python312\Lib\site-packages\locust\contrib\fasthttp.py", line 383, in rest
    if resp.text is None:
       ^^^^^^^^^
  File "C:\Users\princemelvin\AppData\Local\Programs\Python\Python312\Lib\site-packages\locust\contrib\fasthttp.py", line 473, in text
    return str(self.content, self.encoding, errors="replace")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
LookupError: unknown encoding: None

Since we are checking resp.text is None it is failing the request. Should we do resp.Content check as well if resp.text is None since text encoding couldn't be detected?

[cyberw]

Are you really on latest version?

[prince-melvin]

Yes.
locust 2.19.1 from C:\Users\princemelvin\AppData\Local\Programs\Python\Python312\Lib\site-packages\locust (python 3.12.0)

[cyberw]

Ah, my line numbers weren't matching yours because there had been a few more changes to the file after the release.

[cyberw]

I'll change it to check .content instead.

[cyberw]

or... will that help? .content isnt None in your case, is it? PR welcome :)

[prince-melvin]

Yeah it isn't none. I'll be happy to raise a PR.
Will do it by today or tomorrow.

[prince-melvin]

Raised a PR - #2512, Please review and let me know.