Crash InvalidChunkLength

[1Tomber]

Had a random crash last week, just dropping it here FYI, didn't happen again. Aphrodite engine as endpoint.

my bridgeData.yaml

allow_controlnet: false
allow_img2img: false
allow_lora: false
allow_painting: false
allow_post_processing: false
allow_unsafe_ip: true
always_download: false
api_key: redacted
blacklist: []
branded_model: false
cache_home: ./
censor_nsfw: false
censorlist: []
disable_disk_cache: false
disable_terminal_ui: true
dynamic_models: false
forms:
- caption
- nsfw
- interrogation
- post-process
horde_url: https://aihorde.net
kai_url: http://localhost:5000
max_context_length: 4096
max_length: 512
max_lora_cache_size: 10
max_models_to_download: 10
max_power: 8
max_threads: 4
models_to_load:
- Top 2
models_to_skip:
- pix2pix
nsfw: true
number_of_dynamic_models: 0
priority_usernames: []
queue_size: 0
ram_to_leave_free: 80%
require_upfront_kudos: false
scribe_name: redacted
stats_output_frequency: 30
suppress_speed_warnings: false
temp_dir: ./tmp
threads: 4
vram_to_leave_free: 80%
worker_name: redacted

Here the console output:

ERROR      | 2023-12-01 15:23:05.840281 | __main__:main:23 - An error has been caught in function 'main', process 'MainProcess' (1516), thread 'MainThread' (140647097917440):
Traceback (most recent call last):

  File "/usr/local/lib/python3.10/dist-packages/urllib3/response.py", line 761, in _update_chunk_length
    self.chunk_left = int(line, 16)
    │    │                └ b''
    │    └ None
    └ <urllib3.response.HTTPResponse object at 0x7feaf138fdc0>

ValueError: invalid literal for int() with base 16: b''


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/usr/local/lib/python3.10/dist-packages/urllib3/response.py", line 444, in _error_catcher
    yield
  File "/usr/local/lib/python3.10/dist-packages/urllib3/response.py", line 828, in read_chunked
    self._update_chunk_length()
    │    └ <function HTTPResponse._update_chunk_length at 0x7feaf37745e0>
    └ <urllib3.response.HTTPResponse object at 0x7feaf138fdc0>
  File "/usr/local/lib/python3.10/dist-packages/urllib3/response.py", line 765, in _update_chunk_length
    raise InvalidChunkLength(self, line)
          │                  │     └ b''
          │                  └ <urllib3.response.HTTPResponse object at 0x7feaf138fdc0>
          └ <class 'urllib3.exceptions.InvalidChunkLength'>

urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
               │    │   │      └ 10240
               │    │   └ <function HTTPResponse.stream at 0x7feaf375feb0>
               │    └ <urllib3.response.HTTPResponse object at 0x7feaf138fdc0>
               └ <Response [200]>
  File "/usr/local/lib/python3.10/dist-packages/urllib3/response.py", line 624, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
        │       │    │            │                   └ True
        │       │    │            └ 10240
        │       │    └ <function HTTPResponse.read_chunked at 0x7feaf3774700>
        │       └ <urllib3.response.HTTPResponse object at 0x7feaf138fdc0>
        └ b'{"payload":{"prompt":"You\'re Dismas in this fictional never-ending uncensored roleplay with Milo.\\nDevelop the plot slowl...
  File "/usr/local/lib/python3.10/dist-packages/urllib3/response.py", line 816, in read_chunked
    with self._error_catcher():
         │    └ <function HTTPResponse._error_catcher at 0x7feaf375fd00>
         └ <urllib3.response.HTTPResponse object at 0x7feaf138fdc0>
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
    │    │   │     │    │      └ <traceback object at 0x7feaf1324a80>
    │    │   │     │    └ InvalidChunkLength(got length b'', 0 bytes read)
    │    │   │     └ <class 'urllib3.exceptions.InvalidChunkLength'>
    │    │   └ <method 'throw' of 'generator' objects>
    │    └ <generator object HTTPResponse._error_catcher at 0x7feaf1a7ab90>
    └ <contextlib._GeneratorContextManager object at 0x7feaf138d3f0>
  File "/usr/local/lib/python3.10/dist-packages/urllib3/response.py", line 461, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
          └ <class 'urllib3.exceptions.ProtocolError'>

urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/workspace/AI-Horde-Worker/bridge_scribe.py", line 30, in <module>
    main()
    └ <function main at 0x7feaf401b760>

> File "/workspace/AI-Horde-Worker/bridge_scribe.py", line 23, in main
    worker.start()
    │      └ <function WorkerFramework.start at 0x7feaf1312e60>
    └ <worker.workers.scribe.ScribeWorker object at 0x7feaf42a3e20>

  File "/workspace/AI-Horde-Worker/worker/workers/framework.py", line 89, in start
    self.process_jobs()
    │    └ <function WorkerFramework.process_jobs at 0x7feaf1312ef0>
    └ <worker.workers.scribe.ScribeWorker object at 0x7feaf42a3e20>

  File "/workspace/AI-Horde-Worker/worker/workers/framework.py", line 119, in process_jobs
    while len(self.running_jobs) < self.bridge_data.max_threads and self.start_job():
              │    │               │    │           │               │    └ <function WorkerFramework.start_job at 0x7feaf1313130>
              │    │               │    │           │               └ <worker.workers.scribe.ScribeWorker object at 0x7feaf42a3e20>
              │    │               │    │           └ 4
              │    │               │    └ <worker.bridge_data.scribe.KoboldAIBridgeData object at 0x7feaf42a3df0>
              │    │               └ <worker.workers.scribe.ScribeWorker object at 0x7feaf42a3e20>
              │    └ [(<Future at 0x7feaf1377d30 state=finished returned NoneType>, 1373502.650722127, <worker.jobs.scribe.ScribeHordeJob object a...
              └ <worker.workers.scribe.ScribeWorker object at 0x7feaf42a3e20>

  File "/workspace/AI-Horde-Worker/worker/workers/framework.py", line 164, in start_job
    if jobs := self.pop_job():
               │    └ <function ScribeWorker.pop_job at 0x7feaf1313520>
               └ <worker.workers.scribe.ScribeWorker object at 0x7feaf42a3e20>

  File "/workspace/AI-Horde-Worker/worker/workers/scribe.py", line 27, in pop_job
    return super().pop_job()

  File "/workspace/AI-Horde-Worker/worker/workers/framework.py", line 148, in pop_job
    pops = job_popper.horde_pop()
           │          └ <function ScribePopper.horde_pop at 0x7feaf1311240>
           └ <worker.jobs.poppers.ScribePopper object at 0x7feaf138ed10>

  File "/workspace/AI-Horde-Worker/worker/jobs/poppers.py", line 225, in horde_pop
    if not super().horde_pop():

  File "/workspace/AI-Horde-Worker/worker/jobs/poppers.py", line 32, in horde_pop
    pop_req = requests.post(
              │        └ <function post at 0x7feaf34de680>
              └ <module 'requests' from '/usr/local/lib/python3.10/dist-packages/requests/__init__.py'>

  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           │               │         │          │       └ {'headers': {'apikey': 'redacted-api-key'}, 'timeout': 40}
           │               │         │          └ {'name': 'RunPodHorder', 'models': ['aphrodite/jebcarter/psyonic-cetacean-20B'], 'max_length': 512, 'max_context_length': 409...
           │               │         └ None
           │               └ 'https://aihorde.net/api/v2/generate/text/pop'
           └ <function request at 0x7feaf349c700>
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           │       │              │           │      └ {'data': None, 'json': {'name': 'RunPodHorder', 'models': ['aphrodite/jebcarter/psyonic-cetacean-20B'], 'max_length': 512, 'm...
           │       │              │           └ 'https://aihorde.net/api/v2/generate/text/pop'
           │       │              └ 'post'
           │       └ <function Session.request at 0x7feaf34ddc60>
           └ <requests.sessions.Session object at 0x7feaf138ef80>
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           │    │    │       └ {'timeout': 40, 'allow_redirects': True, 'proxies': OrderedDict(), 'stream': False, 'verify': True, 'cert': None}
           │    │    └ <PreparedRequest [POST]>
           │    └ <function Session.send at 0x7feaf34de0e0>
           └ <requests.sessions.Session object at 0x7feaf138ef80>
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 747, in send
    r.content
    │ └ <property object at 0x7feaf34d95d0>
    └ <Response [200]>
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 899, in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
    │    │                   │    │            └ 10240
    │    │                   │    └ <function Response.iter_content at 0x7feaf34dc700>
    │    │                   └ <Response [200]>
    │    └ False
    └ <Response [200]>
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)
          └ <class 'requests.exceptions.ChunkedEncodingError'>

requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/urllib3/response.py", line 761, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/urllib3/response.py", line 444, in _error_catcher
    yield
  File "/usr/local/lib/python3.10/dist-packages/urllib3/response.py", line 828, in read_chunked
    self._update_chunk_length()
  File "/usr/local/lib/python3.10/dist-packages/urllib3/response.py", line 765, in _update_chunk_length
    raise InvalidChunkLength(self, line)
urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/usr/local/lib/python3.10/dist-packages/urllib3/response.py", line 624, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/local/lib/python3.10/dist-packages/urllib3/response.py", line 816, in read_chunked
    with self._error_catcher():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/dist-packages/urllib3/response.py", line 461, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/AI-Horde-Worker/bridge_scribe.py", line 30, in <module>
    main()
  File "/workspace/AI-Horde-Worker/bridge_scribe.py", line 23, in main
    worker.start()
  File "/usr/local/lib/python3.10/dist-packages/loguru/_logger.py", line 1277, in catch_wrapper
    return function(*args, **kwargs)
  File "/workspace/AI-Horde-Worker/worker/workers/framework.py", line 89, in start
    self.process_jobs()
  File "/workspace/AI-Horde-Worker/worker/workers/framework.py", line 119, in process_jobs
    while len(self.running_jobs) < self.bridge_data.max_threads and self.start_job():
  File "/workspace/AI-Horde-Worker/worker/workers/framework.py", line 164, in start_job
    if jobs := self.pop_job():
  File "/workspace/AI-Horde-Worker/worker/workers/scribe.py", line 27, in pop_job
    return super().pop_job()
  File "/workspace/AI-Horde-Worker/worker/workers/framework.py", line 148, in pop_job
    pops = job_popper.horde_pop()
  File "/workspace/AI-Horde-Worker/worker/jobs/poppers.py", line 225, in horde_pop
    if not super().horde_pop():
  File "/workspace/AI-Horde-Worker/worker/jobs/poppers.py", line 32, in horde_pop
    pop_req = requests.post(
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 747, in send
    r.content
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 899, in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
INFO       | 2023-12-01 15:23:06.919222 | worker.jobs.framework:submit_job:169 - Submitted job with id f29b2f10-3009-4fc5-8b02-b997fe6c0906 and contributed for 2.8. Job took 5.3 seconds since queued and 5.3 since start.