Ollama ResponseNotRead: Attempted to access streaming response content, without having called read().

[alpiyev99]

How to fix it?

[electriquo]

i am having the same problem. for more information, that's what i did

in one terminal window, spin up ollama container

$ docker run --rm -it -e OLLAMA_DEBUG=1 -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
time=2024-03-25T11:32:37.135Z level=INFO source=images.go:806 msg="total blobs: 0"
time=2024-03-25T11:32:37.135Z level=INFO source=images.go:813 msg="total unused blobs removed: 0"
time=2024-03-25T11:32:37.135Z level=INFO source=routes.go:1110 msg="Listening on [::]:11434 (version 0.1.29)"
time=2024-03-25T11:32:37.135Z level=INFO source=payload_common.go:112 msg="Extracting dynamic libraries to /tmp/ollama3513124258/runners ..."
time=2024-03-25T11:32:38.984Z level=INFO source=payload_common.go:139 msg="Dynamic LLM libraries [cpu cuda_v11]"
time=2024-03-25T11:32:38.984Z level=DEBUG source=payload_common.go:140 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2024-03-25T11:32:38.984Z level=INFO source=gpu.go:77 msg="Detecting GPU type"
time=2024-03-25T11:32:38.984Z level=INFO source=gpu.go:191 msg="Searching for GPU management library libnvidia-ml.so"
time=2024-03-25T11:32:38.984Z level=DEBUG source=gpu.go:209 msg="gpu management search paths: [/usr/local/cuda/lib64/libnvidia-ml.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-ml.so* /usr/lib/x86_64-linux-gnu/libnvidia-ml.so* /usr/lib/wsl/lib/libnvidia-ml.so* /usr/lib/wsl/drivers/*/libnvidia-ml.so* /opt/cuda/lib64/libnvidia-ml.so* /usr/lib*/libnvidia-ml.so* /usr/local/lib*/libnvidia-ml.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libnvidia-ml.so* /usr/lib/aarch64-linux-gnu/libnvidia-ml.so* /opt/cuda/targets/x86_64-linux/lib/stubs/libnvidia-ml.so* /usr/local/nvidia/lib/libnvidia-ml.so* /usr/local/nvidia/lib64/libnvidia-ml.so*]"
time=2024-03-25T11:32:38.984Z level=INFO source=gpu.go:237 msg="Discovered GPU libraries: []"
time=2024-03-25T11:32:38.984Z level=INFO source=cpu_common.go:18 msg="CPU does not have vector extensions"
time=2024-03-25T11:32:38.984Z level=DEBUG source=amd_linux.go:263 msg="amdgpu driver not detected /sys/module/amdgpu"
time=2024-03-25T11:32:38.984Z level=INFO source=routes.go:1133 msg="no GPU detected"

i modified the shell_gpt configuration to ollama as described in docs and installed litellm and started shell_gpt in another terminal window

$ python --version
Python 3.12.2

$ cat requirements.txt
litellm==1.24.5
shell_gpt==1.4.0

$ sgpt "Who are you?"
...
│ in content                                                                                       │
│                                                                                                  │
│    567 │   @property                                                                             │
│    568 │   def content(self) -> bytes:                                                           │
│    569 │   │   if not hasattr(self, "_content"):                                                 │
│ ❱  570 │   │   │   raise ResponseNotRead()                                                       │
│    571 │   │   return self._content                                                              │
│    572 │                                                                                         │
│    573 │   @property                                                                             │
│                                                                                                  │
│ ╭───────────── locals ──────────────╮                                                            │
│ │ self = <Response [404 Not Found]> │                                                            │
│ ╰───────────────────────────────────╯                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ResponseNotRead: Attempted to access streaming response content, without having called `read()`.

in the ollama terminal, i got also an error

[GIN] 2024/03/25 - 11:32:47 | 404 |     337.875µs |      172.17.0.1 | POST     "/api/generate"

i also tried using shell_gpt version 1.3.0 and 1.3.1 as noted in #467 but the error remains.

[dcleri]

Same problem here.
I am using ollama/llama3

this is the output from ollama serve when running sgpt prompt:

ollama serve

time=2024-04-24T11:33:09.589+02:00 level=INFO source=images.go:817 msg="total blobs: 0"
time=2024-04-24T11:33:09.589+02:00 level=INFO source=images.go:824 msg="total unused blobs removed: 0"time=2024-04-24T11:33:09.589+02:00 level=INFO source=routes.go:1143 msg="Listening on 127.0.0.1:11434 (version 0.1.32)"
time=2024-04-24T11:33:09.589+02:00 level=INFO source=payload.go:28 msg="extracting embedded files" dir=/tmp/ollama548878943/runnerstime=2024-04-24T11:33:11.096+02:00 level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cuda_v11 rocm_v60002 cpu cpu_avx cpu_avx2]"
time=2024-04-24T11:33:11.096+02:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"time=2024-04-24T11:33:11.096+02:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"time=2024-04-24T11:33:11.098+02:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama548878943/runners/cuda_v11/libcudart.so.11.0]"
time=2024-04-24T11:33:11.098+02:00 level=INFO source=gpu.go:343 msg="Unable to load cudart CUDA management library /tmp/ollama548878943/runners/cuda_v11/libcudart.so.11.0: your nvidia driver is too old or missing, please upgrade to run ollama"time=2024-04-24T11:33:11.098+02:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library libnvidia-ml.so"time=2024-04-24T11:33:11.099+02:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: []"
time=2024-04-24T11:33:11.099+02:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"time=2024-04-24T11:33:11.099+02:00 level=INFO source=amd_linux.go:50 msg="AMD Driver: 6.3.6"
time=2024-04-24T11:33:11.100+02:00 level=INFO source=amd_linux.go:88 msg="detected amdgpu versions [gfx1100 gfx1036]"time=2024-04-24T11:33:11.101+02:00 level=INFO source=amd_linux.go:121 msg="amdgpu [0] gfx1100 is supported"
time=2024-04-24T11:33:11.101+02:00 level=WARN source=amd_linux.go:116 msg="amdgpu [1] gfx1036 is not supported by /tmp/ollama548878943/rocm [gfx1030 gfx1100 gfx1101 gfx1102 gfx900 gfx906 gfx908 gfx90a gfx940 gfx941 gfx942]"time=2024-04-24T11:33:11.101+02:00 level=WARN source=amd_linux.go:118 msg="See https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides for HSA_OVERRIDE_GFX_VERSION usage"
time=2024-04-24T11:33:11.101+02:00 level=INFO source=amd_linux.go:263 msg="[0] amdgpu totalMemory 46064M"time=2024-04-24T11:33:11.101+02:00 level=INFO source=amd_linux.go:264 msg="[0] amdgpu freeMemory  46064M"
time=2024-04-24T11:33:11.101+02:00 level=INFO source=amd_common.go:54 msg="Setting HIP_VISIBLE_DEVICES=0"
[GIN] 2024/04/24 - 11:35:03 | 404 |    1.812173ms |       127.0.0.1 | POST     "/api/generate"
[GIN] 2024/04/24 - 11:35:59 | 404 |     114.635µs |       127.0.0.1 | POST     "/api/generate"
[GIN] 2024/04/24 - 11:39:40 | 404 |     124.072µs |       127.0.0.1 | POST     "/api/generate"

[gra]

Make sure that
API_BASE_URL=http://localhost:11434
has no / in the end

[richardstevenhack]

I have the same problem. I am using the correct API_BASE_URL as mentioned above. Here is the output I get:

`rhack@localhost:/Data2/Work> sgpt --model ollama/gwen2:latest "Who are you?"

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/rhack/.local/share/pipx/venvs/shell-gpt/lib64/python3.11/site-packages/sgpt/app.py:229 in │
│ main │
│ │
│ 226 │ │ │ functions=function_schemas, │
│ 227 │ │ ) │
│ 228 │ else: │
│ ❱ 229 │ │ full_completion = DefaultHandler(role_class, md).handle( │
│ 230 │ │ │ prompt=prompt, │
│ 231 │ │ │ model=model, │
│ 232 │ │ │ temperature=temperature, │
│ │
│ ╭─────────────────────────────── locals ────────────────────────────────╮ │
│ │ cache = True │ │
│ │ chat = None │ │
│ │ code = False │ │
│ │ create_role = None │ │
│ │ describe_shell = False │ │
│ │ editor = False │ │
│ │ function_schemas = None │ │
│ │ functions = False │ │
│ │ install_functions = None │ │
│ │ install_integration = None │ │
│ │ interaction = True │ │
│ │ list_chats = None │ │
│ │ list_roles = None │ │
│ │ md = True │ │
│ │ model = 'ollama/gwen2:latest' │ │
│ │ prompt = 'Who are you?' │ │
│ │ repl = None │ │
│ │ role = None │ │
│ │ role_class = <sgpt.role.SystemRole object at 0x7f09ae7ff210> │ │
│ │ shell = False │ │
│ │ show_chat = None │ │
│ │ show_role = None │ │
│ │ stdin_passed = False │ │
│ │ temperature = 0.0 │ │
│ │ top_p = 1.0 │ │
│ │ version = None │ │
│ ╰───────────────────────────────────────────────────────────────────────╯ │
│ │
│ /home/rhack/.local/share/pipx/venvs/shell-gpt/lib64/python3.11/site-packages/sgpt/handlers/handl │
│ er.py:158 in handle │
│ │
│ 155 │ │ │ caching=caching, │
│ 156 │ │ │ **kwargs, │
│ 157 │ │ ) │
│ ❱ 158 │ │ return self.printer(generator, not disable_stream) │
│ 159 │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ caching = True │ │
│ │ disable_stream = False │ │
│ │ functions = None │ │
│ │ generator = <generator object Cache.call..wrapper at 0x7f09ae634280> │ │
│ │ kwargs = {} │ │
│ │ messages = [ │ │
│ │ │ { │ │
│ │ │ │ 'role': 'system', │ │
│ │ │ │ 'content': 'You are ShellGPT\nYou are programming and system │ │
│ │ administration assistant.\nYou ar'+284 │ │
│ │ │ }, │ │
│ │ │ {'role': 'user', 'content': 'Who are you?'} │ │
│ │ ] │ │
│ │ model = 'ollama/gwen2:latest' │ │
│ │ prompt = 'Who are you?' │ │
│ │ self = <sgpt.handlers.default_handler.DefaultHandler object at 0x7f09afe361d0> │ │
│ │ temperature = 0.0 │ │
│ │ top_p = 1.0 │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /home/rhack/.local/share/pipx/venvs/shell-gpt/lib64/python3.11/site-packages/sgpt/printer.py:23 │
│ in call │
│ │
│ 20 │ │
│ 21 │ def call(self, chunks: Generator[str, None, None], live: bool = True) -> str: │
│ 22 │ │ if live: │
│ ❱ 23 │ │ │ return self.live_print(chunks) │
│ 24 │ │ with self.console.status("[bold green]Loading..."): │
│ 25 │ │ │ full_completion = "".join(chunks) │
│ 26 │ │ self.static_print(full_completion) │
│ │
│ ╭─────────────────────────────────── locals ────────────────────────────────────╮ │
│ │ chunks = <generator object Cache.call..wrapper at 0x7f09ae634280> │ │
│ │ live = True │ │
│ │ self = <sgpt.printer.MarkdownPrinter object at 0x7f09afd18750> │ │
│ ╰───────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /home/rhack/.local/share/pipx/venvs/shell-gpt/lib64/python3.11/site-packages/sgpt/printer.py:38 │
│ in live_print │
│ │
│ 35 │ def live_print(self, chunks: Generator[str, None, None]) -> str: │
│ 36 │ │ full_completion = "" │
│ 37 │ │ with Live(console=self.console) as live: │
│ ❱ 38 │ │ │ for chunk in chunks: │
│ 39 │ │ │ │ full_completion += chunk │
│ 40 │ │ │ │ markdown = Markdown(markup=full_completion, code_theme=self.theme) │
│ 41 │ │ │ │ live.update(markdown, refresh=True) │
│ │
│ ╭──────────────────────────────────────── locals ────────────────────────────────────────╮ │
│ │ chunks = <generator object Cache.call..wrapper at 0x7f09ae634280> │ │
│ │ full_completion = '' │ │
│ │ live = <rich.live.Live object at 0x7f09ae617250> │ │
│ │ self = <sgpt.printer.MarkdownPrinter object at 0x7f09afd18750> │ │
│ ╰────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /home/rhack/.local/share/pipx/venvs/shell-gpt/lib64/python3.11/site-packages/sgpt/cache.py:37 in │
│ wrapper │
│ │
│ 34 │ │ │ │ yield file.read_text() │
│ 35 │ │ │ │ return │
│ 36 │ │ │ result = "" │
│ ❱ 37 │ │ │ for i in func(*args, **kwargs): │
│ 38 │ │ │ │ result += i │
│ 39 │ │ │ │ yield i │
│ 40 │ │ │ if "@functioncall" not in result: │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ args = (<sgpt.handlers.default_handler.DefaultHandler object at 0x7f09afe361d0>,) │ │
│ │ file = PosixPath('/tmp/cache/5497462a3313794b1c88d0ce87b18e50') │ │
│ │ func = <function Handler.get_completion at 0x7f09ae65d760> │ │
│ │ key = '5497462a3313794b1c88d0ce87b18e50' │ │
│ │ kwargs = { │ │
│ │ │ 'model': 'ollama/gwen2:latest', │ │
│ │ │ 'temperature': 0.0, │ │
│ │ │ 'top_p': 1.0, │ │
│ │ │ 'messages': [ │ │
│ │ │ │ { │ │
│ │ │ │ │ 'role': 'system', │ │
│ │ │ │ │ 'content': 'You are ShellGPT\nYou are programming and system │ │
│ │ administration assistant.\nYou ar'+284 │ │
│ │ │ │ }, │ │
│ │ │ │ {'role': 'user', 'content': 'Who are you?'} │ │
│ │ │ ], │ │
│ │ │ 'functions': None │ │
│ │ } │ │
│ │ result = '' │ │
│ │ self = <sgpt.cache.Cache object at 0x7f09b1b3fe90> │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /home/rhack/.local/share/pipx/venvs/shell-gpt/lib64/python3.11/site-packages/sgpt/handlers/handl │
│ er.py:110 in get_completion │
│ │
│ 107 │ │ ) │
│ 108 │ │ │
│ 109 │ │ try: │
│ ❱ 110 │ │ │ for chunk in response: │
│ 111 │ │ │ │ delta = chunk.choices[0].delta │
│ 112 │ │ │ │ # LiteLLM uses dict instead of Pydantic object like OpenAI does. │
│ 113 │ │ │ │ function_call = ( │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ arguments = '' │ │
│ │ functions = None │ │
│ │ is_code_role = False │ │
│ │ is_dsc_shell_role = False │ │
│ │ is_shell_role = False │ │
│ │ messages = [ │ │
│ │ │ { │ │
│ │ │ │ 'role': 'system', │ │
│ │ │ │ 'content': 'You are ShellGPT\nYou are programming and system │ │
│ │ administration assistant.\nYou ar'+284 │ │
│ │ │ }, │ │
│ │ │ {'role': 'user', 'content': 'Who are you?'} │ │
│ │ ] │ │
│ │ model = 'ollama/gwen2:latest' │ │
│ │ name = '' │ │
│ │ response = <generator object ollama_completion_stream at 0x7f09ae6343a0> │ │
│ │ self = <sgpt.handlers.default_handler.DefaultHandler object at 0x7f09afe361d0> │ │
│ │ temperature = 0.0 │ │
│ │ top_p = 1.0 │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /home/rhack/.local/share/pipx/venvs/shell-gpt/lib64/python3.11/site-packages/litellm/llms/ollama │
│ .py:257 in ollama_completion_stream │
│ │
│ 254 │ │ │ for transformed_chunk in streamwrapper: │
│ 255 │ │ │ │ yield transformed_chunk │
│ 256 │ │ except Exception as e: │
│ ❱ 257 │ │ │ raise e │
│ 258 │
│ 259 │
│ 260 async def ollama_async_streaming(url, data, model_response, encoding, logging_obj): │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ data = { │ │
│ │ │ 'model': 'gwen2:latest', │ │
│ │ │ 'prompt': 'You are ShellGPT\nYou are programming and system administration │ │
│ │ assistant.\nYou ar'+296, │ │
│ │ │ 'options': {'temperature': 0.0, 'top_p': 1.0}, │ │
│ │ │ 'stream': True │ │
│ │ } │ │
│ │ logging_obj = <litellm.utils.Logging object at 0x7f09ae7fccd0> │ │
│ │ response = <Response [404 Not Found]> │ │
│ │ url = 'http://localhost:11434/api/generate' │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /home/rhack/.local/share/pipx/venvs/shell-gpt/lib64/python3.11/site-packages/litellm/llms/ollama │
│ .py:245 in ollama_completion_stream │
│ │
│ 242 │ │ try: │
│ 243 │ │ │ if response.status_code != 200: │
│ 244 │ │ │ │ raise OllamaError( │
│ ❱ 245 │ │ │ │ │ status_code=response.status_code, message=response.text │
│ 246 │ │ │ │ ) │
│ 247 │ │ │ │
│ 248 │ │ │ streamwrapper = litellm.CustomStreamWrapper( │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ data = { │ │
│ │ │ 'model': 'gwen2:latest', │ │
│ │ │ 'prompt': 'You are ShellGPT\nYou are programming and system administration │ │
│ │ assistant.\nYou ar'+296, │ │
│ │ │ 'options': {'temperature': 0.0, 'top_p': 1.0}, │ │
│ │ │ 'stream': True │ │
│ │ } │ │
│ │ logging_obj = <litellm.utils.Logging object at 0x7f09ae7fccd0> │ │
│ │ response = <Response [404 Not Found]> │ │
│ │ url = 'http://localhost:11434/api/generate' │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /home/rhack/.local/share/pipx/venvs/shell-gpt/lib64/python3.11/site-packages/httpx/_models.py:57 │
│ 6 in text │
│ │
│ 573 │ @Property │
│ 574 │ def text(self) -> str: │
│ 575 │ │ if not hasattr(self, "_text"): │
│ ❱ 576 │ │ │ content = self.content │
│ 577 │ │ │ if not content: │
│ 578 │ │ │ │ self._text = "" │
│ 579 │ │ │ else: │
│ │
│ ╭───────────── locals ──────────────╮ │
│ │ self = <Response [404 Not Found]> │ │
│ ╰───────────────────────────────────╯ │
│ │
│ /home/rhack/.local/share/pipx/venvs/shell-gpt/lib64/python3.11/site-packages/httpx/_models.py:57 │
│ 0 in content │
│ │
│ 567 │ @Property │
│ 568 │ def content(self) -> bytes: │
│ 569 │ │ if not hasattr(self, "_content"): │
│ ❱ 570 │ │ │ raise ResponseNotRead() │
│ 571 │ │ return self._content │
│ 572 │ │
│ 573 │ @Property │
│ │
│ ╭───────────── locals ──────────────╮ │
│ │ self = <Response [404 Not Found]> │ │
│ ╰───────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ResponseNotRead: Attempted to access streaming response content, without having called read().
`

[fabceolin]

This should fix the issue: BerriAI/litellm#5148