Support streaming with Vercel python runtime

[SilenNaihin]

Summary

I'm using vercel dev to start my project with NextJS 13. I'm using the Python runtime on Vercel and try to stream to the frontend. My frontent code is implemented correctly, but for the life of me I cannot get the streaming to send anything. Would love some help!

class handler(BaseHTTPRequestHandler):
    def do_POST(self):
        try:
            # Parse the request body JSON
            content_length = int(self.headers["Content-Length"])
            post_data = self.rfile.read(content_length)
            data = json.loads(post_data)
        except Exception as e:
            self.send_error(400, f"Failed to parse JSON: {e}")
            return

        try:
            self.send_response(200)
            self.send_header("Content-type", "text/plain; charset=utf-8")
            self.send_header("Cache-Control", "no-cache")
            self.send_header("Connection", "keep-alive")
            self.end_headers()

            print("print1")

            test_data = ["chunk1", "chunk2", "chunk3", "chunk4"]

            for text in test_data:
                print(text)
                self.wfile.write(f"{text}".encode("utf-8"))
                self.wfile.flush()

            print("print2")
        except Exception as e:
            self.send_error(500, f"Failed to create response: {str(e)}")
            return

output in console

print1
chunk1
chunk2
chunk3
chunk4
print2

I'm 90% problem isn't in the client but just in case

const response = await fetch(`/api/respond`, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        transcript,
        type,
      }),
    }).catch((error) => {
      console.error('Fetch error:', error);
    });

    if (!response?.ok) {
      throw new Error(`status: ${response?.status}`);
    }

    const data = response.body;

    console.log(data);

Example

No response

Steps to Reproduce

vercel dev, Python http runtime, NextJS-13

test_data = ["chunk1", "chunk2", "chunk3", "chunk4"]

       for text in test_data:
            print(text)
            self.wfile.write(f"{text}".encode("utf-8"))
            self.wfile.flush() 

and receive the data in the front end as you normally would. I showed above a minimal example

[leerob]

Hey there, the Python runtime does not currently supporting streaming responses on Vercel. I'll let the team know about this feature request!

[sumersao]

Any update on this?! We have a system that uses python runtime (fastapi) and nextjs on the FE. After a while of debugging code, we're learning this isn't supported on vercel which is quite sad. We think it's core to what we're building -- should we move off the platform or is a fix coming soon?

[melovy]

Hey @leerob,

when can we anticipate the release of the Python runtime's streaming responses on Vercel? We're facing a similar issue with Next.JS and Python FastAPI and considering alternatives.

Any rough estimates or potential timelines would be much appreciated. Thanks!

[leerob]

Hey there, could you share more about what you're trying to build?

[melovy]

Hey @leerob,

We are working on building a QA chatbot app using Next.JS as the frontend and FastAPI on the backend. The backend communicates with a vector database for search functionality and utilizes OpenAI for text generation. One of our main concerns is reducing user wait time for OpenAI responses.

[melovy]

Hey @leerob

Any rough estimates on the release of streaming support with Python runtime would be greatly appreciated. It's essential for our project, and we might need to consider options if it's not coming soon.

Thank you!

[dreamhunter2333]

I also encountered this problem and can't return text/stream. Hope vercel can support it. Thanks.

[wkevwang]

Would also love support from Vercel on this

[james-julius]

Same! This would be absolutely killer with npm i ai

[Yukixieyuya821]

I also encountered the same problem in golang. I hope to support Flush's API and support sending buffered data to the client.

[untilhamza]

Feb 1, 2024 - still waiting for the feature.
Most AI + LLM packages have better support in Python than javascript so we need to use fastapi on the backend and vercel ai sdk on the frontend.

[ericbellet]

It would be super useful to have the streaming feature for FastAPI on Vercel. I got this error when I tried to use this endpoint:

@app.post("/test")
async def test(message: Message, auth: str = Depends(get_auth), model="gpt-3.5-turbo-0613", temperature=0):
    generator = send_message(user_question=message.user_question, prompt=prompt, content=message.chat_history, model=model, temperature=temperature)
    return StreamingResponse(generator, media_type="text/event-stream")

[ERROR] ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Traceback (most recent call last):
  File "/var/task/vc__handler__python.py", line 305, in vc_handler
    response = asgi_cycle(__vc_module.app, body)
  File "/var/task/vc__handler__python.py", line 208, in call
    loop.run_until_complete(asgi_task)
  File "/var/lang/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/var/task/fastapi/applications.py", line 1054, in call
    await super().call(scope, receive, send)
  File "/var/task/starlette/applications.py", line 123, in call
    await self.middleware_stack(scope, receive, send)
  File "/var/task/starlette/middleware/errors.py", line 186, in call
    raise exc
  File "/var/task/starlette/middleware/errors.py", line 164, in call
    await self.app(scope, receive, _send)
  File "/var/task/starlette/middleware/cors.py", line 83, in call
    await self.app(scope, receive, send)
  File "/var/task/starlette/middleware/exceptions.py", line 62, in call
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/var/task/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/var/task/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/var/task/starlette/routing.py", line 762, in call
    await self.middleware_stack(scope, receive, send)
  File "/var/task/starlette/routing.py", line 782, in app
    await route.handle(scope, receive, send)
  File "/var/task/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/var/task/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/var/task/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/var/task/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/var/task/starlette/routing.py", line 75, in app
    await response(scope, receive, send)
  File "/var/task/starlette/responses.py", line 261, in call
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/var/task/anyio/_backends/_asyncio.py", line 678, in aexit
    raise BaseExceptionGroup(

[thijmenjk]

+1

The Vercel AI SDK is neat, but the server implementation does not yet support the real-time streaming of StreamData: vercel/ai#853. I implemented a Python version that does support the streaming of StreamData, but unfortunately, this only works locally due to streaming not being supported by the Python runtime (implementation can be found here for those interested). It would be really cool if Vercel supported this!

[yachty66]

+1