-
-
Notifications
You must be signed in to change notification settings - Fork 932
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Background tasks don't work with middleware that subclasses BaseHTTPMiddleware
#919
Comments
What versions of Python, Starlette, and Uvicorn are you using? I'm not able to recreate with Python 3.8.2, Starlette 0.13.3, and Uvicorn 0.11.5. |
Python 3.8.2, Uvicorn 0.11.4, Starlette 0.13.2. I'll try updating to see if that makes any difference. |
Nope, now on Starlette 0.13.3 and Uvicorn 0.11.5, but I'm still getting the same behaviour. What OS are you on? I'm using Arch Linux |
I have the same issue (Windows 10, Python 3.7, Starlette 0.13.4, Uvicorn 0.11.5). Is there any workaround? |
@ariefprabowo You can use |
Thanks for the hint @retnikt, will try that. |
Having the same issue. My background task is blocking and needs to run in the thread_pool. Everything works as expected without middleware, but adding middleware causes some requests to block. |
If I run this sample and I make a number of requests with ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/miniconda3/envs/starlette/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 385, in run_asgi
result = await app(self.scope, self.receive, self.send)
File "/miniconda3/envs/starlette/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
return await self.app(scope, receive, send)
File "./starlette/applications.py", line 111, in __call__
await self.middleware_stack(scope, receive, send)
File "./starlette/middleware/errors.py", line 181, in __call__
raise exc from None
File "./starlette/middleware/errors.py", line 159, in __call__
await self.app(scope, receive, _send)
File "./starlette/middleware/base.py", line 26, in __call__
await response(scope, receive, send)
File "./starlette/responses.py", line 228, in __call__
await run_until_first_complete(
File "./starlette/concurrency.py", line 18, in run_until_first_complete
[task.result() for task in done]
File "./starlette/concurrency.py", line 18, in <listcomp>
[task.result() for task in done]
File "./starlette/responses.py", line 225, in stream_response
await send({"type": "http.response.body", "body": b"", "more_body": False})
File "./starlette/middleware/errors.py", line 156, in _send
await send(message)
File "/miniconda3/envs/starlette/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 510, in send
self.transport.write(body)
File "uvloop/handles/stream.pyx", line 673, in uvloop.loop.UVStream.write
File "uvloop/handles/handle.pyx", line 159, in uvloop.loop.UVHandle._ensure_alive
RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False 0x7fdb3008d990>; the handler is closed Only five requests are successfully handled in this case. However, if I modify the class TransparentMiddleware(BaseHTTPMiddleware):
async def __call__(self, scope, receive, send) -> None:
if scope["type"] != "http":
await self.app(scope, receive, send)
return
await self.app(scope, receive, send) More interesting, if I add some logging calls, we can see when things are firing. First, in the exceptions case: class TransparentMiddleware(BaseHTTPMiddleware):
# async def __call__(self, scope, receive, send) -> None:
# if scope["type"] != "http":
# await self.app(scope, receive, send)
# return
# log.info("__calling__ the middlewares!")
# await self.app(scope, receive, send)
async def dispatch(self, request, call_next):
# simple middleware that does absolutely nothing
log.info("dispatching!")
response = await call_next(request)
return response
async def some_sleeper():
log.info("sleeping!")
await asyncio.sleep(10)
log.info("waking up now!")
@app.route("/")
async def test(_):
task = BackgroundTask(some_sleeper)
return JSONResponse("hello", background=task)
if __name__ == '__main__':
uvicorn.run(app, host="0.0.0.0", port=8000) Results in the following: ❯ python -m starlette_middleware_background
INFO: Started server process [53410]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO: dispatching!
INFO: dispatching!
INFO: sleeping!
INFO: sleeping!
INFO: dispatching!
INFO: dispatching!
INFO: dispatching!
INFO: sleeping!
INFO: sleeping!
INFO: sleeping!
INFO: 127.0.0.1:59302 - "GET / HTTP/1.1" 200 OK
INFO: 127.0.0.1:59303 - "GET / HTTP/1.1" 200 OK
INFO: 127.0.0.1:59304 - "GET / HTTP/1.1" 200 OK
INFO: 127.0.0.1:59305 - "GET / HTTP/1.1" 200 OK
INFO: 127.0.0.1:59306 - "GET / HTTP/1.1" 200 OK
INFO: waking up now!
INFO: waking up now!
INFO: waking up now!
INFO: waking up now!
INFO: waking up now!
ERROR: Exception in ASGI application
Traceback (most recent call last):
...[EXCEPTIONS: one per connection]... In the no-exceptions case things are interleaved in a manner closer to what I would have expected: INFO: __calling__ the middlewares!
INFO: 127.0.0.1:59324 - "GET / HTTP/1.1" 200 OK
INFO: sleeping!
INFO: __calling__ the middlewares!
INFO: 127.0.0.1:59321 - "GET / HTTP/1.1" 200 OK
INFO: sleeping!
INFO: __calling__ the middlewares!
INFO: 127.0.0.1:59325 - "GET / HTTP/1.1" 200 OK
INFO: sleeping!
INFO: waking up now!
INFO: __calling__ the middlewares!
INFO: 127.0.0.1:59325 - "GET / HTTP/1.1" 200 OK
INFO: sleeping!
INFO: __calling__ the middlewares!
INFO: 127.0.0.1:59325 - "GET / HTTP/1.1" 200 OK
INFO: sleeping!
...
INFO: waking up now!
INFO: waking up now!
INFO: waking up now!
INFO: waking up now!
INFO: waking up now!
INFO: waking up now!
... The difference in behavior comes down to this |
As written the call_next function starts a StreamingResponse, but dose not exit until the app being wrapped has completed. This includes background tasks. Some clients such as I'm not an expert in asyncio or ASGI, so I'm not sure if this fix has unintended consequences.... Changing: starlette/starlette/middleware/base.py Lines 49 to 56 in 1db8010
To:
allows the StreamingResponse to finish sending data to the client while the task is run in the background. |
The notable thing to me is that In my own projects, I've defined a handful of middleware classes, but I haven't implemented @retnikt can you try to implement your middleware with a I think at the very least that the order of execution is inconsistent between these two implementations. I'm also not sure why the |
@erewok yes, this issue doesn't happen with |
1 similar comment
@erewok yes, this issue doesn't happen with |
I have a similar issue in my current project I'm working on. We are using BackgroundTask for one long operation we have. Lately, I tried to add Middleware inherited from After doing so, I had BackgroundTask functionality broken and frontend waiting for it to finish before realising the request is done. I saw a reference to current issue in this MR. And it looked like it could solve this issue. Unfortunately, until it is merged I had to come up with the solution. And, basically, I tried using new version of So for anyone interested in this issue solved, this should happen after that MR gets reviewed and merged :) Huge thanks to @erewok as this issue appeared to be quite of a blocker. Because it'd be hard for me to use |
Somehow |
Hi all, I had a PR to fix this behavior (#1017 ) but it relied on forcing the response to remain unevaluated, which further confused those users who were relying on having a fully-evaluated response inside their middleware functions (#1022 ). In other words, there are contradictory goals here. After looking at this
Unfortunately, while the Again, these problems should be absent if you avoid subclassing Lastly, I think it's worth leaving this issue open so that other users who have the same problem can see it and see that it remains an open issue. |
@erewok thanks for your efforts! But how raw ASGI middleware may solve the issue? |
@dmig-alarstudios see @retnikt's comment above to that effect:
|
IMO this isn't really (just) an issue with
|
Yes, it is a broader issue with background tasks but that also implies that something should be done differently with how concurrency is being managed in the app. @retnikt see this comment for further discussion. |
Shame for me: never thought about using But I'm still wondering: running the same testcase under |
@erewok Thank you for your work on this. After your suggestion on using raw ASGI middleware, link to Basically, for anyone, who is trying to inject something into request and perform some cleanup on that injected instance after request has been finished, you can do this by placing your instance inside I only hope that wrapping the call of an |
BaseHTTPMiddleware
Finally found a workaround by exploiting from asyncio import get_event_loop
from concurrent.futures import ThreadPoolExecutor
from functools import partial
def run_sync_code(task, *args, **kwargs):
executor = ThreadPoolExecutor()
loop = get_event_loop()
loop.run_in_executor(executor, partial(task, *args, **kwargs)) Then in your favorite FastAPI endpoint: @app.post("/debug/background_task", tags=["debug"])
async def __debug_background_task():
def sleep_task(*args, **kwargs):
print("start sleep....", args, kwargs)
time.sleep(5)
print("end sleep....")
run_sync_code(sleep_task, 1, 2, a="a", b="b") |
I got this bug after updating fastapi 0.67.0 -> 0.70.0 This code worked on my project with few midllewares, but after migrating to newer version, "im alive" is not printing in my logs async def task():
logger.info("im a task")
await sleep(1)
logger.info("im alive")
@app.get("/")
async def root(background_tasks: BackgroundTasks):
background_tasks.add_task(task)
return {"status": "ok"} It happens even i use only one dummy middleware: class DM(BaseHTTPMiddleware):
async def dispatch(self, request, call_next):
return await call_next(request) After removing all middlewares this code works well as expected I tried to reproduce this on new pure project, but i could not. |
If anyone finds this and is trying to add headers to responses this is how i ended up doing it using the information from the above discussions. from starlette.types import ASGIApp, Message, Receive, Scope, Send
from starlette.datastructures import MutableHeaders
# Custom class required to add headers in-order to not block background tasks
# https://github.com/encode/starlette/issues/919
class SecurityHeadersMiddleware:
def __init__(
self,
app: ASGIApp,
) -> None:
self.app = app
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
async def send_wrapper(message: Message) -> None:
if message["type"] == "http.response.start":
headers = MutableHeaders(scope=message)
headers[
"Strict-Transport-Security"
] = "max-age=63072000; includeSubDomains; preload"
await send(message)
await self.app(scope, receive, send_wrapper) |
A related issue is that BackgroundTasks will get cancelled if you have a middleware and the caller sets the header # Python 3.8.13
import asyncio
import fastapi # fastapi==0.75.2
import time
import uvicorn # uvicorn==0.17.6
import logging
logging.basicConfig(format='%(asctime)s.%(msecs)03dZ [%(levelname)s] %(message)s')
log = logging.getLogger()
app = fastapi.FastAPI()
@app.middleware("http")
async def empty_middleware(request: fastapi.Request, call_next):
response = await call_next(request)
return response
@app.get("/")
async def the_route(background_tasks: fastapi.BackgroundTasks):
background_tasks.add_task(a_little_work)
# background_tasks.add_task(shielded_work) <-- this one completes
return {'status': 'ok'}
async def a_little_work(prefix='unshielded'):
start = time.monotonic()
log.warning(f"{prefix} background task starts!")
await asyncio.sleep(0.5)
end = time.monotonic()
# this statement never gets printed if the Connection header is set to close:
log.warning(f"{prefix} background task done, took {end-start}s")
async def shielded_work():
import asyncio
await asyncio.shield(a_little_work('shielded'))
if __name__ == '__main__':
uvicorn.run(app, host="0.0.0.0", port=8000) |
I discovered same issue when trying to upgrade my app from fastapi 0.68 to 0.70. Interesting fact is that background tasks work fine on my workstation Ubuntu 20.04 but fail on server with CentOS 7 (same code, python 3.9 & modules versions). It looks that best solution is to deprecate BaseHTTPMiddleware like mentioned in #1678 |
I propose that this issue be closed - as far as I can tell, the issues reported here are stale, invalid, or duplicates of other issues:
Here is how I think we should proceed on this issue:
|
The GZipMiddleware issue was related to the BaseHTTPMiddleware, but fixed already. 🙏 The other, as you said. 🙌 |
* The usage of BackgroundTasks in the new execute_action code was found to be causing blockages of new requests being picked up by the event loop. I think this maybe a reccurance of encode/starlette#919 which they claim is resolved, but still doesn't appear to be working for us. It may be due to one of our middlewares behaving in a particular way that causes this, but the fix applied here appears to be the simplest fix for now. * In this instance we have replaced the usage of background_tasks.add_task with asyncio.create_task that will run the task in the loop without worrying about its result.
* The usage of BackgroundTasks in the new execute_action code was found to be causing blockages of new requests being picked up by the event loop. I think this maybe a reccurance of encode/starlette#919 which they claim is resolved, but still doesn't appear to be working for us. It may be due to one of our middlewares behaving in a particular way that causes this, but the fix applied here appears to be the simplest fix for now. * In this instance we have replaced the usage of background_tasks.add_task with asyncio.create_task that will run the task in the loop without worrying about its result.
* Fix: DOS-625 Address action execution blocking new requests * The usage of BackgroundTasks in the new execute_action code was found to be causing blockages of new requests being picked up by the event loop. I think this maybe a reccurance of encode/starlette#919 which they claim is resolved, but still doesn't appear to be working for us. It may be due to one of our middlewares behaving in a particular way that causes this, but the fix applied here appears to be the simplest fix for now. * In this instance we have replaced the usage of background_tasks.add_task with asyncio.create_task that will run the task in the loop without worrying about its result. * Fix linting
I met the same issue, is this fixed or should we report an issue on this? |
When using background tasks with middleware, requests are not processed until the background task has finished.
app.add_middleware(TransparentMiddleware)
and re-runThe same behaviour occurs with
asyncio.sleep
(async) andtime.sleep
(sync, run in threadpool)The text was updated successfully, but these errors were encountered: