Skip to content

use 1 worker for log server #54510

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mattp- wants to merge 1 commit into from
Closed

use 1 worker for log server #54510

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions airflow-core/src/airflow/utils/serve_logs/core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,11 @@ def serve_logs(port=None):
logger.info("Starting log server on %s", serve_log_uri)

# Use uvicorn directly for ASGI applications
uvicorn.run("airflow.utils.serve_logs.log_server:app", host=host, port=port, workers=2, log_level="info")
# Note: if we want to use more than 1 workers, we **can't** use the instance of FastAPI directly
# This is way we split the instantiation of log server to a separate module
# This is why we split the instantiation of log server to a separate module
#
# https://github.com/encode/uvicorn/blob/374bb6764e8d7f34abab0746857db5e3d68ecfdd/docs/deployment/index.md?plain=1#L50-L63
uvicorn.run("airflow.utils.serve_logs.log_server:app", host=host, port=port, workers=1, log_level="info")
Copy link
Member

@jason810496 jason810496 Aug 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe which version of uvicorn will cause the problem you described above?

Copy link
Author

@mattp- mattp- Aug 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah gotcha. i put a simple reproducer in Kludex/uvicorn#2679 which is just straight uvicorn
running airflow standalone on main right now also repros for me on rhel 9 with uvicorn 0.35.0

Copy link
Member

@jason810496 jason810496 Aug 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

running airflow standalone on main

Thanks for the reply.
Will it raise the same error for just released Airflow 3.0.4 standalone as well?

Copy link
Author

@mattp- mattp- Aug 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just checked, no problem on 3.0.4

Copy link
Member

@jason810496 jason810496 Aug 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, thanks for the confirmation.
It would be great to raise another issue to track this behavior.

This means there might not be uvicorn issue but other changes in Airflow that cause this error.
Since there is not problem on 3.0.4 but there is error on latest main branch.

Copy link
Author

@mattp- mattp- Aug 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

mphillips81@mmp-pw-dev /tmp/af3/.venv/lib/python3.10/site-packages/airflow % grep -rsi 'uvicorn'
cli/commands/api_server_command.py:import uvicorn
cli/commands/api_server_command.py:        log.warning("Running in dev mode, ignoring uvicorn args")
cli/commands/api_server_command.py:                Running the uvicorn with:
cli/commands/api_server_command.py:        uvicorn.run(
Binary file cli/commands/__pycache__/api_server_command.cpython-310.pyc matches

i think uvicorn is only being hit from api-server in the released 3.0.4 - the problem in main is the uvicorn.run is happening wrapped by a multiprocessing.Process when log-server is spawned - those changes aren't in 3.0.4



if __name__ == "__main__":
Expand Down
Loading