-
Notifications
You must be signed in to change notification settings - Fork 14.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ability to generate temporary downloadable link for task logs(stored on cloud storage) on UI #44753
Comments
I have also espcially seen that the webserver gets OOM even if you download because the FileTaskHandler tries to sort+merge different log sources. One important thing is not only the download but also the FileTaskHandler must 1:1 stream the logs from the backend, else it will go OOM as well. This included when I took a look last time... why actually log sorting and merging is implemented in general I doubt the usefulness because I also had situations where log sorting by timestamp gave "strange" results. In my view logs should not be modified/sorted/merged when served from webserver. This would also remove the OOM problem in general... whereas I have seen situations as well where the browser/client "dies" in rendering. So a limit in logs provided to browser might be a important feature as well (but please not like Github is doing it) |
Mostly agree with @jscheffl -> but I still think merging logs might be useful in some cases, though the "naive" version it is done now should be either limited to certain log size that should be able to fit in memory or fixed to support arbitrary log size. Loading whole log to memory is generally bad idea (but OK if we can confirm they will fit in memory). There are some algorithms that could be used to do it "well" even when the logs are huge, but they require much more sophisticated behaviour and local file storage and likely are not suitable to run in an "API" call. So I am not sure if it at all worth doing it (airflow is NOT a sophisticated logging solution) - but if we have this "download full log" (and even there we could download zipped logs from several sources without merging them) - that could be a useful counterpart for merging for big files. There are two options how to do it, I think:
Generally, yes, I think we should implement it (and prevent the OOM from happening). |
Hi, after tracing down related GitHub issues and attempting to reproduce the OOM issue, I have some questions and ideas about related improvements. Related Issues and Context
Ideas for Related Improvements
Though I couldn't reliably reproduce OOM on the webserver, the browser tab crashing due to large logs remains an issue. QuestionHow can I reliably reproduce the OOM issue when fetching large task logs?
Any tips on how to consistently reproduce the issue? Or is there a specific environment setup recommended for this? |
@jason810496 , I think you need to put resource limits on each airflow component. On managed services, the webserver is usually limited, On Astro its about 1 vcpu and 2GiB memory. |
Thanks, I will try it with docker compose ! |
Description
When a task generates many logs, the web server usually gets OOM Killed while trying to render the task logs. For this situation, we can give the user a downloadable link to the file on UI. If the task logs are located on cloud storage, we can generate the following:
Use case/motivation
To render large task log files
Related issues
#33625
Are you willing to submit a PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: