AIP-38 Refactor TaskInstance Log UI with Streaming and Virtualized Rendering

[jason810496]

Why

The browser can sometimes crash when rendering large logs on the frontend.

After #46827 was merged, the get_log endpoint supports the application/x-ndjson response type, which allows streaming logs in newline-delimited JSON format.
( Once #49470 is merged, the API will fully support streaming responses. )

However, the current frontend still uses the default application/json header instead of application/x-ndjson. When rendering a large JSON response all at once, the browser may crash due to memory overload.

Relevant code:
https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/api_fastapi/core_api/routes/public/log.py#L148-L154

What

This refactor should include:

1. Set the Accept header to application/x-ndjson in the frontend request.
2. Use react-window or react-virtualized to efficiently render large log entries.

These changes should be applied to the TaskInstance Log page:
airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[jason810496]

cc @bbovenzi, @pierrejeambrun

Any suggestions or concerns regarding this issue? Is there anything contributors should be aware of for this refactor that I may have missed? I'm not deeply familiar with the current frontend architecture.

(For context: I previously attempted to refactor the log page with react-window, but couldn't fully get it working. I'm opening this issue to track the idea and encourage contributions.)

[guan404ming]

IMO I think @tanstack/react-virtual might be a more modern option and also closer to our ui stack since we’re already using @tanstack/react-query. It also seems like Chakra UI doesn't plan to implement this feature at now, so I’d prefer to go with the Tanstack option if we need it. I'm also open to any discussion and happy to help with anything!

[bbovenzi]

This would be a great feature

+1 on tanstack/react-virtual
I've been pretty happy with all their tools we've used so far. react-window hasn't been updated in years.

[jason810496]

Thanks @guan404ming and @bbovenzi for the feedback, it's definitely more aligned with our current tech stack!

@guan404ming, would you be interested in working on this issue?
If so, I’d be happy to assign it to you, thanks!

[guan404ming]

Sure, I’m really interested to work on this. Thanks! Btw, I enjoyed with their tools too. Happy to add it in!🚀

[jason810496]

Thanks @guan404ming for helping , just assigned.

[pierrejeambrun]

That would be great.

+1 for tan stack too.