Skip to content

Conversation

@choo121600
Copy link
Member

@choo121600 choo121600 commented Jul 12, 2025

Summary

Adds visual indicators in the Grid view to help users identify Dag version changes and detect mixed-version TaskInstances within a single DagRun.

Problem

  • After updating a Dag file, it's difficult to know from which DagRun the new version is applied.
  • Some TaskInstances within the same DagRun may use different Dag versions, but this is not visible in the UI.

Solution

This PR introduces visual cues in the Grid view to address these pain points:

  • Dag Version Indicator: Vertical/horizontal orange line with tooltip showing version number
  • Bundle Version Indicator: Git commit icon with tooltip showing bundle version
  • Toggle Options: Users can choose to show None, Dag Version Indicator only, Bundle Version Indicator only, or All indicators

Change

Backend

  • Added bundle_version and dag_version_number fields to GridRunsResponse
  • Modified grid query to JOIN with DagVersion table for efficient single-query data fetch
  • Added dag_version_number to LightGridTaskInstanceSummary for mixed version detection

Frontend

  • New VersionIndicator.tsx component(BundleVersionIndicator, DagVersionIndicator)
  • New useGridRunsWithVersionFlags hook to calculate version change flags
  • Updated Bar.tsx, Grid.tsx, TaskInstancesColumn.tsx to display indicators
  • Added version indicator toggle in PanelButtons.tsx

Screenshots

Version Change

LocalDagBundle

image

Dag Version Indicator Tooltip(vertical)
image

Dag Version Indicator Tooltip(horizontal)
image

GitDagBundle

image

Dag Version Indicator Tooltip
image

Bundle Version Indicator Tooltip
image

Dark Mode

image image

Toggle Option(with Legend)

Users can choose 'Show All', 'Show Dag Version', 'Show Bundle Version', 'Hide All'

image

Show All

image

Show Bundle Version

image

Show Dag Version

image

Hide All

image

Indicator Scenarios

Scenario 1: Normal

  • All tasks run with the same version
  • No indicator (default state)
image

Scenario 2: Version Change

  • Entire run with new version
  • Left indicator + version number tooltip
image image

Scenario 3: Mixed Version

  • Multiple versions within single run
  • Left, Row indicator + version number tooltip
image image image

Scenario 4: Bundle Version & DagVersion Change

image image image

Scenario 5: Only Bundle Change (Dynamic Dag)

image image

closes: #52286


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Jul 12, 2025
@choo121600 choo121600 force-pushed the feature/version-indicator branch 2 times, most recently from 09ecc2b to bd021e8 Compare July 12, 2025 07:04
Copy link
Contributor

@bugraoz93 bugraoz93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I think this is bringing back the deprecated and removed monolith endpoint with its methods. I think we should integrate the fix into only new structure
cc: @dstandish @pierrejeambrun

@choo121600
Copy link
Member Author

@bugraoz93 Thanks for your quick feedback! I saw something strange while fixing the conflict,
So I started checking the main branch — your comment helped me confirm it.

I think my endpoints follow the old style. I’ll change them to use the new structure.

@bugraoz93
Copy link
Contributor

@bugraoz93 Thanks for your quick feedback! I saw something strange while fixing the conflict,
So I started checking the main branch — your comment helped me confirm it.

I think my endpoints follow the old style. I’ll change them to use the new structure.

Amazing, thanks a lot!

@choo121600
Copy link
Member Author

choo121600 commented Jul 13, 2025

I’ve updated the tests and removed the old structure,
but I'm testing whether the feature I added works as before🥲
After verifying the tests, I will leave a comment!

@choo121600
Copy link
Member Author

choo121600 commented Jul 13, 2025

I’ve tested the scenarios I had in mind, and everything looks good so far 🙌
Reviews are welcome from this point.

@choo121600 choo121600 requested a review from bugraoz93 July 13, 2025 09:10
Copy link
Contributor

@bugraoz93 bugraoz93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes! I believe this PR still contains old deprecated code :)

Copy link
Member

@jason810496 jason810496 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for the PR!

@choo121600
Copy link
Member Author

I thought I deleted the old version, but it still there.
Right now, my new code and the old APIs are mixed together, and it looks like the old version is included when I commit 🤔
This might be a problem with my local setup, so I will check and fix it this week.

@ephraimbuddy
Copy link
Contributor

If we want to visualize version change, in my opinion, we should do it for bundle version and not dag version that changes everytime especially for dynamic dags cc @jedcunningham

@bbovenzi
Copy link
Contributor

If we want to visualize version change, in my opinion, we should do it for bundle version and not dag version that changes everytime especially for dynamic dags cc @jedcunningham

In that case, we might need to use an icon and then show the bundle version in a tooltip. Maybe something like <FiGitCommit />

@pierrejeambrun
Copy link
Member

pierrejeambrun commented Jul 16, 2025

If we want to visualize version change, in my opinion, we should do it for bundle version and not dag version that changes everytime especially for dynamic dags cc @jedcunningham

Maybe we need both with different colors, dag version change (to task level) + bundle version change (dag run level only).

If we were to keep only one, I would be in favor of keeping the DagVersion because it is consistant with the Graph(Graph is showing structures of specific dag versions), And because I think source code is less important than what was actually executed.

Copy link
Member

@pierrejeambrun pierrejeambrun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the code looks very complicated for what we are trying to achieve. (And also there are some unused stuff I believe from the legacy grid)

The backend need to serialize the dag_run.versions in the GridRunsResponse, same for tasks.

And in the UI, we just need a simple logic to check if dag_run.version of Nth run is equal to dag_run.version of (N-1)th run.

(Same logic goes for tasks, and yes if DagRun.dag_versions.lenght === 1 no need to go try the task level logic)

Maybe I missed something though.

@pierrejeambrun
Copy link
Member

In that case, we might need to use an icon and then show the bundle version in a tooltip. Maybe something like

I'm afraid that we can potentially have as many tooltips as TI in the grid. Which would most likely bring back tooltip performance issue we observed in the past.

@choo121600
Copy link
Member Author

It's hard to keep track of the context here.

Can you please 'resolve' comments that you have addressed so we know what effort remains on the PR before we need to do another round of reviews. (There are a lot of things unresolved at the moment, but I guess most of them you addressed.)

Thanks.

The reason I haven’t resolved the comments yet is that the actions were failing during the freeze period.
Once the freeze is over and the actions turn green, I’ll make sure to resolve and leave comments 😉
(I’ll also update the PR description along with the comments, since there were significant code changes while adding the DagBundle feature and the show/hide functionality.)

@choo121600 choo121600 changed the title Feat: Visualize DAG version changes and mixed versions in Grid view 🚧 Feat: Visualize DAG version changes and mixed versions in Grid view Sep 18, 2025
@choo121600 choo121600 changed the title 🚧 Feat: Visualize DAG version changes and mixed versions in Grid view Feat: Visualize DAG version changes and mixed versions in Grid view Sep 27, 2025
Copy link
Member

@pierrejeambrun pierrejeambrun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see some activity on the PR, let us know when this is ready for a second round of reviews. (need rebase and a few comments are still worked on it appears).

Thank you for your work

@bbovenzi
Copy link
Contributor

Thanks for taking this on!

We should make sure that the display works when the version count is is in the double or even triple digits. Perhaps we should fallback to just use an icon?

@choo121600
Copy link
Member Author

We should make sure that the display works when the version count is is in the double or even triple digits. Perhaps we should fallback to just use an icon?

Thank you! you're right it displays fine for up to two digits, but with three digits the horizontal indicator starts overlapping other elements.
We could use the FiRefreshCw icon as the default and show the version number on hover. What do you think?

Screen.Recording.2025-10-17.at.12.48.00.AM.mov
image

@bbovenzi
Copy link
Contributor

bbovenzi commented Oct 16, 2025

I think the git line+circle icon you had before is better. We use the CwRefresh for clearing dag runs

Also, we probably will want to add a checkbox in Options to turn off the version indicators.

@choo121600
Copy link
Member Author

choo121600 commented Oct 22, 2025

Also, we probably will want to add a checkbox in Options to turn off the version indicators.

Would it be better to have it as a checkbox instead of a "Hide All" option in the dropdown menu?
image

Copy link
Member

@pierrejeambrun pierrejeambrun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that dropdown seems nice to me to be able to hide / display version markers.

Do you have an updated screenshot of what it looks like now with both bundle change and version change on the same grid. (most likely what happens if at the same spot we have both a version change and bundle change)

@pierrejeambrun
Copy link
Member

pierrejeambrun commented Oct 24, 2025

Vertical pin should probably be a few pixels taller, so it doesn't overlap with the 'dag run' bar:

Screenshot 2025-10-24 at 16 35 36

@choo121600
Copy link
Member Author

DO you have an updated screenshot of what it looks like now?

I’ll make sure to update all the feature-related screenshots by tomorrow ;)
Once it’s done, I’ll mention you and Bovenzi.

@choo121600
Copy link
Member Author

@bbovenzi @pierrejeambrun I’ve updated all the screenshots in the description ;)

Copy link
Contributor

@bbovenzi bbovenzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should add a legend to make it easier for users to learn dag versions vs bundle version changes.

transition="all 0.2s"
width={2}
/>
<Box
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we make a custom tooltip instead of the one from chakra? I know there were performance issues with it before, but now it seems much better.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used a custom tooltip to prevent it from covering other task status boxes in the Task Instance grid view, allowing it to naturally overlap with the indicator.
However, this approach made the CSS more complex and harder to maintain, so I’ve been reconsidering it.

As you suggested, I think it would be better to use the Chakra tooltip provided in the component for better consistency and simplicity, and I’ll update it accordingly.

image image

@pierrejeambrun pierrejeambrun added the type:new-feature Changelog: New Features label Nov 4, 2025
@choo121600
Copy link
Member Author

choo121600 commented Nov 6, 2025

@bbovenzi I’ve been thinking about the legend, and currently, the horizontal indicator for Dag version changes and the indicator for bundle version changes use the same icon.
So, distinguishing them by icon alone seems difficult. Instead, I modified the horizontal Dag version change indicator to use a consistent style with the vertical one(a combination of a line and a circle.)

Here’s an example of what it looks like. What do you think?

image image

@choo121600 choo121600 requested a review from bbovenzi November 14, 2025 06:45
@choo121600
Copy link
Member Author

@bbovenzi I’ve been thinking about the legend, and currently, the horizontal indicator for Dag version changes and the indicator for bundle version changes use the same icon. So, distinguishing them by icon alone seems difficult. Instead, I modified the horizontal Dag version change indicator to use a consistent style with the vertical one(a combination of a line and a circle.)

Here’s an example of what it looks like. What do you think?

@bbovenzi @pierrejeambrun This part hasn’t been implemented yet. I’d like to hear your ideas regarding the legend.

@choo121600 choo121600 changed the title Feat: Visualize DAG version changes and mixed versions in Grid view Feat: Add version change indicators for Dag and bundle versions in Grid view Nov 14, 2025
Copy link
Member

@pierrejeambrun pierrejeambrun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, looking better and better.

A few suggestions to try to make things simpler and improve UX.

run_after: datetime
state: DagRunState | None
run_type: DagRunType
dag_version_number: int | None = None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you need to pass this down, because the grid is calling get_dag_details and the latest dag_version is already there in the payload.

So you might get it in the front-end from there. same for the bundle name and bundle version.

run_type: DagRunType
dag_version_number: int | None = None
bundle_version: str | None = None
has_mixed_versions: bool = False
Copy link
Member

@pierrejeambrun pierrejeambrun Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this. You don't need to compute it. We have the get_dag_run query in the context of the grid, that returns dag_versions already, if the list is bigger than 1 we have mixed version. (only thing is that this list consider TaskInstanceHistory too, which might or might not be appropriate depending on how we want to visualize things)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing that will also simplify the backend code.

Copy link
Member Author

@choo121600 choo121600 Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand the suggestion to reuse the existing APIs, and I explored the frontend-only approach, However, I have some concerns.

One main issue is the Grid shows multiple Dag runs at the same time, and if we try to fetch the version information for each run individually using get_dag_run, the number of API calls quickly adds up.

For example, if there are 10 runs displayed, we’d end up making 11 API calls in total—one for get_dag_details and one for each of the 10 runs.

Could you let me know if what I found out is incorrect?

Copy link
Member

@pierrejeambrun pierrejeambrun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to mark addressed comments as resolved.

Let me know when this needs another round of review.

@choo121600
Copy link
Member Author

Except for this part #53216 (comment)
I’ve marked the addressed comments as resolved and updated all the screenshots.
@pierrejeambrun Could you review it once more?

@bbovenzi
Copy link
Contributor

bbovenzi commented Jan 8, 2026

We'll need to rebase this after virtualizing the grid's vertical scroll.

@choo121600 choo121600 force-pushed the feature/version-indicator branch from fad0cf6 to f48e7ec Compare January 10, 2026 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. type:new-feature Changelog: New Features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UI - Grid indicator for version change

8 participants