Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graph is empty after update to 2.9.0 on some DAGs #39038

Closed
2 tasks done
sicarul opened this issue Apr 15, 2024 · 5 comments · Fixed by #39069
Closed
2 tasks done

Graph is empty after update to 2.9.0 on some DAGs #39038

sicarul opened this issue Apr 15, 2024 · 5 comments · Fixed by #39069
Assignees
Labels
affected_version:2.9 area:core area:UI Related to UI/UX. For Frontend Developers. kind:bug This is a clearly a bug

Comments

@sicarul
Copy link
Contributor

sicarul commented Apr 15, 2024

Apache Airflow version

2.9.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Since upgrading from 2.8.3 to 2.9.0, some DAGs are now empty

What you think should happen instead?

Graphs should show in the UI

How to reproduce

Some times when i open the graph it shows for a fraction of a second and then disappears, and other times i see it completely empty; I'm not sure how to reproduce it, but when i open the dev console i see this, which mentions a shape and then names a model that is not part of this dag, which seems even weirder. The dag has a task group if that's relevant:

Error: org.eclipse.elk.graph.json.JsonImportException: Referenced shape does not exist: dbt_models.stg_mysql_single_user_organization
image

In case it's relevant, this dag is generated from a single python file that generates dozens of dags from a yaml, not sure if they can somehow be.

An even weirder fact about all this, is that running the same DAG locally using docker-compose, i see the graph just fine, so it's only happening on the hosted version that has exactly the same DAGs and Docker (Maybe some old dag run in the database that had something wrong?)

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon[aiobotocore]==8.20.0
apache-airflow-providers-common-sql==1.12.0
apache-airflow-providers-http==4.10.1
apache-airflow-providers-snowflake==5.4.0
apache-airflow-providers-sqlite==3.7.1
apache-airflow-providers-redis==3.6.1
apache-airflow-providers-celery==3.6.2

Deployment

Other Docker-based deployment

Deployment details

Running on Docker images on AWS ECS

Anything else?

I am willing to submit a PR but i have no idea where/why this happens, and zero knowledge of Airflow's codebase, if someone can at least guide me as to where could i fix this issue

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@sicarul sicarul added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Apr 15, 2024
Copy link

boring-cyborg bot commented Apr 15, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@sicarul
Copy link
Contributor Author

sicarul commented Apr 15, 2024

If it helps to decipher this problem, i found another case:
image

And in this one, i was able to find old tasks with that name (That task does not exist any more) - However even after deleting the deprecated task instances using the UI, the same error persists in the console, my impression is it seems some "phantom records" are being picked up by the graph library?.

Upon further investigation, it seems this might be caused by the new datasets viz in the graph? Because i see that we get this record that is not present in the datasets UI, because it's from a deprecated table:

    {
      "consuming_dags": [],
      "created_at": "2023-10-16T18:24:02.812802+00:00",
      "extra": {},
      "id": 1081,
      "producing_tasks": [
        {
          "created_at": "2023-10-16T18:24:02.827186+00:00",
          "dag_id": "dbt_pipeline_agg_alb_logs",
          "task_id": "dbt_models.raw_alb_logs_api_requests_categorized",
          "updated_at": "2023-10-16T18:24:02.827200+00:00"
        }
      ],
      "updated_at": "2023-10-16T23:11:20.408211+00:00",
      "uri": "raw_alb_logs_api_requests_categorized"
    }

(This is from /api/v1/datasets?dag_ids=dbt_pipeline_agg_alb_logs)

Is there a way to clean it up? Or to make Airflow ignore this record instead of breaking the graph?

@eladkal eladkal added area:UI Related to UI/UX. For Frontend Developers. affected_version:2.9 and removed needs-triage label for new issues that we didn't triage yet labels Apr 15, 2024
@eladkal
Copy link
Contributor

eladkal commented Apr 15, 2024

cc @bbovenzi

@bbovenzi bbovenzi self-assigned this Apr 16, 2024
@bbovenzi
Copy link
Contributor

So this only happens to DAGs with Datasets? And it sounds like it happens when we retrieve datasets but either the dataset or task no longer exists so we're pointing to a nonexistent node. Sounds straightforward enough to add a check for that.

@sicarul
Copy link
Contributor Author

sicarul commented Apr 16, 2024

@bbovenzi Yes, that seems to be the case. My guess is this didn't happen before because Airflow was not drawing datasets in the graph

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affected_version:2.9 area:core area:UI Related to UI/UX. For Frontend Developers. kind:bug This is a clearly a bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants