Dag Import Error Hidden for existing dags leading to nondescriptive UI errors

[collinmcnulty]

Python version: 3.7.10

Apache Airflow version: 2.1.1

Apache Airflow Provider versions (please include all providers that are relevant to your bug): None

Kubernetes version (if you are using kubernetes): version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.19-gke.1701", GitCommit:"d7cecefb99b58e8968f59b59d76448eb1e6ea403", GitTreeState:"clean", BuildDate:"2021-06-23T21:51:59Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: GCP
• OS (e.g. from /etc/os-release): Debian

What happened:

We upgraded our airflow from 1.10.14 to 2.1.1 (not sure if the recent upgrade is relevant). Several dags already existed in the metadata db at the time of the upgrade. After the upgrade, some of the dags would give the following error messages if you clicked on Code view or Graph view.

Graph view

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.7/site-packages/airflow/www/auth.py", line 34, in decorated
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/www/decorators.py", line 97, in view_func
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/www/decorators.py", line 60, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 2231, in graph
    nodes = task_group_to_dict(dag.task_group)
  File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 233, in task_group_to_dict
    task_group_to_dict(child) for child in sorted(task_group.children.values(), key=lambda t: t.label)
AttributeError: 'NoneType' object has no attribute 'children'

Code view

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.7/site-packages/airflow/www/auth.py", line 34, in decorated
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 905, in code
    wrapped=conf.getboolean('webserver', 'default_wrap'),
  File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 443, in render_template
    **kwargs,
  File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/baseviews.py", line 288, in render_template
    template, **dict(list(kwargs.items()) + list(self.extra_args.items()))
  File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 140, in render_template
    ctx.app,
  File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 120, in _render
    rv = template.render(context)
  File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/dag_code.html", line 20, in top-level template code
    {% extends "airflow/dag.html" %}
  File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/dag.html", line 21, in top-level template code
    {% from 'appbuilder/dag_docs.html' import dag_docs %}
  File "/usr/local/lib/python3.7/site-packages/astronomer/airflow/version_check/templates/astro-baselayout.html", line 1, in top-level template code
    {% extends airflow_base_template %}
  File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/main.html", line 20, in top-level template code
    {% extends 'appbuilder/baselayout.html' %}
  File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 2, in top-level template code
    {% import 'appbuilder/baselib.html' as baselib %}
  File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 37, in top-level template code
    {% block body %}
  File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 19, in block "body"
    {% block content %}
  File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/dag_code.html", line 30, in block "content"
    {{ super() }}
  File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/dag.html", line 139, in block "content"
    <input type="hidden" name="origin" value="{{ url_for('Airflow.' + dag.default_view, dag_id=dag.dag_id) }}">
TypeError: can only concatenate str (not "NoneType") to str

We tried deleting one of the affected dags with the "Delete Dag" button and then when the dag file was re-parsed, it through a dag import error, which was helpful.

What you expected to happen:

Get the dag import error immediately, instead of allowing me to click on the dag and get nondescriptive error message.

How to reproduce it:

1. Write a valid dag
2. Have it parsed by an Airflow instance (might need to be pre-2.0)
3. Upload a new, broken version of the dag file.
4. (Unclear if necessary) Upgrade to airflow 2.1.1
5. In the UI, go to the dag and click on Code or Graph view.

Anything else we need to know:

It's possible that this problem was affecting our 1.10.14 instance as well. These dags were only checked as part of verifying the upgrade and had not run or been looked at since the change that caused the import error.

[ephraimbuddy]

Fixed in #17818. @collinmcnulty we now show import errors on dag views, please verify

[collinmcnulty]

@ephraimbuddy Confirmed. Tested this on main.

[michaelswierszcz]

Same error, deleted dag and reprocessed to resolve