Airflow 2.3.0 webserver fails to start

[alexvkaam]

Apache Airflow version

2.3.0 (latest released)

What happened

upgraded from 2.2.5

airflow db upgrade showed a list of messages inline with what is in the changelog:

Database configuration moved to new section (#22284)

The following configurations have been moved from [core] to the new [database] section. However when reading the new option, the old option will be checked to see if it exists. If it does a DeprecationWarning will be issued and the old option will be used instead.

sql_alchemy_conn
sql_engine_encoding
sql_engine_collation_for_ids
sql_alchemy_pool_enabled
sql_alchemy_pool_size
sql_alchemy_max_overflow
sql_alchemy_pool_recycle
sql_alchemy_pool_pre_ping
sql_alchemy_schema
sql_alchemy_connect_args
load_default_connections
max_db_retries

so nice as I am I moved those lines from [core] to a new [database] section and airflow db upgrade did not mention them again so all was well.

After that I started the services

sudo systemctl start airflow-webserver.service
sudo systemctl start airflow-scheduler.service

but the webservice failed with this error:

May 04 08:46:21 postest airflow[108983]: [2022-05-04 08:46:21 +0200] [108983] [ERROR] Exception in worker process
May 04 08:46:21 postest airflow[108983]: [2022-05-04 08:46:21,254] {configuration.py:494} WARNING - section/key [core/sql_alchemy_conn] not found in config
May 04 08:46:21 postest airflow[108982]: [2022-05-04 08:46:21 +0200] [108982] [INFO] Worker exiting (pid: 108982)
May 04 08:46:21 postest airflow[108982]: airflow.exceptions.AirflowConfigException: section/key [core/sql_alchemy_conn] not found in config

for now I duplicated the lines, so they are both in [core] and [database], so no big issue.

What you think should happen instead

service should start and find the settings in the [database] section

How to reproduce

upgrade from 2.2.5, move the lines in question to a new [database] section and start the webservice.

Operating System

Ubuntu 20.04.4 LTS (Focal Fossa)

Versions of Apache Airflow Providers

apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-sqlite==2.1.3

Deployment

Other

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Likeluy your worker processes are not upgraded yet. Check which airflow versions they have.

[potiuk]

No idea. maybe different virtualenv??

[potiuk]

What does your webserver version UI report ? Can you send some screenshots and more logs from starting up as a gist - of both scheduler and webserver?

[alexvkaam]

hi

attached systemd log files from starting the webserver and scheduler when there is no sql_alchemy_conn in [core] and with sql_alchemy_conn in [core]

webserver gives the issue, the scheduler does not.
webserver version on screen:

Version: v2.3.0
Git Version: .dev0+6b772f68db4f80b01ede836ddd47437d4aa1b43d.dirty

upgrade from 2.2.5 to 2.3.0:

pip3.7 install apache-airflow[postgres,password]==2.3.0 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.3.0/constraints-3.7.txt

• Alex

airflow-scheduler.txt
airflow-webserver.txt

[potiuk]

Apparently you have some very old webserver_config.py. This line was removed before we released Airflow 2.0.0 (there webserver started to read configuration internally - before that, the entry was duplicated in webserver_config.py

May 11 07:13:05 xxxxxxx airflow[831498]:   File "/home/akm/airflow/webserver_config.py", line 32, in <module>
May 11 07:13:05 xxxxxxx airflow[831498]:     SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')

Look at default_webserver_config.py in Airflow 2.3.0 and replace your config with one following Airlfow 2 defaults

[alexvkaam]

that was the cause.
the system has been updated before from 1.x so probably something left there.
thank you !

[potiuk]

Converted to a discussion.

[pritampbonde]

@potiuk , We upgraded to 2.3.0 and used mySQL db.

In airflow DB one of the table dag_run although it exits also has entries of all the dags which are running, but when the scheduler triggers the Dag, it gives the error mentioned in the "Error" section.

Request, to help us to know if there are any configuration-related issues or if any RBAC got introduced in 2.3.0 release. Also let us know if you guide us to open a new discussion thread. Thought to use this thread as the issue is on the similar lines

Apache Airflow version
2.3.0 (latest released)

What happened
upgraded from 2.2.5

Issue
After upgrade 2.3.0

Error
[2022-05-23, 08:52:47 UTC] {taskinstance.py:1377} INFO - Executing <Task(PythonOperator): run_bi_reporting> on 2022-05-22 12:00:00+00:00
[2022-05-23, 08:52:47 UTC] {standard_task_runner.py:52} INFO - Started process 1293 to run task
[2022-05-23, 08:52:47 UTC] {standard_task_runner.py:79} INFO - Running: ['airflow', 'tasks', 'run', 'some_dag', 'run_some_dag', 'scheduled__2022-05-22T12:00:00+00:00', '--job-id', '14', '--raw', '--subdir', 'DAGS_FOLDER/some_dag.py', '--cfg-path', '/tmp/tmp93nsyc6c', '--error-file', '/tmp/tmp411a8_5i']
[2022-05-23, 08:52:47 UTC] {standard_task_runner.py:80} INFO - Job 14: Subtask run_bi_reporting
[2022-05-23, 08:52:47 UTC] {standard_task_runner.py:97} ERROR - Failed to execute job 14 for task run_some_dag ((sqlite3.OperationalError) no such table: dag_run
[SQL: SELECT dag_run.state AS dag_run_state, dag_run.id AS dag_run_id, dag_run.dag_id AS dag_run_dag_id, dag_run.queued_at AS dag_run_queued_at, dag_run.execution_date AS dag_run_execution_date, dag_run.start_date AS dag_run_start_date, dag_run.end_date AS dag_run_end_date, dag_run.run_id AS dag_run_run_id, dag_run.creating_job_id AS dag_run_creating_job_id, dag_run.external_trigger AS dag_run_external_trigger, dag_run.run_type AS dag_run_run_type, dag_run.conf AS dag_run_conf, dag_run.data_interval_start AS dag_run_data_interval_start, dag_run.data_interval_end AS dag_run_data_interval_end, dag_run.last_scheduling_decision AS dag_run_last_scheduling_decision, dag_run.dag_hash AS dag_run_dag_hash, dag_run.log_template_id AS dag_run_log_template_id
FROM dag_run
WHERE dag_run.dag_id = ? AND dag_run.run_id = ?
LIMIT ? OFFSET ?]
[parameters: ('some_dag', 'scheduled__2022-05-22T12:00:00+00:00', 1, 0)]

Providers info
apache-airflow-providers-ftp | 2.1.2
apache-airflow-providers-http | 2.1.2
apache-airflow-providers-imap | 2.2.3
apache-airflow-providers-sqlite | 2.1.3

[potiuk]

Likely this error has been fixed in 2.3.1.

If that's it (very likely) you can solve without waiting for 2.3.1 it by following the deprecation warmings you should see in Airflow logs. The "sql_alchemy_conn" configuration moved from "core" section to "database" and for "run_as_user" case the deprecated parameter were overwritten by the default value of "database" parameter.

#23723

If you use config - move the "sql_alchemy_conn" value from "core" to "database" (I recommend to address also all other deprecation configuration messages if you have them). If you use env vars or secrets you have to change them accordingly (for example env var should be changed from AIRFLOW__CORE__SQL_ALCHEMY_CONN to AIRFLOW__DATABASE__SQL_ALCHEMY_CONN .

In 2.3.1 (just being voted) it should work regardless if you changed the deprecated value to the new or not.

[pritampbonde]

Thanks for the quick reply @potiuk , we shall try to change the configuration and will update you soon

[pritampbonde]

Hello @potiuk , we have made the changes and now airflow 2.3.0 is working. Although, in logs it appears to be WARNING but seems its not working if we do not introduce the [database] section. Once we introduce [database] and pull all the database related keys:values under this section its working fine. Thanks for point out and fixing this issue in 2.3.1 release

[dimon222]

This still occurs for me for Webserver even on 2.3.1. Is it solved for sure?
I use this env variable: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN

[potiuk]

Well. It is impossible to answer your question authoritatively. We believe it is. And if you do not provide any more details that woudl allow us to change the understanding, tthere is no logical way we can change our understanding, so your questions is pretty rhethorical.

Let me answer it then : Yes. According to our knowledge it is and people confirmed it. Very likely you have a different problem even if you produces similar error. Please take your time and describe the circumstances and logs and open a new issue if you have reproducible case (or start discussion with all the details if you don't have a reproducible case but would like to have some help with diagnosis).

[dimon222]

1. I use 2.3.1 from pypi with latest constraints file for 2.3.1 on python 3.9 release
2. I don't use airflow.cfg to configure sql alchemy connection, I export environment variable AIRFLOW__DATABASE__SQL_ALCHEMY_CONN instead.
3. The error that occurs is 100% same as what OP has. That is:

Worker exiting (pid: 108982)
airflow.exceptions.AirflowConfigException: section/key [core/sql_alchemy_conn] not found in config

4. If I change the variable back to old AIRFLOW__CORE__SQL_ALCHEMY_CONN I get the warning that its deprecated, but it works and I don't encounter any webserver's worker exit.

My theory - it was fixed for airflow.cfg configuration controls, but due to some specific circumstances the environment variable way is behaving somewhat differently. Its worth to mention that I do also use airflow.cfg, its just the sql alchemy conn variable is not defined there (purposefully, to not hardcode the password). The setup was working fine before the upgrade to 2.3.x tree. (from 2.2.5)

[potiuk]

Then it is cimpletely different problem. You have likely exactly the same problem see the Answer (and unfold it):

The problem was (and it is likely yours too) that some part of the system (workers) were not upgraded:

that was the cause.
the system has been updated before from 1.x so probably something left there.
thank you !

Please review your deployment and make sure you upgrade ALL components.

[dimon222]

Resolved the issue after all.
Pointer was:

the system has been updated before from 1.x so probably something left there.

A bit more detail for whoever might encounter same:
I used custom webserver_config.py back from 1.10.x times. In PR #11364 the baseline default_webserver_config.py was changed with one of things becoming redundant - sql alchemy URI was moved to more generic reusable component. Unfortunately, my review of major changes from 1.x. to 2.x didn't catch that change so this ended up technical debt until issue occurred now when this source section changed.