Airflow 2.3.1 - gunicorn keeps removing and adding Permission menu access on Permissions to role Admin #23926

gbonazzoli · 2022-05-26T04:00:43Z

Apache Airflow version

2.3.1 (latest released)

What happened

gunicorn log file is full of these messages.

This behavior was present also in release 2.3.0

I don't think that was a problem in Airflow 2.2.x

The problem is that web server now is much weekier than before and it is the second time in a week that I found it not running.

[2022-05-26 05:50:55,083] {manager.py:585} INFO - Removed Permission menu access on Permissions to role Admin
[2022-05-26 05:50:55,098] {manager.py:543} INFO - Removed Permission View: menu_access on Permissions
[2022-05-26 05:50:55,159] {manager.py:508} INFO - Created Permission View: menu access on Permissions
[2022-05-26 05:50:55,164] {manager.py:568} INFO - Added Permission menu access on Permissions to role Admin
[2022-05-26 05:51:25,688] {manager.py:585} INFO - Removed Permission menu access on Permissions to role Admin
[2022-05-26 05:51:25,704] {manager.py:543} INFO - Removed Permission View: menu_access on Permissions
[2022-05-26 05:51:25,763] {manager.py:508} INFO - Created Permission View: menu access on Permissions
[2022-05-26 05:51:25,769] {manager.py:568} INFO - Added Permission menu access on Permissions to role Admin
[2022-05-26 05:51:56,283] {manager.py:585} INFO - Removed Permission menu access on Permissions to role Admin
[2022-05-26 05:51:56,297] {manager.py:543} INFO - Removed Permission View: menu_access on Permissions
[2022-05-26 05:51:56,353] {manager.py:508} INFO - Created Permission View: menu access on Permissions
[2022-05-26 05:51:56,358] {manager.py:568} INFO - Added Permission menu access on Permissions to role Admin
[2022-05-26 05:52:26,954] {manager.py:585} INFO - Removed Permission menu access on Permissions to role Admin
[2022-05-26 05:52:26,971] {manager.py:543} INFO - Removed Permission View: menu_access on Permissions
[2022-05-26 05:52:27,033] {manager.py:508} INFO - Created Permission View: menu access on Permissions
[2022-05-26 05:52:27,040] {manager.py:568} INFO - Added Permission menu access on Permissions to role Admin
[2022-05-26 05:52:57,595] {manager.py:585} INFO - Removed Permission menu access on Permissions to role Admin
[2022-05-26 05:52:57,610] {manager.py:543} INFO - Removed Permission View: menu_access on Permissions
[2022-05-26 05:52:57,679] {manager.py:508} INFO - Created Permission View: menu access on Permissions
[2022-05-26 05:52:57,684] {manager.py:568} INFO - Added Permission menu access on Permissions to role Admin
[2022-05-26 05:53:28,130] {manager.py:585} INFO - Removed Permission menu access on Permissions to role Admin
[2022-05-26 05:53:28,148] {manager.py:543} INFO - Removed Permission View: menu_access on Permissions
[2022-05-26 05:53:28,209] {manager.py:508} INFO - Created Permission View: menu access on Permissions
[2022-05-26 05:53:28,214] {manager.py:568} INFO - Added Permission menu access on Permissions to role Admin

What you think should happen instead

Log must be clear and gunicorn stable as a rock.

How to reproduce

Just launch gunicorn

Operating System

Ubuntu 20.04.4 LTS

Versions of Apache Airflow Providers

apache-airflow-providers-celery==2.1.4
apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-microsoft-mssql==2.1.3
apache-airflow-providers-microsoft-winrm==2.0.5
apache-airflow-providers-mysql==2.2.3
apache-airflow-providers-openfaas==2.0.3
apache-airflow-providers-oracle==2.2.3
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-samba==3.0.4
apache-airflow-providers-sftp==2.6.0
apache-airflow-providers-sqlite==2.1.3
apache-airflow-providers-ssh==2.4.4

Deployment

Virtualenv installation

Deployment details

gunicorn is configured with AUTH_LDAP option

Anything else

no

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

jhtimmins · 2022-05-26T22:43:53Z

@gbonazzoli Thanks for reporting this. I've been able to replicate the issue locally and will work on a fix.

lukas-at-harren · 2022-05-27T12:47:47Z

@jhtimmins can you push a PR asap, because we might need that very quickly and are willing to help look into the problem as well. Happy to see your failure and issue replication.

ashb · 2022-05-27T14:50:32Z

@lukas-at-harren A possible workaround for now is to set AIRFLOW__WEBSERVER__UPDATE_FAB_PERMS=False env var. That should disable the code that is causing this, and I think have minimal impact (the only possibly one is if you have DAG level permissions set you might need to run a CLI command to update the perms.

dataminding · 2022-05-27T15:59:13Z

@ashb I tried with the env var you mentioned but the webserver workers keep crashing and the web UI doesn't load. Tried with Airflow versions 2.3.0 and 2.3.1 . Some logs I found relevant:

objc[88319]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[88319]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
[2022-05-27 17:52:13 +0200] [86717] [WARNING] Worker with pid 88319 was terminated due to signal 9
[2022-05-27 17:52:13 +0200] [88376] [INFO] Booting worker with pid: 88376

potiuk · 2022-05-27T16:10:54Z

@dataminding - you seem to run Airlfow on MacOS which is not something that we support natively. Please try to replicate it on linux.

gbonazzoli · 2022-05-27T16:17:03Z

@ashb On Linux (Ubuntu 20.04) AIRFLOW__WEBSERVER__UPDATE_FAB_PERMS=False is a valid work around !

Many thanks.

zachliu · 2022-05-29T03:49:54Z

hey i already found a layman solution here: #23512 (comment) 😁

renatogav · 2022-06-06T18:29:53Z

great, that worked

gitruhul · 2022-08-03T07:17:16Z

I am not sure if it is allowed to comment on a closed issue.

After adding AIRFLOW__WEBSERVER__UPDATE_FAB_PERMS=False, My role mapping is not working properly.

I have integrated Airflow with an Identity Gateway through OAuth. By the mapping, I was supposed to get an Admin role UI, but I am seeing no roles (Public roles) page.

If I set the above flag to True, I am seeing the Admin role (UI with all menus) but there are webserver-related errors as mentioned in #23512.

tschroeder-zendesk · 2022-11-17T01:18:42Z

We upgraded from 2.3.4 -> 2.4.2 in hopes this would be fixed, but we are still getting tons of duplicate entry errors.

mysql.connector.errors.IntegrityError: 1062 (23000): Duplicate entry '2783-1' for key 'permission_view_id'

This is causing so many headaches. Our schedulers keep dying and we have to manually restart task and several other related issues.

zachliu · 2022-11-17T03:01:56Z

@tschroeder-zendesk you need to go to 2.4.3

gbonazzoli added area:core kind:bug This is a clearly a bug labels May 26, 2022

ashb assigned jhtimmins May 27, 2022

uranusjr mentioned this issue May 29, 2022

Random "duplicate key value violates unique constraint" errors when initializing the postgres database #23512

Closed

2 tasks

jhtimmins mentioned this issue Jun 1, 2022

Rename Permissions to Permission Pairs. #24065

Merged

jhtimmins closed this as completed in #24065 Jun 2, 2022

uranusjr mentioned this issue Jun 8, 2022

Webserver keeps restarting due to CrashLoopBackOff #24295

Closed

2 tasks

This was referenced Jul 2, 2022

Status of testing of Apache Airflow 2.3.3rc1 #24806

Closed

Status of testing of Apache Airflow 2.3.3rc3 #24863

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Airflow 2.3.1 - gunicorn keeps removing and adding Permission menu access on Permissions to role Admin #23926

Airflow 2.3.1 - gunicorn keeps removing and adding Permission menu access on Permissions to role Admin #23926

gbonazzoli commented May 26, 2022

jhtimmins commented May 26, 2022

lukas-at-harren commented May 27, 2022

ashb commented May 27, 2022

dataminding commented May 27, 2022 •

edited

Loading

potiuk commented May 27, 2022 •

edited

Loading

gbonazzoli commented May 27, 2022

zachliu commented May 29, 2022

renatogav commented Jun 6, 2022

gitruhul commented Aug 3, 2022 •

edited

Loading

tschroeder-zendesk commented Nov 17, 2022

zachliu commented Nov 17, 2022

Airflow 2.3.1 - gunicorn keeps removing and adding Permission menu access on Permissions to role Admin #23926

Airflow 2.3.1 - gunicorn keeps removing and adding Permission menu access on Permissions to role Admin #23926

Comments

gbonazzoli commented May 26, 2022

Apache Airflow version

What happened

What you think should happen instead

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else

Are you willing to submit PR?

Code of Conduct

jhtimmins commented May 26, 2022

lukas-at-harren commented May 27, 2022

ashb commented May 27, 2022

dataminding commented May 27, 2022 • edited Loading

potiuk commented May 27, 2022 • edited Loading

gbonazzoli commented May 27, 2022

zachliu commented May 29, 2022

renatogav commented Jun 6, 2022

gitruhul commented Aug 3, 2022 • edited Loading

tschroeder-zendesk commented Nov 17, 2022

zachliu commented Nov 17, 2022

dataminding commented May 27, 2022 •

edited

Loading

potiuk commented May 27, 2022 •

edited

Loading

gitruhul commented Aug 3, 2022 •

edited

Loading