Logging seems to freeze or break airflow

[noklam]

Description

Short description of the problem here.

1. Kedro-Airflow not working with Astrocloud #13 (comment) It was first found that logging is breaking kedro-airflow sometimes, this causing the airflow nodes fail immediately during the execution of first node. The workaround is documented in the link.

2. A new symptoms seem to be rich is interacting with airflow.
   @DimedS :
   When deploying a Kedro project to Airflow using Astronomer, as detailed in this manual, an issue was encountered where the first node would fail after a few minutes, culminating in a {task_context_logger.py:104} ERROR - Detected zombie job error message.

The problem was resolved by modifying the logging.yml file to utilize custom logging settings.
from

root:
  handlers: [rich, info_file_handler]

to

root:
  handlers: [console]

Context

How has this bug affected you? What were you trying to accomplish?

Steps to Reproduce

1. [First Step]
2. [Second Step]
3. [And so on...]

Expected Result

Tell us what should happen.

Actual Result

Tell us what happens instead.

-- If you received an error, place it here.

-- Separate them if you have more than one.

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• Kedro version used (pip show kedro or kedro -V):
• Kedro plugin and kedro plugin version used (pip show kedro-airflow):
• Python version used (python -V):
• Operating system and version:

[astrojuanlu]

The pressure is mounting to do tackle kedro-org/kedro#2928

[astrojuanlu]

Also I'm almost sure I've been affected by this and the only workaround I found was using a Dockerized version of Airflow instead of the pip install I tried initially.

[noklam]

@astrojuanlu Do you know why would a dockerized version help? Good to know this as a workaround though.

[noklam]

The pressure is mounting to do tackle kedro-org/kedro#2928

This may solve problem 2, but not 1 (not sure if it still persists), from experience it also depends on the airflow version. I tried to look at both rich or airflow Github repo and couldn't find any related problem, is it worth to do some investigation to see if this is an issue created by Kedro instead?

[em-pe]

@noklam Not sure how much of help this would be but I managed to fix both of the issues disabling kedro logging at all by commenting out following line.

rich made the pipelinie stuck and "disable_existing_loggers": True made the configure_project fail with -1 error code and no error logs. When you disable logging changes done by kedro everything seems to work fine, including logging information from airflow tasks.

[noklam]

@em-pe I think you are right I found that previously though the root cause is still unknown.

[CF-FHB-X]

Not sure if there are any updates on this issue. I can confirm that @em-pe 's changes worked (commenting out the line in kedro's framework/project/__init__.py file), at which point logging in Airflow then works as expected. Unfortunately setting disable_existing_loggers: True turns off all logging, including Airflow's.

Right now my workflow for deploying to Airflow (don't judge me) has a step to comment out that line of code in kedro's site_package folder. It's far from ideal, but seems to be the only way I can get kedro pipelines deployed and working.

Have there been further discussions on making changes to get this working?