DagBag Processing SIGSEGV causes runaway tasks with LocalExecutor

[amcraig]

Apache Airflow version

3.1.0

If "Other Airflow 2/3 version" selected, which one?

No response

What happened?

Using Airflow 3.1.0 - airflow standalone with the LocalExecutor on MacOS 14 - M3 ARM

I've noticed that occasionally tasks that are dynamically created (with .expand(), I don't think this is the issue fyi, just including), a subset of the them can runway, and will not fail even with a execution_timeout set. It completely disregards the timeout after this error is found in the logs:

ERROR - Received SIGSEGV signal while processing <my dag path>. source=airflow.models.dagbag.DagBag loc=dagbag.py:375

Here's some screenshots from a manually failed run, note the duration of 30 minutes even though a max timeout of 3 minutes:

Log from the Task:

Dag definition (linting errors are unrelated to this issue and are in a different task.):

---

I have a hunch this happens when the DAG Processor starts running at the same time as all of the tasks are dynamically created, but I truthfully have no clue.

I don't really have an issue with tasks like this failing on standalone due to a perfect storm of conditions (and non-intended OSes) as I only use it for local testing, but I would like the tasks to not run away.

---

Possibly Related Ticket:
#47011
#55838
#50303
#35967

What you think should happen instead?

No response

How to reproduce

No clue, besides try casting a wide net with many dynamically created tasks in the chance that this error bubbles up.
I've included my DAG code besides my custom BoxHook in case this helps.

Operating System

MacOS 14 - M3 ARM

Versions of Apache Airflow Providers

apache-airflow-providers-google==18.0.0
apache-airflow-providers-standard==1.8.0

Deployment

Other

Deployment details

Local Standalone

Anything else?

I think it bubbles up in the DAG run about 20% of the time.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Corrrect - currently timeout is handled in the task itself via SIGALRM - and if another signal (SIGSEGV) comes it might get task into a hanging state. This happens usually when you have native code that runs in a long tight loop and does not handle signals in Python. We've discussed it in the past and solution to that is to handle the timeout in supervisor task (which is in another, parent process - tasks are in forked processes).

In order to communicate that we have to make a new task -> supervisor API (to send the timeout information after parsing the DAG, because the supervisor does not have that information as it d does not parse the task). This would generally handle all the possible cases where the task hangs - including possible escalation of signals to kill such forked tasks (SIGTERM followed by SIGKILL after a short additional timeout if the task does not exit).

That's all possible and if someone would like to take on that task - it's not even that difficult. I marked it as a good-first issue.

cc: @ashb @amoghrajesh if you have something to add.

[LeonardoIshida]

Hi, can I be assigned to this?

[potiuk]

Feel free.

[amoghrajesh]

@LeonardoIshida this is the actual ticket: #53337, feel free to pick it up and if you have any qns, feel free to ask on that issue

[qwe-kev]

Hi, I would like to work on this.

[potiuk]

Added you - assuming @LeonardoIshida is not about to work on it :) ?