Task exited with return code Negsignal.SIGSEGV when dag triggered from UI #24463

shelly2904 · 2022-06-15T10:20:31Z

shelly2904
Jun 15, 2022

Hi,

I am trying to trigger a DAG from Airflow UI which internally has a task to call requests.get(). This results in below error:
Task exited with return code Negsignal.SIGSEGV

This is working fine when tested using command airflow dags test <dag_id> <execution_start_date>

Below versions are used:

Airflow==2.3.1
Python==3.8.0
requests==2.27.1

Can you please advise?

Thanks.

potiuk · 2022-06-15T10:57:54Z

potiuk
Jun 15, 2022
Collaborator

This is not an airflow issue. It's issue with your code.

You need to look at your code and see which component fails with sigsegv. It's some of your code and custom libraries, sigsegv comes from C-libraries usually. You need to look how your deployment is done and try to reproduce it there. When you run stuff manually (from terminal) you have different environment than when airflow is run as a "daemon" or service (for example airflow has no terminal with user). You need to check how your airflow is deployed (possibly with your admin/devops) and analyse logs/core dumps produced on the server. Usually (depends on your server configuration) when a process dies with sigsegv it produces "core dumps" - and you can trace them, but how and where - this is something that only you and your admin can track.

Again - not airflow issue, look at your deployment. Good luck.

3 replies

maver1ck Dec 28, 2023

@potiuk
It's issue connected with using os.fork. Do you know if the Airflow is using it?
Reference: python/cpython#105912

potiuk Dec 28, 2023
Collaborator

Yes. A lot. and some of the usages are configurable. But look through the whole thread here for various ways usage of some libraries might trigger SIGSEGV. You wil find many ways that you might try to mitigate your issue depending on the library you use. Unfortunately - (despite the 26 thumbs down that do not bring any value) this does not change the fact that Airflow on its own cannot make it any clearer on which library and usage of yours causes the problem.

Taragolis Dec 28, 2023
Collaborator

Started from 2.5.1 we even write message if SIGSEGV happen in LocalTaskJobRunner how to debug

airflow/airflow/jobs/local_task_job_runner.py

Lines 47 to 74 in 3c0a714

    
           ******************************************* Received SIGSEGV ******************************************* 
        
           SIGSEGV (Segmentation Violation) signal indicates Segmentation Fault error which refers to 
        
           an attempt by a program/library to write or read outside its allocated memory. 
        
           In Python environment usually this signal refers to libraries which use low level C API. 
        
           Make sure that you use right libraries/Docker Images 
        
           for your architecture (Intel/ARM) and/or Operational System (Linux/macOS). 
        
           Suggested way to debug 
        
           ====================== 
        
             - Set environment variable 'PYTHONFAULTHANDLER' to 'true'. 
        
             - Start airflow services. 
        
             - Restart failed airflow task. 
        
             - Check 'scheduler' and 'worker' services logs for additional traceback 
        
               which might contain information about module/library where actual error happen. 
        
           Known Issues 
        
           ============ 
        
           Note: Only Linux-based distros supported as "Production" execution environment for Airflow. 
        
           macOS 
        
           ----- 
        
            1. Due to limitations in Apple's libraries not every process might 'fork' safe. 
        
               One of the general error is unable to query the macOS system configuration for network proxies. 
        
               If your are not using a proxy you could disable it by set environment variable 'no_proxy' to '*'. 
        
               See: https://github.com/python/cpython/issues/58037 and https://bugs.python.org/issue30385#msg293958 
        
           ********************************************************************************************************"""

yakimzhang · 2022-09-10T14:27:52Z

yakimzhang
Sep 10, 2022

the same error on mac

0 replies

shelly2904 · 2022-09-10T14:43:16Z

shelly2904
Sep 10, 2022
Author

Hi, I was trying to use airflow in MacOS and was getting the same error. On debugging, I found out that it was crashing in urllib. Below is the bug https://bugs.python.org/issue28342 I fixed it by adding an environmental variable NO_PROXY to bypass the required host. Let me know if this is helpful. Thanks.

…

On Sat, 10 Sep, 2022, 7:58 pm Yakim12, ***@***.***> wrote: the same error on mac — Reply to this email directly, view it on GitHub <#24463 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AECBVZQZIA3QHDXLDBFZ4BDV5SLHHANCNFSM5Y2YCJRA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

24 replies

potiuk Apr 9, 2023
Collaborator

I believe this is an issue with some incompatibilities and racing conditions of basic SSL libraries in Python on ARM simply c-generated code on ARM has a bug that triggers the issue when looking for proxy (i believe). This has nothing to do with airflow or even requests i think. You can find corresponding issues in Python. (just look it up in Google) for NO_PROXY and SIGSEGV and you will find relevant discussions if you are curious and want to dig deeper.

If you find more details that cna shed some light - please bring it back here if you find more exact and correct explanation. I would be courious to learn but had no time to dig deeper but f someeone has the curiosity and time i will be super happy to learn more.

csm10495 Apr 9, 2023

It's not just ARM. It can happen on amd64 too.

potiuk Apr 9, 2023
Collaborator

This is the first time I hear it that it can happen on AMD with relation to NO_PROXY at least. It is likely another issue, but since there are no details, I am not
aware about NO_PROXY fix in relation to AMD.

Let me just say that SIGSEGV is completely independent from airflow because airflow is python program. You can develop any c-binary that produces SIGSEGV with two lines of code, so generally any binary that is executed might generate it if it has a bug. There might be as many sigsegv as many differnt binary libraries or binary processes you run (or more). Each of them might be different and each of them caused by different bug. And with the nature of sigsegv stopping an interpreter pretty much immediately, there is usually not much information airflow can provide about the root cause.

We discussed it extensively and in recent version of airflow you should get some hints on how you can debug it and try to find out which of the binaries cause it by setting PYTHONFAULTHANDLER ENV variable https://docs.python.org/3/library/faulthandler.html.
But basically eeach SIGSEGV might have a different reason and is pretty much always external to Airflow.

csm10495 Apr 9, 2023

The solution here: #24463 (comment) was how we worked around it regardless of arch. We definitely were hitting it on amd64.

I know it's not an airflow specific problem, it just so happens to get hit with airflow. This thread is very useful for folks who hit it in that case.

potiuk Apr 9, 2023
Collaborator

Ah. OK. It's the first time I 've heard it is AMD as well. Thanks for letting me know @csm10495 )

I did some digging in and it does seem like a limitation of scproxy library not working correctly with MacOS and likely it traces back to an old error that is closed as "3r-party bug".

https://bugs.python.org/issue13829

And looks like more details are @keiraaa-xrq):

The crash in _scproxy is due to limitations in Apple's libraries, in particular they don't work in child processes created with os.fork() without calling execv*()

yakimzhang · 2022-09-14T01:43:09Z

yakimzhang
Sep 14, 2022

from web3 import Web3
web3.eth.get_balance(address)

This code also triggers an Negsignal.SIGSEGV ，But I don't know how to fix yet

0 replies

chajath · 2022-09-19T22:12:29Z

chajath
Sep 19, 2022

I also ran into this kind of issue when locally running the DAG that uses s3fs. Looking at the trace, I think root cause of a lot of issues reported here can be traced to issues like post-fork incompatibility on mac: python/cpython#58037
It makes sense that these issues don't pop up when running with dag test but it is shown when running with a serial executor, as the executor forks before running the task.

To mitigate the issue, one easy solution could be to use a real simple executor that does not fork. Will see if there is existing Airflow setup that can achieve this through configuration.

5 replies

potiuk Sep 20, 2022
Collaborator

Actualy this is a good idea - We have it https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#execute-tasks-new-python-interpreter and maybe it could be set to True by default for MacOS ?

@chajath maybe you care for a PR implementing it ?

potiuk Sep 20, 2022
Collaborator

If not I can do it, but it would be a nice way to become contributor :)

chajath Sep 20, 2022

Great. I'll take a stab at it sometime this week when I get the chance to.

leonardara Sep 30, 2022

@potiuk i run into this issue with the GCSFileTransformOperator, but seems even after i set the env var AIRFLOW__CORE__EXECUTE_TASKS_NEW_PYTHON_INTERPRETER to True the issue still persists, how can i get around of it?

potiuk Sep 30, 2022
Collaborator

No idea. There must be other reasons - as mentioned before, we are not able to track SIGSEGV source as it is independent from Airflow. Wild guess is that if you are on M1 and have older Python version/SSL people have (not only with Airflow - it's a general issue) similar problems. look for SIGSEGV in airflow issues/discussions and you will find some recent tasks that refer to it.

herrykarmito92 · 2022-11-14T01:13:20Z

herrykarmito92
Nov 14, 2022

I'm facing same problem like on this forum.
Thanks to @potiuk , mentions about library broken and nothing do with airflows.

I figure it out, last week I have a POC with telegram bot and installed a ton of Library package on my airflow virtual env.

Than I tried to so many tools to figure it out , and questioning my self :
Q : How can i know which library using pip freeze or pip list is fresh installed 7 days from now ?
A : Lucky, I found this tools : pip-date and try to uninstall some error library after using this tools scanning and run the airflow.

After re-run airflow server again , airflow show error about no module.
Because I save the command before I run it in my notepad , then I can re-track again which library I need to installed again.

Note :

every version of ariflow needed very specific version of library

0 replies

csm10495 · 2022-12-14T22:32:26Z

csm10495
Dec 14, 2022

I started hitting this when using requests with Airflow 2.5.0 on Mac.

We seem to be able to workaround it by running:

    from _scproxy import _get_proxy_settings

    _get_proxy_settings()

in the process that kicks off Airflow's scheduler, workers, etc.

Not a great solution.. but leaving here in case it helps someone else.

2 replies

Noushath Jan 12, 2023

I started hitting this when using requests with Airflow 2.5.0 on Mac.

We seem to be able to workaround it by running:
    from _scproxy import _get_proxy_settings

    _get_proxy_settings()
in the process that kicks off Airflow's scheduler, workers, etc.

Not a great solution.. but leaving here in case it helps someone else.

True Saviour!!!
For mac airflow Task exited with return code Negsignal.SIGSEGV error adding below to dag definition file saved my day with Airflow 2.5.0
from _scproxy import _get_proxy_settings

_get_proxy_settings()

Please find my configs below:

`Apache Airflow
version | 2.5.0
executor | SequentialExecutor

System info
OS | Mac OS
architecture | arm
uname | uname_result(system='Darwin', node='US-P2QH240FMG', release='22.2.0', version='Darwin Kernel Version 22.2.0: Fri Nov 11 02:03:51 PST 2022;
| root:xnu-8792.61.2~4/RELEASE_ARM64_T6000', machine='arm64')
locale | ('en_GB', 'UTF-8')
python_version | 3.10.9 (main, Dec 15 2022, 17:11:09) [Clang 14.0.0 (clang-1400.0.29.202)]
`

htrueman Jan 16, 2023

You helped A LOT. Many thanks!

RusabKhan · 2024-03-24T11:19:40Z

RusabKhan
Mar 24, 2024

This problem still exists on macOS.

Details:
macOS: Sonoma 14.4
Python: 3.9.18
Airflow: 2.8.1

A workaround is to add os.environ['NO_PROXY'] = '*' to your airflow dags.

1 reply

potiuk Mar 24, 2024
Collaborator

Yes. This is problem with Python not Airflow, so yep. It still exists.

EricGao888 · 2024-05-15T06:29:52Z

EricGao888
May 15, 2024

Just encountered with this issue yesterday and spent half a day searching clues until reading this discussion.

The issue appeared when remote task handler trying to fetching logs from remote storage. Error output like:

[2024-05-15 14:22:38 +0800] [54502] [ERROR] Worker (pid:54823) was sent SIGSEGV!
[2024-05-15 14:22:38 +0800] [54827] [INFO] Booting worker with pid: 54827

After setting env vars no_proxy=*, things worked out.

macOS Ventura 13.1
Python 3.10

Looks like not an Airflow problem. I record the log here just in case other users / developers come into the same issue when using remote logging.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Task exited with return code Negsignal.SIGSEGV when dag triggered from UI #24463

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 9 comments 35 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Task exited with return code Negsignal.SIGSEGV when dag triggered from UI #24463

Replies: 9 comments · 35 replies

potiuk Jun 15, 2022 Collaborator

potiuk Dec 28, 2023 Collaborator

Taragolis Dec 28, 2023 Collaborator

shelly2904 Sep 10, 2022 Author

potiuk Apr 9, 2023 Collaborator

potiuk Apr 9, 2023 Collaborator

potiuk Apr 9, 2023 Collaborator

potiuk Sep 20, 2022 Collaborator

potiuk Sep 20, 2022 Collaborator

potiuk Sep 30, 2022 Collaborator

potiuk Mar 24, 2024 Collaborator

Replies: 9 comments 35 replies

potiuk
Jun 15, 2022
Collaborator

potiuk Dec 28, 2023
Collaborator

Taragolis Dec 28, 2023
Collaborator

shelly2904
Sep 10, 2022
Author

potiuk Apr 9, 2023
Collaborator

potiuk Apr 9, 2023
Collaborator

potiuk Apr 9, 2023
Collaborator

potiuk Sep 20, 2022
Collaborator

potiuk Sep 20, 2022
Collaborator

potiuk Sep 30, 2022
Collaborator

potiuk Mar 24, 2024
Collaborator