segmentation fault - Python 3.10 on Mac M3

[slanton-a]

Environment

Python: 3.10.14
Mac OS: 14.4

Delta-rs version:
0.15.0
Binding:

Environment:

• Cloud provider:
• OS: Mac OS 14.4
• Other:

---

Bug

Importing deltalake results in segmentation fault error
What happened:

1. Start a new env
2. install packages with pip install pyarrow==8.0.0 deltalake==0.15.0 numpy==1.26.4
3. Start a Python shell and type:

from deltalake import DeltaTable, write_deltalake
exit()

5. Once the shell exit you will see the segmentation fault error
   What you expected to happen:
   No error on exit
   How to reproduce it:
   see above
   More details:

Python 3.10.14 (main, Jul 25 2024, 15:47:37) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from deltalake import DeltaTable, write_deltalake
>>> exit()
[1]    17729 segmentation fault  python

[ion-elgreco]

@slanton-a please use the latest version and report back :)

[slanton-a]

@ion-elgreco Same result with 0.18.2

[ion-elgreco]

@slanton-a can you compile from main and then import?

[slanton-a]

@ion-elgreco Not sure I follow, give a step by step and I will try to reproduce

[sherlockbeard]

@slanton-a
short stuff
use pyarrow>=9.0.0 if you want to exit without error (maybe time we bump up pyarrow)

long answer
trigged lldb python with your example
output

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
    frame #0: 0x000000010825ca30 libarrow.800.dylib`Aws::Http::CurlHandleContainer::~CurlHandleContainer() + 52
libarrow.800.dylib`Aws::Http::CurlHandleContainer::~CurlHandleContainer:
->  0x10825ca30 <+52>: ldr    x8, [x8, #0x10]
    0x10825ca34 <+56>: blr    x8
    0x10825ca38 <+60>: cmp    w0, #0x4
    0x10825ca3c <+64>: b.lt   0x10825cb30               ; <+308>

libarrow -> pyarrow
which i believe is from aws/aws-sdk-cpp#1833

[ion-elgreco]

@sherlockbeard thanks for checking!

We can keep the pyarrow at min 8.0.0 though, since most platforms that works fine

I'll close this issue then

[slanton-a]

What is the upper limit for pyarrow? @ion-elgreco

[ion-elgreco]

@slanton-a there is no upper limit

[slanton-a]

@ion-elgreco sorry I meant to ask about a lower limit. Since 8.0.0 is the current one but it's not working. In my large project that has hundreds package, it took me several day to figure out the issue is with deltalake and pyarrow

[ion-elgreco]

@ion-elgreco sorry I meant to ask about a lower limit. Since 8.0.0 is the current one but it's not working. In my large project that has hundreds package, it took me several day to figure out the issue is with deltalake and pyarrow

Lower limit is 8.0.0. But this is simply the requirement for pyarrow code to be compatible with our codebase.

In this case your platform of choice is not working with pyarrow I guess, but that's not something we could resolve

[slanton-a]

Got it, the error indeed seems to come from the line import pyarrow.fs as pa_fs