Aws::Crt::Io::ClientBootstrap destructor may launch thread at process exit, and crash

[pitrou]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though Developer Guide and API reference
• I've searched for previous similar issues and didn't find any solution

Describe the bug
We're getting a user report of a crash, seemingly at process shutdown, on Windows:
conda-forge/arrow-cpp-feedstock#567

Apparently the ClientBootstrap destructor can indirectly trigger the launch of a new thread using aws_thread_launch. The thread launch fails at process shutdown, at least on Windows, triggering an assertion error and therefore a process crash.

SDK version number
1.9.120

Platform/OS/Hardware/Device
Windows/10.0.17763
(also reported on CentOS 8 and Ubuntu: https://issues.apache.org/jira/browse/ARROW-15141)

To Reproduce (observed behavior)
Basically conda-forge/arrow-cpp-feedstock#567 (comment), but I'm not sure what the exact steps are (I'm not the original reporter).

Expected behavior
Failing to launch a thread at process shutdown should probably not crash the process.

[pitrou]

@xhochy

[kylekeppler]

Does not happen with SDK version 1.8.186.

[jdblischak]

In case it could help narrow down the source of the bug, I tested a few different versions of aws-sdk-cpp on CentOS 7:

• No issue: 1.1.186
• Bug: 1.9.120, 1.9.140

[KaibaLopez]

Hi @jdblischak ,
Can you share how you are reproducing this?
In the post linked by the op there seems to be a fix provided by conda-forge so I'm wondering if this is on their side rather than the sdk?

[xhochy]

The fix on the conda-forge side was to revert back to the 1.8.186 SDK version. With the current issue, we cannot use a newer SDK on Windows.

[jdblischak]

@KaibaLopez Thanks for following up

In the post linked by the op there seems to be a fix provided by conda-forge so I'm wondering if this is on their side rather than the sdk?

As @xhochy commented, the conda-forge workaround is to pin to an older version of aws-sdk-cpp. Personally I fixed it by specifying aws-sdk-cpp=1.8.186=h9ad65fb_2 for my conda env.

Can you share how you are reproducing this?

I was able to reproduce the bug using the code below:

mamba create -n test-aws python=3.9 pandas=1.2 pyarrow=2.0 aws-sdk-cpp=1.9.120
conda activate test-aws
python test-arrow.py

where test-arrow.py is the reproducible example script copied from conda-forge/arrow-cpp-feedstock#567

import numpy as np
import pandas as pd

def test_error():

    df = pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))

    df.to_parquet('test.parquet')

if __name__ == '__main__':
    test_error()

Here is the full error and traceback that I observe:

% python test-arrow.py
Fatal error condition occurred in /home/conda/feedstock_root/build_artifacts/aws-c-io_1633633131324/work/source/event_loop.c:72: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
################################################################################
Stack trace:
################################################################################
~/mambaforge/envs/test-aws/lib/python3.9/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_backtrace_print+0x59) [0x2aaac1581f19]
~/mambaforge/envs/test-aws/lib/python3.9/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_fatal_assert+0x48) [0x2aaac1573098]
~/mambaforge/envs/test-aws/lib/python3.9/site-packages/pyarrow/../../.././././libaws-c-io.so.1.0.0(+0x10a43) [0x2aaac17bca43]
~/mambaforge/envs/test-aws/lib/python3.9/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_ref_count_release+0x1d) [0x2aaac1583fad]
~/mambaforge/envs/test-aws/lib/python3.9/site-packages/pyarrow/../../.././././libaws-c-io.so.1.0.0(+0xe35a) [0x2aaac17ba35a]
~/mambaforge/envs/test-aws/lib/python3.9/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_ref_count_release+0x1d) [0x2aaac1583fad]
~/mambaforge/envs/test-aws/lib/python3.9/site-packages/pyarrow/../../../././libaws-crt-cpp.so(_ZN3Aws3Crt2Io15ClientBootstrapD1Ev+0x3a) [0x2aaac1526f5a]
~/mambaforge/envs/test-aws/lib/python3.9/site-packages/pyarrow/../../.././libaws-cpp-sdk-core.so(+0x5f570) [0x2aaac0faa570]
/lib64/libc.so.6(+0x39c99) [0x2aaaab835c99]
/lib64/libc.so.6(+0x39ce7) [0x2aaaab835ce7]
/lib64/libc.so.6(__libc_start_main+0xfc) [0x2aaaab81e50c]
python(+0x20aa51) [0x55555575ea51]
Aborted

[pitrou]

Update: this error has been reported on Ubuntu and CentOS as well: https://issues.apache.org/jira/browse/ARROW-15141