Segmentation fault creating new scanner in Java

[ejf-ibm]

Describe the bug, including details regarding any error messages, version, and platform.

I'm currently hitting a segfault when trying to read in AWS CUR 2.0 billing files, writing out partitioned arrow files. The segfault is reproducible in our containerized dev environment (k8s deployment), but I have not been able to reproduce is through a synthetic functional test. I'm not able to include the entire heap dump, but the call stack and some relevant information are included below.

mediation-awscloudbilling-58b58d Unhandled exception
mediation-awscloudbilling-58b58d Type=Segmentation error vmState=0x00040000
mediation-awscloudbilling-58b58d J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
mediation-awscloudbilling-58b58d Handler1=00007F4FD22916B0 Handler2=00007F4FD21E6730 InaccessibleAddress=00007F4958F20A2F
mediation-awscloudbilling-58b58d RDI=00007F4958F20A27 RSI=0000000000000002 RAX=3A5987A4DE9A52E2 RBX=00007F4958F209F7
mediation-awscloudbilling-58b58d RCX=3A5987A4DE9A52E2 RDX=0000000000000000 R8=648C56E80D97FBD5 R9=00007F4F6C00A200
mediation-awscloudbilling-58b58d R10=00000000C70F6907 R11=00007F4F6C000020 R12=00007F4F798F3700 R13=00007F4F798F3940
mediation-awscloudbilling-58b58d R14=00007F4F798F3F40 R15=00007F4F798F3700
mediation-awscloudbilling-58b58d RIP=00007F4E9D9D718F GS=0000 FS=0000 RSP=00007F4F798F3590
mediation-awscloudbilling-58b58d EFlags=0000000000010246 CS=0033 RBP=00007F4F798F3940 ERR=0000000000000004
mediation-awscloudbilling-58b58d TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=00007F4958F20A2F
mediation-awscloudbilling-58b58d xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
mediation-awscloudbilling-58b58d xmm1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
mediation-awscloudbilling-58b58d xmm2 00007f4f6c00b570 (f: 1811985792.000000, d: 6.915886e-310)
mediation-awscloudbilling-58b58d xmm3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
mediation-awscloudbilling-58b58d xmm4 00007f4f6c00a220 (f: 1811980800.000000, d: 6.915886e-310)
mediation-awscloudbilling-58b58d xmm5 00007f4f6c006fe0 (f: 1811968000.000000, d: 6.915886e-310)
mediation-awscloudbilling-58b58d xmm6 00007f4eac051650 (f: 2886014464.000000, d: 6.915727e-310)
mediation-awscloudbilling-58b58d xmm7 00007f4eac051650 (f: 2886014464.000000, d: 6.915727e-310)
mediation-awscloudbilling-58b58d xmm8 ffffffac00000000 (f: 0.000000, d: -nan)
mediation-awscloudbilling-58b58d xmm9 ffff00ff000000ff (f: 255.000000, d: -nan)
mediation-awscloudbilling-58b58d xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
mediation-awscloudbilling-58b58d xmm11 2bedb0c3811e1752 (f: 2166232832.000000, d: 4.343789e-97)
mediation-awscloudbilling-58b58d xmm12 0000000000000001 (f: 1.000000, d: 4.940656e-324)
mediation-awscloudbilling-58b58d xmm13 08090a0b0c0d0e0f (f: 202182160.000000, d: 5.924543e-270)
mediation-awscloudbilling-58b58d xmm14 0094257f6bf44044 (f: 1811169408.000000, d: 7.172383e-306)
mediation-awscloudbilling-58b58d xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
mediation-awscloudbilling-58b58d Module=/tmp/jnilib-12024434061044301193.tmp
mediation-awscloudbilling-58b58d Module_base_address=00007F4E9C5ED000 Symbol=_ZNK5arrow7compute16FunctionRegistry20FunctionRegistryImpl11GetFunctionERKSs
mediation-awscloudbilling-58b58d Symbol_address=00007F4E9D9D7160
mediation-awscloudbilling-58b58d Target=2_90_20240416_760 (Linux 5.15.0-125-generic)
mediation-awscloudbilling-58b58d CPU=amd64 (8 logical CPUs) (0xfb2c63000 RAM)
mediation-awscloudbilling-58b58d ----------- Stack Backtrace -----------
mediation-awscloudbilling-58b58d _ZNK5arrow7compute16FunctionRegistry20FunctionRegistryImpl11GetFunctionERKSs+0x2f (0x00007F4E9D9D718F [jnilib-12024434061044301193.tmp+0x13ea18f])
mediation-awscloudbilling-58b58d _ZNK5arrow7compute16FunctionRegistry11GetFunctionERKSs+0xd (0x00007F4E9D9CD91D [jnilib-12024434061044301193.tmp+0x13e091d])
mediation-awscloudbilling-58b58d _ZN5arrow7compute12_GLOBAL__N_116BindNonRecursiveENS0_10Expression4CallEbPNS0_11ExecContextE+0x1a8 (0x00007F4E9D9F8338 [jnilib-12024434061044301193.tmp+0x140b338])
mediation-awscloudbilling-58b58d _ZN5arrow7compute12_GLOBAL__N_18BindImplINS_6SchemaEEENS_6ResultINS0_10ExpressionEEES5_RKT_PNS0_11ExecContextE+0x68f (0x00007F4E9D9F9BAF [jnilib-12024434061044301193.tmp+0x140cbaf])
mediation-awscloudbilling-58b58d _ZN5arrow7compute12_GLOBAL__N_18BindImplINS_6SchemaEEENS_6ResultINS0_10ExpressionEEES5_RKT_PNS0_11ExecContextE+0x64d (0x00007F4E9D9F9B6D [jnilib-12024434061044301193.tmp+0x140cb6d])
mediation-awscloudbilling-58b58d _ZNK5arrow7compute10Expression4BindERKNS_6SchemaEPNS0_11ExecContextE+0x33 (0x00007F4E9D9F9DF3 [jnilib-12024434061044301193.tmp+0x140cdf3])
mediation-awscloudbilling-58b58d _ZN5arrow7dataset15ProjectionDescr20FromStructExpressionERKNS_7compute10ExpressionERKNS_6SchemaE+0x25 (0x00007F4E9D24AB55 [jnilib-12024434061044301193.tmp+0xc5db55])
mediation-awscloudbilling-58b58d _ZN5arrow7dataset15ProjectionDescr15FromExpressionsESt6vectorINS_7compute10ExpressionESaIS4_EES2_ISsSaISsEERKNS_6SchemaE+0x41f (0x00007F4E9D24DFAF [jnilib-12024434061044301193.tmp+0xc60faf])
mediation-awscloudbilling-58b58d _ZN5arrow7dataset15ProjectionDescr9FromNamesESt6vectorISsSaISsEERKNS_6SchemaEb+0x620 (0x00007F4E9D272910 [jnilib-12024434061044301193.tmp+0xc85910])
mediation-awscloudbilling-58b58d _ZN5arrow7dataset14ScannerBuilder7ProjectESt6vectorISsSaISsEE+0x57 (0x00007F4E9D274817 [jnilib-12024434061044301193.tmp+0xc87817])
mediation-awscloudbilling-58b58d Java_org_apache_arrow_dataset_jni_JniWrapper_createScanner+0x208 (0x00007F4E9D22EB28 [jnilib-12024434061044301193.tmp+0xc41b28])
mediation-awscloudbilling-58b58d ffi_call_unix64+0x52 (0x00007F4FD2459D3A [libj9vm29.so+0x20bd3a])
mediation-awscloudbilling-58b58d ffi_call_int+0x1a1 (0x00007F4FD2458ED1 [libj9vm29.so+0x20aed1])
mediation-awscloudbilling-58b58d _ZN37VM_DebugBytecodeInterpreterCompressed3runEP10J9VMThread+0x184be (0x00007F4FD234B52E [libj9vm29.so+0xfd52e])
mediation-awscloudbilling-58b58d debugBytecodeLoopCompressed+0xd2 (0x00007F4FD2333062 [libj9vm29.so+0xe5062])
mediation-awscloudbilling-58b58d  (0x00007F4FD23E6AC2 [libj9vm29.so+0x198ac2])
mediation-awscloudbilling-58b58d ---------------------------------------
mediation-awscloudbilling-58b58d JVMDUMP039I Processing dump event "gpf", detail "" at 2024/11/30 13:54:52 - please wait.
mediation-awscloudbilling-58b58d JVMDUMP032I JVM requested System dump using '//core.20241130.135452.1.0005.dmp' in response to an event
mediation-awscloudbilling-58b58d JVMDUMP030W Cannot write dump to file //core.20241130.135452.1.0005.dmp: Read-only file system
mediation-awscloudbilling-58b58d JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.155.  Review the manual for the external program to find where the core dump is written and ensure the program does not truncate it.
mediation-awscloudbilling-58b58d 
mediation-awscloudbilling-58b58d JVMPORT049I The core file created by child process with pid = 155 was not found. Review the documentation for the /proc/sys/kernel/core_pattern program "|/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" to find where the core file is written and ensure that program does not truncate it.
mediation-awscloudbilling-58b58d 
mediation-awscloudbilling-58b58d JVMDUMP012E Error in System dump: /tmp/core.20241130.135452.1.0005.dmp
mediation-awscloudbilling-58b58d JVMDUMP032I JVM requested Java dump using '/STDOUT/' in response to an event
mediation-awscloudbilling-58b58d 0SECTION       TITLE subcomponent dump routine
mediation-awscloudbilling-58b58d NULL           ===============================
mediation-awscloudbilling-58b58d 1TICHARSET     ANSI_X3.4-1968
mediation-awscloudbilling-58b58d 1TISIGINFO     Dump Event "gpf" (00002000) received
mediation-awscloudbilling-58b58d 1TIDATETIMEUTC Date: 2024/11/30 at 13:54:58:059 (UTC)
mediation-awscloudbilling-58b58d 1TIDATETIME    Date: 2024/11/30 at 13:54:58:059
mediation-awscloudbilling-58b58d 1TITIMEZONE    Timezone: UTC (UTC)
mediation-awscloudbilling-58b58d 1TINANOTIME    System nanotime: 1107936331581339
mediation-awscloudbilling-58b58d 1TIFILENAME    Javacore filename:    /STDOUT/
mediation-awscloudbilling-58b58d 1TIREQFLAGS    Request Flags: 0x41 (exclusive+preempt)
mediation-awscloudbilling-58b58d 1TIPREPSTATE   Prep State: 0x80 (trace_disabled)
mediation-awscloudbilling-58b58d 1TIPREPINFO    Exclusive VM access not taken: data may not be consistent across javacore sections
mediation-awscloudbilling-58b58d NULL           ------------------------------------------------------------------------
mediation-awscloudbilling-58b58d 0SECTION       GPINFO subcomponent dump routine
mediation-awscloudbilling-58b58d NULL           ================================
mediation-awscloudbilling-58b58d 2XHOSLEVEL     OS Level         : Linux 5.15.0-125-generic
mediation-awscloudbilling-58b58d 2XHCPUS        Processors -
mediation-awscloudbilling-58b58d 3XHCPUARCH       Architecture   : amd64
mediation-awscloudbilling-58b58d 3XHNUMCPUS       How Many       : 8
mediation-awscloudbilling-58b58d 3XHNUMASUP       NUMA is either not supported or has been disabled by user
mediation-awscloudbilling-58b58d NULL
mediation-awscloudbilling-58b58d 1XHEXCPCODE    J9Generic_Signal_Number: 00000018
mediation-awscloudbilling-58b58d 1XHEXCPCODE    Signal_Number: 0000000B
mediation-awscloudbilling-58b58d 1XHEXCPCODE    Error_Value: 00000000
mediation-awscloudbilling-58b58d 1XHEXCPCODE    Signal_Code: 00000001
mediation-awscloudbilling-58b58d 1XHEXCPCODE    Handler1: 00007F4FD22916B0
mediation-awscloudbilling-58b58d 1XHEXCPCODE    Handler2: 00007F4FD21E6730
mediation-awscloudbilling-58b58d 1XHEXCPCODE    InaccessibleAddress: 00007F4958F20A2F
mediation-awscloudbilling-58b58d NULL
mediation-awscloudbilling-58b58d 1XHEXCPMODULE  Module: /tmp/jnilib-12024434061044301193.tmp
mediation-awscloudbilling-58b58d 1XHEXCPMODULE  Module_base_address: 00007F4E9C5ED000
mediation-awscloudbilling-58b58d 1XHEXCPMODULE  Symbol: _ZNK5arrow7compute16FunctionRegistry20FunctionRegistryImpl11GetFunctionERKSs
mediation-awscloudbilling-58b58d 1XHEXCPMODULE  Symbol_address: 00007F4E9D9D7160
mediation-awscloudbilling-58b58d NULL
mediation-awscloudbilling-58b58d 1XHREGISTERS   Registers:
mediation-awscloudbilling-58b58d 2XHREGISTER      RDI: 00007F4958F20A27
mediation-awscloudbilling-58b58d 2XHREGISTER      RSI: 0000000000000002
mediation-awscloudbilling-58b58d 2XHREGISTER      RAX: 3A5987A4DE9A52E2
mediation-awscloudbilling-58b58d 2XHREGISTER      RBX: 00007F4958F209F7
mediation-awscloudbilling-58b58d 2XHREGISTER      RCX: 3A5987A4DE9A52E2
mediation-awscloudbilling-58b58d 2XHREGISTER      RDX: 0000000000000000
mediation-awscloudbilling-58b58d 2XHREGISTER      R8: 648C56E80D97FBD5
mediation-awscloudbilling-58b58d 2XHREGISTER      R9: 00007F4F6C00A200
mediation-awscloudbilling-58b58d 2XHREGISTER      R10: 00000000C70F6907
mediation-awscloudbilling-58b58d 2XHREGISTER      R11: 00007F4F6C000020
mediation-awscloudbilling-58b58d 2XHREGISTER      R12: 00007F4F798F3700
mediation-awscloudbilling-58b58d 2XHREGISTER      R13: 00007F4F798F3940
mediation-awscloudbilling-58b58d 2XHREGISTER      R14: 00007F4F798F3F40
mediation-awscloudbilling-58b58d 2XHREGISTER      R15: 00007F4F798F3700
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm0: 0000000000000000
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm1: 0000000000000000
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm2: 00007F4F6C00B570
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm3: 0000000000000000
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm4: 00007F4F6C00A220
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm5: 00007F4F6C006FE0
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm6: 00007F4EAC051650
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm7: 00007F4EAC051650
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm8: FFFFFFAC00000000
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm9: FFFF00FF000000FF
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm10: 0000000000000000
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm11: 2BEDB0C3811E1752
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm12: 0000000000000001
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm13: 08090A0B0C0D0E0F
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm14: 0094257F6BF44044
mediation-awscloudbilling-58b58d 2XHREGISTER      xmm15: 0000000000000000
mediation-awscloudbilling-58b58d 2XHREGISTER      RIP: 00007F4E9D9D718F
mediation-awscloudbilling-58b58d 2XHREGISTER      GS: 0000
mediation-awscloudbilling-58b58d 2XHREGISTER      FS: 0000
mediation-awscloudbilling-58b58d 2XHREGISTER      RSP: 00007F4F798F3590
mediation-awscloudbilling-58b58d 2XHREGISTER      EFlags: 0000000000010246
mediation-awscloudbilling-58b58d 2XHREGISTER      CS: 0033
mediation-awscloudbilling-58b58d 2XHREGISTER      RBP: 00007F4F798F3940
mediation-awscloudbilling-58b58d 2XHREGISTER      ERR: 0000000000000004
mediation-awscloudbilling-58b58d 2XHREGISTER      TRAPNO: 000000000000000E
mediation-awscloudbilling-58b58d 2XHREGISTER      OLDMASK: 0000000000000000
mediation-awscloudbilling-58b58d 2XHREGISTER      CR2: 00007F4958F20A2F
mediation-awscloudbilling-58b58d NULL
mediation-awscloudbilling-58b58d 1XHFLAGS       VM flags:0000000000040000
mediation-awscloudbilling-58b58d NULL
mediation-awscloudbilling-58b58d NULL           ------------------------------------------------------------------------
mediation-awscloudbilling-58b58d 0SECTION       ENVINFO subcomponent dump routine
mediation-awscloudbilling-58b58d NULL           =================================
mediation-awscloudbilling-58b58d 1CIJAVAVERSION JRE 17 Linux amd64-64 (build 17.0.11+9)
mediation-awscloudbilling-58b58d 1CIVMVERSION   20240416_760
mediation-awscloudbilling-58b58d 1CIJ9VMTAG     openj9-0.44.0
mediation-awscloudbilling-58b58d 1CIJ9VMVERSION b0699311c7
mediation-awscloudbilling-58b58d 1CIJITVERSION  j9jit_20240522_2205_
mediation-awscloudbilling-58b58d 1CIOMRVERSION  254af5a04_CMPRSS
mediation-awscloudbilling-58b58d 1CIJCLVERSION  5d7d758b682 based on jdk-17.0.11+9
mediation-awscloudbilling-58b58d 1CIVENDOR      IBM Corporatio
mediation-awscloudbilling-58b58d 1CIPRODUCT     IBM Semeru Runtime Open Edition
mediation-awscloudbilling-58b58d 1CIEXTVERSION  17.0.11.0
mediation-awscloudbilling-58b58d 1CIJITMODES    JIT enabled, AOT enabled, FSD enabled, HCR enabled
mediation-awscloudbilling-58b58d 1CIRUNNINGAS   Running as an embedded JVM
mediation-awscloudbilling-58b58d 1CIVMIDLESTATE VM Idle State: ACTIVE
mediation-awscloudbilling-58b58d 1CICONTINFO    Running in container : TRUE
mediation-awscloudbilling-58b58d 1CICGRPINFO    JVM support for cgroups enabled : TRUE
mediation-awscloudbilling-58b58d 1CISTARTTIME   JVM start time: 2024/11/30 at 13:50:50:425
mediation-awscloudbilling-58b58d 1CISTARTNANO   JVM start nanotime: 1107688697153833
mediation-awscloudbilling-58b58d 1CIPROCESSID   Process ID: 1 (0x1)

Arrow artifacts:

• org.apache.arrow:arrow-vector:18.0.0
• org.apache.arrow:arrow-memory-core:18.0.0
• org.apache.arrow:arrow-memory-unsafe:18.0.0
• org.apache.arrow:arrow-dataset:18.0.0
• org.apache.arrow:arrow-compression:18.0.0

Any suggestions on debug options would be very much appreciated. Thanks!

Component(s)

Java