dd-java-trace SO crash in ElfParser::parseProgramHeaders

[g-dx]

Hi,

We have experienced a JVM crash caused by native code in the tracing SO code.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fe2f533a2c3, pid=3936, tid=4737
#
# JRE version: OpenJDK Runtime Environment Corretto-21.0.4.7.1 (21.0.4+7) (build 21.0.4+7-LTS)
# Java VM: OpenJDK 64-Bit Server VM Corretto-21.0.4.7.1 (21.0.4+7-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [libjavaProfiler13745968698683089031.so+0x192c3]  ElfParser::parseProgramHeaders(CodeCache*, char const*, char const*, bool)+0x33
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# JFR recording file will be written. Location: /opt/app/hs_err_pid3936.jfr
#
# An error report file with more information is saved as:
# /opt/app/jvm-crash-3936.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/corretto/corretto-21/issues/
# The crash happened outside the Java Virtual Machine in native code.
# S

Tracer configuration is as follows:

{
  "version": "1.37.1~75639e37b3",
  "os_name": "Linux",
  "os_version": "5.10.220-209.869.amzn2.x86_64",
  "architecture": "amd64",
  "lang": "jvm",
  "lang_version": "21.0.4",
  "jvm_vendor": "Amazon.com Inc.",
  "jvm_version": "21.0.4+7-LTS",
  "java_class_version": "65.0",
  "http_nonProxyHosts": "s3.eu-west-1.amazonaws.com,secretsmanager.eu-west-1.amazonaws.com,ssmmessages.eu-west-1.amazonaws.com,ec2messages.eu-west-1.amazonaws.com,ssm.eu-west-1.amazonaws.com,logs.eu-west-1.amazonaws.com,monitoring.eu-west-1.amazonaws.com,169.254.169.254,169.254.170.2,internal.lb,127.0.0.1,localhost,instance-data,172.17.0.1,docker",
  "http_proxyHost": "proxy",
  "enabled": true,
  "service": "XXXi",
  "agent_url": "http://localhost:8126",
  "agent_error": false,
  "debug": false,
  "trace_propagation_style_extract": [
    "datadog",
    "tracecontext"
  ],
  "trace_propagation_style_inject": [
    "datadog",
    "tracecontext"
  ],
  "analytics_enabled": false,
  "priority_sampling_enabled": true,
  "logs_correlation_enabled": true,
  "profiling_enabled": true,
  "remote_config_enabled": true,
  "debugger_enabled": false,
  "debugger_exception_enabled": false,
  "appsec_enabled": "ENABLED_INACTIVE",
  "telemetry_enabled": true,
  "telemetry_dependency_collection_enabled": true,
  "telemetry_log_collection_enabled": false,
  "dd_version": "2024-07-31.d85d96d6",
  "health_checks_enabled": true,
  "configuration_file": "no config file present",
  "runtime_id": "62199029-9580-4809-a4fc-9d15617e63cd",
  "logging_settings": {
    "levelInBrackets": false,
    "dateTimeFormat": "'[dd.trace 'yyyy-MM-dd HH:mm:ss:SSS Z']'",
    "logFile": "System.err",
    "configurationFile": "simplelogger.properties",
    "showShortLogName": false,
    "showDateTime": true,
    "showLogName": true,
    "showThreadName": true,
    "defaultLogLevel": "INFO",
    "warnLevelString": "WARN",
    "embedException": false
  },
  "cws_enabled": false,
  "cws_tls_refresh": 5000,
  "datadog_profiler_enabled": true,
  "datadog_profiler_safe": true,
  "datadog_profiler_enabled_overridden": false,
  "data_streams_enabled": false
}

The full crash log and JFR files can be viewed in the associated Datadog support ticket here:

https://help.datadoghq.com/hc/en-us/requests/1790594

[r1viollet]

I opened a PR and will get feedback on the proposed approach.
There is also a similar report against async-profiler.

[g-dx]

Great! Thanks for the update.

[github-actions]

🤖 This issue has been addressed in the latest release. See full details in the Release Notes.