Wrong build recipe generate a crash in OtlpHttpExporterFactory::Create

[nleclercq]

We have a c++ app offering the ability to choose between HTTP and gRPC to export its traces. It is consequently linked against opentelemetry-cpp::trace, opentelemetry-cpp::otlp_http_exporter and opentelemetry-cpp::otlp_grpc_exporter.

We observe a crash in opentelemetry::exporter::otlp::OtlpHttpExporterFactory::Create when using a HTTP endpoint (see callstack below). There's no problem with a gRPC one.

If we disable gPRC (at compile time) so that the application is no longer linked against opentelemetry-cpp::otlp_grpc_exporter, everything works fine and we can successfully instantiated our HTTP span exporter.

So, it seems that the opentelemetry-cpp::otlp_http_exporter and opentelemetry-cpp::otlp_grpc_exporter libraries don't coexist very well.

Sorry if I missed a piece of documentation saying that what we try to achieve is not supported.

libc.so.6!___pthread_rwlock_wrlock(pthread_rwlock_t * rwlock) (pthread_rwlock_wrlock.c:26)
libcrypto.so!CRYPTO_STATIC_MUTEX_lock_write (Unknown Source:0)
libcrypto.so!CRYPTO_get_ex_new_index (Unknown Source:0)
libcurl.so.4![Unknown/Just-In-Time compiled code] (Unknown Source:0)
libopentelemetry_http_client_curl.so!opentelemetry::v1::ext::http::client::curl::HttpCurlGlobalInitializer::HttpCurlGlobalInitializer() (Unknown Source:0)
libopentelemetry_http_client_curl.so!opentelemetry::v1::ext::http::client::curl::HttpCurlGlobalInitializer::GetInstance() (Unknown Source:0)
libopentelemetry_http_client_curl.so!opentelemetry::v1::ext::http::client::curl::HttpClient::HttpClient() (Unknown Source:0)
libopentelemetry_http_client_curl.so!void __gnu_cxx::new_allocator<opentelemetry::v1::ext::http::client::curl::HttpClient>::construct<opentelemetry::v1::ext::http::client::curl::HttpClient>(opentelemetry::v1::ext::http::client::curl::HttpClient*) (Unknown Source:0)
libopentelemetry_http_client_curl.so!void std::allocator_traits<std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient> >::construct<opentelemetry::v1::ext::http::client::curl::HttpClient>(std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient>&, opentelemetry::v1::ext::http::client::curl::HttpClient*) (Unknown Source:0)
libopentelemetry_http_client_curl.so!std::_Sp_counted_ptr_inplace<opentelemetry::v1::ext::http::client::curl::HttpClient, std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<>(std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient>) (Unknown Source:0)
libopentelemetry_http_client_curl.so!std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<opentelemetry::v1::ext::http::client::curl::HttpClient, std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient>>(opentelemetry::v1::ext::http::client::curl::HttpClient*&, std::_Sp_alloc_shared_tag<std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient> >) (Unknown Source:0)
libopentelemetry_http_client_curl.so!std::__shared_ptr<opentelemetry::v1::ext::http::client::curl::HttpClient, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient>>(std::_Sp_alloc_shared_tag<std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient> >) (Unknown Source:0)
libopentelemetry_http_client_curl.so!std::shared_ptr<opentelemetry::v1::ext::http::client::curl::HttpClient>::shared_ptr<std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient>>(std::_Sp_alloc_shared_tag<std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient> >) (Unknown Source:0)
libopentelemetry_http_client_curl.so!std::shared_ptr<opentelemetry::v1::ext::http::client::curl::HttpClient> std::allocate_shared<opentelemetry::v1::ext::http::client::curl::HttpClient, std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient>>(std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient> const&) (Unknown Source:0)
libopentelemetry_http_client_curl.so!std::shared_ptr<opentelemetry::v1::ext::http::client::curl::HttpClient> std::make_shared<opentelemetry::v1::ext::http::client::curl::HttpClient>() (Unknown Source:0)
libopentelemetry_http_client_curl.so!opentelemetry::v1::ext::http::client::HttpClientFactory::Create() (Unknown Source:0)
libopentelemetry_exporter_otlp_http_client.so!opentelemetry::v1::exporter::otlp::OtlpHttpClient::OtlpHttpClient(opentelemetry::v1::exporter::otlp::OtlpHttpClientOptions&&) (Unknown Source:0)
libopentelemetry_exporter_otlp_http.so!opentelemetry::v1::exporter::otlp::OtlpHttpExporter::OtlpHttpExporter(opentelemetry::v1::exporter::otlp::OtlpHttpExporterOptions const&) (Unknown Source:0)
libopentelemetry_exporter_otlp_http.so!opentelemetry::v1::exporter::otlp::OtlpHttpExporterFactory::Create(opentelemetry::v1::exporter::otlp::OtlpHttpExporterOptions const&) (Unknown Source:0)
libtango.so.10!Tango::telemetry::Interface::Implementation::init_trace_provider(Tango::telemetry::Interface::Implementation * const this) (/home/nicolas/projects/cpptango@gitlab/src/common/telemetry.cpp:370)

[owent]

Could you please provide the version of opentelemetry-cpp, the build system and CMakeCache.txt if you build opentelemetry-cpp with cmake?
We use both OTLP gRPC exporters and OTLP HTTP exporters for a long time, and we didn't find any conflict before.

[nleclercq]

Well, we are using the main - which is certainly not a good thing. Let me give the v1.13.0 a try.
The recipe we use to compile otel-cpp is there.
BTW, do you have an official version of both gPRC, protobuf and libcurl?

[owent]

Well, we are using the main - which is certainly not a good thing. Let me give the v1.13.0 a try. The recipe we use to compile otel-cpp is there. BTW, do you have an official version of both gPRC, protobuf and libcurl?

Yes, we use otel-cpp 1.11.0, gRPC v1.58.1, protobuf v24.2, curl 8.2.1 and openssl 3.1.2 with quic. We plan to upgrade all dependencies recently.
The default options of gRPC will use it's bundle protobuf, could you please compile protobuf before gRPC and use -DgRPC_PROTOBUF_PACKAGE_TYPE=CONFIG -DgRPC_PROTOBUF_PROVIDER=package -DCMAKE_PREFIX_PATH=<Where you install protobuf> to tell gRPC to use a custom version of protobuf?-DCMAKE_PREFIX_PATH=<Where you install protobuf> can also be use to tell otel-cpp where to find protobuf.

[nleclercq]

Ok. The problem is fixed. Thanks for you help. I let you close this ticket.

For the record, here is the recipe that works for me:

ENV

export OTEL_INSTALL_PATH=~/otel-cpp-1.11
export PATH=$OTEL_INSTALL_PATH/bin:$PATH
export LD_LIBRARY_PATH=$OTEL_INSTALL_PATH/lib:$LD_LIBRARY_PATH
export NCPUS=8

CURL

using local Ubuntu-22.04 installation: curl:amd64/jammy-security 7.81.0-1ubuntu1.15

OPENSSL

using local Ubuntu-22.04 installation: openssl:amd64/jammy-security 3.0.2-0ubuntu1.12

ABSEIL

curl -LO https://github.com/abseil/abseil-cpp/archive/refs/tags/20230802.1.tar.gz
tar -xzf 20230802.1.tar.gz
cd abseil-cpp-20230802.1
mkdir build; cd build
cmake -DBUILD_TESTING=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$OTEL_INSTALL_PATH -DABSL_ENABLE_INSTALL=ON -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_SHARED_LIBS=ON -DABSL_PROPAGATE_CXX_STD=ON .. 
make -j $NCPUS install

PROTOBUF

curl -LO https://github.com/protocolbuffers/protobuf/archive/refs/tags/v24.3.tar.gz
tar -xzf ./v24.3.tar.gz
cd protobuf-24.3
mkdir build; cd build
cmake -DCMAKE_INSTALL_PREFIX=$OTEL_INSTALL_PATH -DCMAKE_PREFIX_PATH=$OTEL_INSTALL_PATH -DCMAKE_VERBOSE_MAKEFILE=TRUE -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_SHARED_LIBS=ON -Dprotobuf_ABSL_PROVIDER=package -DCMAKE_CXX_STANDARD=17 -Dprotobuf_BUILD_TESTS=OFF ..
make -j $NCPUS install

gRPC

curl -LO https://github.com/grpc/grpc/archive/refs/tags/v1.58.2.tar.gz
tar -xzf ./v24.3.tar.gz
cd grpc-1.58.2
mkdir build; cd build
cmake -DCMAKE_INSTALL_PREFIX=$OTEL_INSTALL_PATH -DCMAKE_PREFIX_PATH=$OTEL_INSTALL_PATH -DCMAKE_VERBOSE_MAKEFILE=TRUE -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=$OTEL_INSTALL_PATH -DCMAKE_PREFIX_PATH=$OTEL_INSTALL_PATH -DCMAKE_VERBOSE_MAKEFILE=TRUE -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_SHARED_LIBS=ON -DgRPC_ABSL_PROVIDER=package -DgRPC_CARES_PROVIDER=package -DgRPC_PROTOBUF_PROVIDER=package -DgRPC_RE2_PROVIDER=package -DgRPC_SSL_PROVIDER=package -DgRPC_ZLIB_PROVIDER=package ..
make -j $NCPUS install

OPENTELEMETRY

curl -LO https://github.com/open-telemetry/opentelemetry-cpp/archive/refs/tags/v1.11.0.tar.gz
tar -xzf v1.11.0.tar.gz
cd opentelemetry-cpp-1.11.0
mkdir build; cd build
cmake -DCMAKE_INSTALL_PREFIX=$OTEL_INSTALL_PATH -DCMAKE_PREFIX_PATH=$OTEL_INSTALL_PATH -DCMAKE_VERBOSE_MAKEFILE=TRUE -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_SHARED_LIBS=ON -DWITH_OTLP_GRPC=ON -DWITH_OTLP_HTTP=ON -DWITH_ABSEIL=ON -DBUILD_TESTING=OFF -DWITH_BENCHMARK=OFF ..
make -j $NCPUS install

[marcalff]

Closing as can not reproduce. Thanks for the details.