Segmentation Fault when cross-compiling for Raspberry PI

[davyuan]

Environment

Dev machine: Ubuntu 18.04
Linux DL1 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Cross Compiler: g++ 4.9.3
arm-linux-gnueabihf-g++ --version
arm-linux-gnueabihf-g++ (crosstool-NG crosstool-ng-1.22.0-88-g8460611) 4.9.3

Target machine: RPI3B+
Linux RPI3 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux

Source code from Github, tip of the tree.

Description

1. Compile the cpprestsdk source on RPI with samples.
2. Run the sample BingRequest and make sure everything is working. The sample is just making a call to bing.com and fetch the query results and save to a local file.
3. Copy the libcpprest.so and libcpprest.so.2.10 to the cross compile sand-box on the Ubuntu Dev machine.
4. Create a new test project with cmake, and add the same BingRequest source code to it.
5. Cross Compile it and there is no issue with compiling.
6. Copy the executable back to the RPI and run it.

Expected result

It runs

Actual result

Segmentation Fault.

Debug

Run the executable in gdb and check the callstack

pi@RPI3:~/dev/Slideshow/build $ gdb --args ./testRest hello hello.txt
GNU gdb (Raspbian 7.7.1+dfsg-5+rpi1) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./testRest...(no debugging symbols found)...done.
(gdb) handle SIGILL nostop noprint
Signal Stop Print Pass to program Description
SIGILL No No Yes Illegal instruction
(gdb) run
Starting program: /home/pi/dev/build/testRest hello hello.txt
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[New Thread 0x7479d450 (LWP 7214)]
[New Thread 0x73f9d450 (LWP 7215)]
[New Thread 0x7379d450 (LWP 7216)]
[New Thread 0x72f9d450 (LWP 7217)]
[New Thread 0x7279d450 (LWP 7218)]
[New Thread 0x71f9d450 (LWP 7219)]
[New Thread 0x7179d450 (LWP 7220)]
[New Thread 0x70f9d450 (LWP 7221)]
[New Thread 0x7079d450 (LWP 7222)]
[New Thread 0x6ff9d450 (LWP 7223)]
[New Thread 0x6f79d450 (LWP 7224)]
[New Thread 0x6ef9d450 (LWP 7225)]
[New Thread 0x6e79d450 (LWP 7226)]
[New Thread 0x6df9d450 (LWP 7227)]
[New Thread 0x6d79d450 (LWP 7228)]
[New Thread 0x6cf9d450 (LWP 7229)]
[New Thread 0x6c79d450 (LWP 7230)]
[New Thread 0x6bf9d450 (LWP 7231)]
[New Thread 0x6b79d450 (LWP 7232)]
[New Thread 0x6af9d450 (LWP 7233)]
[New Thread 0x6a79d450 (LWP 7234)]
[New Thread 0x69f9d450 (LWP 7235)]
[New Thread 0x6979d450 (LWP 7236)]
[New Thread 0x68f9d450 (LWP 7237)]
[New Thread 0x6879d450 (LWP 7238)]
[New Thread 0x67f9d450 (LWP 7239)]
[New Thread 0x6779d450 (LWP 7240)]
[New Thread 0x66f9d450 (LWP 7241)]
[New Thread 0x6679d450 (LWP 7242)]
[New Thread 0x65f9d450 (LWP 7243)]
[New Thread 0x6579d450 (LWP 7244)]
[New Thread 0x64f9d450 (LWP 7245)]
[New Thread 0x6479d450 (LWP 7246)]
[New Thread 0x63f9d450 (LWP 7247)]
[New Thread 0x6379d450 (LWP 7248)]
[New Thread 0x62f9d450 (LWP 7249)]
[New Thread 0x6279d450 (LWP 7250)]
[New Thread 0x61f9d450 (LWP 7251)]
[New Thread 0x6179d450 (LWP 7252)]
[New Thread 0x60f9d450 (LWP 7253)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x61f9d450 (LWP 7251)]
0x76adfda8 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)1>::_M_release()
() from /usr/local/lib/libcpprest.so.2.10
(gdb) backtrace
#0 0x76adfda8 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)1>::_M_release() () from /usr/local/lib/libcpprest.so.2.10
#1 0x76adaf20 in std::__shared_count<(__gnu_cxx::_Lock_policy)1>::__shared_count() () from /usr/local/lib/libcpprest.so.2.10
#2 0x76ad6740 in std::__shared_ptr<Concurrency::streams::details::basic_istream_helper, (__gnu_cxx::_Lock_policy)1>::__shared_ptr() ()
from /usr/local/lib/libcpprest.so.2.10
#3 0x76ad676c in std::shared_ptr<Concurrency::streams::details::basic_istream_helper >::~shared_ptr() () from /usr/local/lib/libcpprest.so.2.10
#4 0x76ad6798 in Concurrency::streams::basic_istream::~basic_istream() () from /usr/local/lib/libcpprest.so.2.10
#5 0x76b02650 in web::http::details::http_msg_base::_prepare_to_receive_data()
() from /usr/local/lib/libcpprest.so.2.10
#6 0x76acd98c in web::http::client::details::request_context::request_context(std::shared_ptrweb::http::client::details::_http_client_communicator const&, web::http::http_request const&) () from /usr/local/lib/libcpprest.so.2.10
#7 0x76bc7208 in web::http::client::details::asio_context::asio_context(std::shared_ptrweb::http::client::details::_http_client_communicator const&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection const&) () from /usr/local/lib/libcpprest.so.2.10
#8 0x76c104ac in void __gnu_cxx::new_allocatorweb::http::client::details::asio_context::construct<web::http::client::details::asio_context, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&>(web::http::client::details::asio_context*, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&) () from /usr/local/lib/libcpprest.so.2.10
#9 0x76c0d8a0 in std::enable_if<std::_and<std::allocator_traits<std::allocatorweb::http::client::details::asio_context >::__construct_helper<web::http::client::details::asio_context, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&>::type>::value, void>::type std::allocator_traits<std::allocatorweb::http::client::details::asio_context >::_S_construct<web::http::client::details::asio_context, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&>(std::allocatorweb::http::client::details::asio_context&, web::http::client::details::asio_context*, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&) ()
from /usr/local/lib/libcpprest.so.2.10
#10 0x76c0a8cc in decltype (_S_construct({parm#1}, {parm#2}, (forward<std::shared_ptrweb::http::client::details::_http_client_communicator&>)({parm#3}), (forwardweb::http::http_request&)({parm#3}), (forward<std::shared_ptr<web::http::cl---Type to continue, or q to quit---
ient::details::asio_connection>&>)({parm#3}))) std::allocator_traits<std::allocatorweb::http::client::details::asio_context >::construct<web::http::client::details::asio_context, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&>(std::allocatorweb::http::client::details::asio_context&, web::http::client::details::asio_context*, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&) ()
from /usr/local/lib/libcpprest.so.2.10
#11 0x76c06844 in std::_Sp_counted_ptr_inplace<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)1>::_Sp_counted_ptr_inplace<std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&>(std::allocatorweb::http::client::details::asio_context, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&) () from /usr/local/lib/libcpprest.so.2.10
#12 0x76c03020 in void __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)1> >::construct<std::_Sp_counted_ptr_inplace<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)1>, std::allocatorweb::http::client::details::asio_context const, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&>(std::_Sp_counted_ptr_inplace<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)1>, std::allocatorweb::http::client::details::asio_context const&&, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&) () from /usr/local/lib/libcpprest.so.2.10
#13 0x76bfde64 in std::enable_if<std::_and<std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)1> > >::__construct_helper<std::_Sp_counted_ptr_inplace<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)1>, std::allocatorweb::http::client::details::asio_context const, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&>::type>::value, void>::type std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)1> > >::_S_construct<std::_Sp_counted_ptr_inplace<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)---Type to continue, or q to quit---
1>, std::allocatorweb::http::client::details::asio_context const, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&>(std::allocator<std::_Sp_counted_ptr_inplace<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)1> >&, std::_Sp_counted_ptr_inplace<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)1>, std::allocatorweb::http::client::details::asio_context const&&, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&) ()
from /usr/local/lib/libcpprest.so.2.10
#14 0x76bf9098 in decltype (_S_construct({parm#1}, {parm#2}, (forward<std::allocatorweb::http::client::details::asio_context const>)({parm#3}), (forward<std::shared_ptrweb::http::client::details::_http_client_communicator&>)({parm#3}), (forwardweb::http::http_request&)({parm#3}), (forward<std::shared_ptrweb::http::client::details::asio_connection&>)({parm#3}))) std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)1> > >::construct<std::_Sp_counted_ptr_inplace<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)1>, std::allocatorweb::http::client::details::asio_context const, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&>(std::allocator<std::_Sp_counted_ptr_inplace<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)1> >&, std::_Sp_counted_ptr_inplace<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)1>, std::allocatorweb::http::client::details::asio_context const&&, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&) () from /usr/local/lib/libcpprest.so.2.10
#15 0x76bf3acc in std::__shared_count<(__gnu_cxx::_Lock_policy)1>::__shared_count<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&>(std::_Sp_make_shared_tag, web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context const&, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&)
() from /usr/local/lib/libcpprest.so.2.10
#16 0x76bece40 in std::__shared_ptr<web::http::client::details::asio_context, (__gnu_cxx::_Lock_policy)1>::__shared_ptr<std::allocatorweb::http::client::details::asio_context, std::shared_ptr<web::http::client::details::_http_client_commu---Type to continue, or q to quit---
nicator>&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&>(std::_Sp_make_shared_tag, std::allocatorweb::http::client::details::asio_context const&, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&) () from /usr/local/lib/libcpprest.so.2.10
#17 0x76be5044 in std::shared_ptrweb::http::client::details::asio_context::shared_ptr<std::allocatorweb::http::client::details::asio_context, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&>(std::_Sp_make_shared_tag, std::allocatorweb::http::client::details::asio_context const&, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&) () from /usr/local/lib/libcpprest.so.2.10
#18 0x76bdc3d4 in std::shared_ptrweb::http::client::details::asio_context std::allocate_shared<web::http::client::details::asio_context, std::allocatorweb::http::client::details::asio_context, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&>(std::allocatorweb::http::client::details::asio_context const&, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&) () from /usr/local/lib/libcpprest.so.2.10
#19 0x76bd4720 in std::shared_ptrweb::http::client::details::asio_context std::make_shared<web::http::client::details::asio_context, std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&>(std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&, std::shared_ptrweb::http::client::details::asio_connection&) ()
from /usr/local/lib/libcpprest.so.2.10
#20 0x76bc75d8 in web::http::client::details::asio_context::create_request_context(std::shared_ptrweb::http::client::details::_http_client_communicator&, web::http::http_request&) () from /usr/local/lib/libcpprest.so.2.10
#21 0x76bbb7c0 in web::http::client::details::asio_client::propagate(web::http::http_request) () from /usr/local/lib/libcpprest.so.2.10
#22 0x76ad8100 in web::http::oauth2::details::oauth2_handler::propagate(web::http::http_request) () from /usr/local/lib/libcpprest.so.2.10
#23 0x76ad7d08 in web::http::oauth1::details::oauth1_handler::propagate(web::http::http_request) () from /usr/local/lib/libcpprest.so.2.10
#24 0x76ada408 in web::http::client::http_pipeline::propagate(web::http::http_request) () from /usr/local/lib/libcpprest.so.2.10
#25 0x76ace7f4 in web::http::client::http_client::request(web::http::http_request, pplx::cancellation_token const&) () from /usr/local/lib/libcpprest.so.2.10
#26 0x0006b8f0 in main::{lambda(Concurrency::streams::streambuf)#1}::operator()(Concurrency::streams::streambuf) const ()
---Type to continue, or q to quit---
#27 0x0006bc20 in std::_Function_handler<pplx::taskweb::http::http_response (Concurrency::streams::streambuf), main::{lambda(Concurrency::streams::streambuf)#1}>::_M_invoke(std::_Any_data const&, Concurrency::streams::streambuf) ()
#28 0x0006acc8 in pplx::details::_PPLTaskHandle<web::http::http_response, pplx::task<Concurrency::streams::streambuf >::_ContinuationTaskHandle<Concurrency::streams::streambuf, web::http::http_response, main::{lambda(Concurrency::streams::streambuf)#1}, std::integral_constant<bool, false>, pplx::details::_TypeSelectorAsyncTask>, pplx::details::_ContinuationTaskHandleBase>::invoke() const ()
#29 0x0006cd68 in pplx::details::_TaskProcHandle::_RunChoreBridge(void*) ()
#30 0x76bb33cc in void boost::_bi::list1<boost::_bi::value<void*> >::operator()<void ()(void), boost::_bi::list0>(boost::_bi::type, void (&)(void), boost::_bi::list0&, int) () from /usr/local/lib/libcpprest.so.2.10
#31 0x76bb3374 in boost::_bi::bind_t<void, void ()(void), boost::_bi::list1<boost::_bi::value<void*> > >::operator()() ()
from /usr/local/lib/libcpprest.so.2.10
#32 0x76bb3320 in void boost::asio::asio_handler_invoke<boost::_bi::bind_t<void, void ()(void), boost::_bi::list1<boost::_bi::value<void*> > > >(boost::_bi::bind_t<void, void ()(void), boost::_bi::list1<boost::_bi::value<void*> > >&, ...) () from /usr/local/lib/libcpprest.so.2.10
#33 0x76bb3304 in void boost_asio_handler_invoke_helpers::invoke<boost::_bi::bind_t<void, void ()(void), boost::_bi::list1<boost::_bi::value<void*> > >, boost::_bi::bind_t<void, void ()(void), boost::_bi::list1<boost::_bi::value<void*> > > >(boost::_bi::bind_t<void, void ()(void), boost::_bi::list1<boost::_bi::value<void*> > >&, boost::_bi::bind_t<void, void ()(void), boost::_bi::list1<boost::_bi::value<void*> > >&) () from /usr/local/lib/libcpprest.so.2.10
#34 0x76bb3258 in boost::asio::detail::completion_handler<boost::_bi::bind_t<void, void ()(void), boost::_bi::list1<boost::_bi::value<void*> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned int) ()
from /usr/local/lib/libcpprest.so.2.10
#35 0x76f46a50 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /usr/lib/arm-linux-gnueabihf/libboost_log.so.1.55.0
#36 0x76bb7d68 in boost::asio::io_service::run() ()
from /usr/local/lib/libcpprest.so.2.10
#37 0x76bb5878 in (anonymous namespace)::threadpool_impl::thread_start(void*)
() from /usr/local/lib/libcpprest.so.2.10
#38 0x76bb57b8 in (anonymous namespace)::threadpool_impl::add_thread()::{lambda()#1}::operator()() const () from /usr/local/lib/libcpprest.so.2.10
#39 0x76bb6710 in boost::asio::detail::posix_thread::func<(anonymous namespace)::threadpool_impl::add_thread()::{lambda()#1}>::run() ()
from /usr/local/lib/libcpprest.so.2.10
---Type to continue, or q to quit---
#40 0x76f476bc in boost_asio_detail_posix_thread_function ()
from /usr/lib/arm-linux-gnueabihf/libboost_log.so.1.55.0
#41 0x7655ae90 in start_thread (arg=0x61f9d450) at pthread_create.c:311
#42 0x763db598 in ?? ()
at ../ports/sysdeps/unix/sysv/linux/arm/nptl/../clone.S:92
from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[davyuan]

It turned out to be a compiler flag I'm using. -mcpu=cortex-a53

After I remove it, the compiled executable starts to work.

Closing the issue and sharing the findings.