Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[docdb] coredump in YSQL webserver #7683

Closed
bmatican opened this issue Mar 16, 2021 · 3 comments
Closed

[docdb] coredump in YSQL webserver #7683

bmatican opened this issue Mar 16, 2021 · 3 comments
Assignees
@es1024
Labels
area/docdb YugabyteDB core features area/ysql Yugabyte SQL (YSQL) kind/bug This issue is a bug priority/low Low priority

Comments

@bmatican
Copy link
Contributor

bmatican commented Mar 16, 2021 •
edited by yugabyte-ci
Loading

Jira Link: DB-1734
Not much to go on here, these were all the 3 threads in the coredump:

Thread 3 (LWP 982):
#0  0x00007f8a8e3065cd in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f8a8843d225 in master_thread () from /home/yugabyte/yb-software/yugabyte-2.4.2.0-b12-centos-x86_64/lib/yb/libserver_process.so
#2  0x00007f8a8ebd2694 in start_thread (arg=0x7f8a77a0e700) at pthread_create.c:333
#3  0x00007f8a8e30f41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (LWP 981):
#0  0x00007f8a911fec70 in ?? () from /home/yugabyte/yb-software/yugabyte-2.4.2.0-b12-centos-x86_64/linuxbrew/lib/libatomic.so.1
#1  0x00007f8a9141632a in _dl_fini () at dl-fini.c:235
#2  0x00007f8a8e25e969 in __run_exit_handlers (status=status@entry=0, listp=0x7f8a8e5bf5c0 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#3  0x00007f8a8e25e9b5 in __GI_exit (status=status@entry=0) at exit.c:104
#4  0x000000000084021c in proc_exit (code=code@entry=0) at ../../../../../../../src/postgres/src/backend/storage/ipc/ipc.c:156
#5  0x00007f8a8169c7fa in webserver_worker_main (unused=<optimized out>) at ../../../../../src/postgres/contrib/yb_pg_metrics/yb_pg_metrics.c:369
#6  0x00000000007c45fd in StartBackgroundWorker () at ../../../../../../src/postgres/src/backend/postmaster/bgworker.c:834
#7  0x00000000007d3ff4 in do_start_bgworker (rw=<optimized out>) at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:5774
#8  maybe_start_bgworkers () at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:5987
#9  0x00000000007d6a4c in PostmasterMain (argc=argc@entry=25, argv=argv@entry=0x1b66910) at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1398
#10 0x00000000007229ba in PostgresServerProcessMain (argc=25, argv=0x1b66910) at ../../../../../../src/postgres/src/backend/main/main.c:234
#11 0x0000000000722bb9 in main ()

Thread 1 (LWP 2575):
#0  0x00007f8a8e25c0a7 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007f8a8e25d4aa in __GI_abort () at abort.c:89
#2  0x00007f8a8f0905cd in __gnu_cxx::__verbose_terminate_handler() () from /home/yugabyte/yb-software/yugabyte-2.4.2.0-b12-centos-x86_64/linuxbrew/lib/libstdc++.so.6
#3  0x00007f8a8f08e636 in ?? () from /home/yugabyte/yb-software/yugabyte-2.4.2.0-b12-centos-x86_64/linuxbrew/lib/libstdc++.so.6
#4  0x00007f8a8f08e681 in std::terminate() () from /home/yugabyte/yb-software/yugabyte-2.4.2.0-b12-centos-x86_64/linuxbrew/lib/libstdc++.so.6
#5  0x00007f8a8f08e898 in __cxa_throw () from /home/yugabyte/yb-software/yugabyte-2.4.2.0-b12-centos-x86_64/linuxbrew/lib/libstdc++.so.6
#6  0x00007f8a883f3190 in cds::throw_exception<cds::threading::pthread::Manager::pthread_exception>(<unknown type in /home/yugabyte/yb-software/yugabyte-2.4.2.0-b12-centos-x86_64/lib/yb/libserver_process.so, CU 0x4c33af, DIE 0x509ca8>, int, const char *) (exception=exception@entry=<unknown type in /home/yugabyte/yb-software/yugabyte-2.4.2.0-b12-centos-x86_64/lib/yb/libserver_process.so, CU 0x4c33af, DIE 0x509ca8>, line=77, file=<optimized out>)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210210192532-45c97f45f1-centos7-linuxbrew/installed/uninstrumented/include/cds/details/throw_exception.h:43
#7  0x00007f8a88424309 in alloc () at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210210192532-45c97f45f1-centos7-linuxbrew/installed/uninstrumented/include/cds/threading/details/pthread_manager.h:77
#8  _threadData (nAction=cds::threading::pthread::Manager::do_attachThread) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210210192532-45c97f45f1-centos7-linuxbrew/installed/uninstrumented/include/cds/threading/details/pthread_manager.h:109
#9  attachThread () at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210210192532-45c97f45f1-centos7-linuxbrew/installed/uninstrumented/include/cds/threading/details/pthread_manager.h:159
#10 yb::Webserver::EnterWorkerThreadCallbackStatic () at ../../src/yb/server/webserver.cc:542
#11 0x00007f8a88442d2a in worker_thread () from /home/yugabyte/yb-software/yugabyte-2.4.2.0-b12-centos-x86_64/lib/yb/libserver_process.so
#12 0x00007f8a8ebd2694 in start_thread (arg=0x7f8a7720d700) at pthread_create.c:333
#13 0x00007f8a8e30f41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Running a small YSQL workload on a 2DC enabled cluster, but that shouldn't be much related to the YSQL webserver..

@mbautin @spolitov you're probably more familiar with the cds::threading piece, any thoughts?
@bmatican bmatican added area/ysql Yugabyte SQL (YSQL) area/docdb YugabyteDB core features labels Mar 16, 2021
@bmatican
Copy link
Contributor Author

notes from Sergei

looks like it failed to call pthread_setspecific and throws exception
we should surround it with try/catch

@m-iancu m-iancu added this to YQL-beta Dec 7, 2021
@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue labels Jun 8, 2022
@bmatican bmatican assigned es1024 and unassigned bmatican, mbautin and spolitov Jul 19, 2022
@bmatican
Copy link
Contributor Author

@es1024 Passing this to you, since you're looking at the core webserver code as well. This is lower pri though.

@bmatican bmatican added priority/low Low priority and removed priority/medium Medium priority issue labels Jul 19, 2022
@es1024 es1024 mentioned this issue Jul 19, 2022
Merged
@es1024 es1024 mentioned this issue Aug 5, 2022
Merged
@es1024
Copy link
Contributor

es1024 commented Aug 16, 2022

Resolved with f2702bb

@es1024 es1024 closed this as completed Aug 16, 2022
@es1024 es1024 moved this to Done in YQL-beta Aug 16, 2022
samiahmedsiddiqui pushed a commit to samiahmedsiddiqui/yugabyte-db that referenced this issue Aug 16, 2022
@es1024
[yugabyte#7683, yugabyte#13349, yugabyte#13064] docdb: Fix webserver …
f2702bb 
…crash when enabling SSL

Summary:
This diff fixes a crash when enable SSL for the webserver, due to an old version
of Squeasel:

1. searching for and loading OpenSSL from the system rather than using the one bundled with
   thirdparty.

2. depending on the dynamic library exposing symbols that have been removed with OpenSSL 1.1.0
   (SSLv23_... methods, which have been replaced with a macro).

This is fixed by rebasing to a newer upstream commit, which does not require those symbols, and
which further presents the ssl_global_init flag to stop Squeasel from loading OpenSSL on its own.

This diff also adds the `webserver_private_key_file` and `webserver_private_key_password` gflags,
corresponding to the `ssl_private_key` and `ssl_private_key_password` options present in newer
versions of Squeasel.

This diff also fixes a crash when pthread_setspecific fails in libcds, by catching the exception
and forcing the worker thread to exit instead of letting it bubble up and kill the whole process.

Also pulling yugabyte/yugabyte-db-thirdparty@fc8dae1 to fix C++ exception handling in Clang-based LTO builds (yugabyte#13064).

Test Plan: `ybd --cxx_test server_webserver-test --gtest_filter WebserverSecureTest.TestIndexPage`

Reviewers: mbautin

Reviewed By: mbautin

Subscribers: cwang, sanketh, ybase, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D18419
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@es1024 es1024

Labels
area/docdb YugabyteDB core features area/ysql Yugabyte SQL (YSQL) kind/bug This issue is a bug priority/low Low priority
Projects
YQL-beta
Status: Done
Milestone
No milestone
Development

No branches or pull requests
5 participants
@bmatican @mbautin @spolitov @es1024 @yugabyte-ci