Skip to content

Commit

Permalink
Fixing occasional MyRocks SIGABRT on shutdown (percona#122)
Browse files Browse the repository at this point in the history
Summary:
RocksDB has some static object dependencies, and without
following dependencies, it hits SIGABRT. This diff has
two fixes.
1. Updating RocksDB revision. D51789 in RocksDB changes the problematic
mutex from global variable to local (function-level) static variable.
2. Changing MyRocks no to call ddl_manager.persist_stats() by
kill_server_thread (which calls rocksdb_done_func()). By calling
ddl_manager.persist_stats(),
rocksdb::ThreadLocalPtr::StaticMeta::OnThreadExit() (which calls
pthread_mutex_lock on "static port::Mutex mutex") is called at the end
of the thread. On the other hand, there is a race between main thread and
kill_server_thread. main thread calls pthread_mutex_destroy for the
mutex. To guarantee not to call pthread_mutex_lock() after
pthread_mutex_destroy(), it is necessary not to call
ddl_manager.persist_stats() from kill_server_thread. This diff fixes the
issue.

update-submodule: rocksdb

Reviewed By: jkedgar

Differential Revision: https://reviews.facebook.net/D51933

fbshipit-source-id: ca6e0722d88
  • Loading branch information
yoshinorim authored and inikep committed Nov 11, 2020
1 parent 08b9159 commit 786028f
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 4 deletions.
2 changes: 1 addition & 1 deletion rocksdb
Submodule rocksdb updated 77 files
+5 −2 CMakeLists.txt
+6 −1 HISTORY.md
+4 −1 Makefile
+3 −3 build_tools/fbcode_config.sh
+3 −3 build_tools/fbcode_config4.8.1.sh
+3 −3 build_tools/run_ci_db_test.ps1
+2 −1 db/builder.cc
+10 −9 db/column_family.cc
+95 −9 db/column_family_test.cc
+2 −2 db/compaction.cc
+100 −20 db/compaction_iterator.cc
+11 −1 db/compaction_iterator.h
+3 −3 db/compaction_iterator_test.cc
+4 −2 db/compaction_job.cc
+7 −0 db/compaction_job.h
+252 −36 db/compaction_job_test.cc
+109 −55 db/db_bench.cc
+54 −1 db/db_compaction_test.cc
+88 −56 db/db_impl.cc
+15 −3 db/db_impl.h
+82 −83 db/db_test.cc
+4 −4 db/db_test_util.cc
+2 −1 db/db_test_util.h
+7 −1 db/db_universal_compaction_test.cc
+3 −3 db/file_indexer.h
+24 −36 db/internal_stats.cc
+2 −19 db/internal_stats.h
+2 −2 db/log_writer.cc
+1 −1 db/log_writer.h
+3 −0 db/snapshot_impl.cc
+21 −2 db/snapshot_impl.h
+2 −1 db/version_builder.cc
+33 −21 db/version_set.cc
+9 −10 db/version_set.h
+1 −1 db/write_controller.cc
+2 −0 db/write_thread.cc
+45 −98 examples/rocksdb_option_file_example.ini
+1 −1 hdfs/README
+3 −1 include/rocksdb/cache.h
+1 −0 include/rocksdb/env.h
+1 −1 include/rocksdb/listener.h
+18 −6 include/rocksdb/options.h
+3 −0 include/rocksdb/snapshot.h
+1 −1 include/rocksdb/table.h
+166 −0 include/rocksdb/utilities/env_mirror.h
+2 −2 include/rocksdb/utilities/transaction.h
+2 −2 java/src/main/java/org/rocksdb/ColumnFamilyOptionsInterface.java
+1 −1 port/port_posix.h
+8 −4 port/win/env_win.cc
+3 −12 port/win/port_win.h
+2 −1 port/win/win_logger.cc
+2 −0 src.mk
+3 −3 tools/auto_sanity_test.sh
+2 −1 tools/db_stress.cc
+1 −1 tools/rdb/API.md
+3 −3 util/cache.cc
+2 −2 util/coding.h
+2 −2 util/compression.h
+18 −304 util/env_posix.cc
+2 −2 util/mutable_cf_options.cc
+5 −4 util/mutable_cf_options.h
+6 −3 util/options.cc
+2 −2 util/options_builder.cc
+4 −2 util/options_helper.cc
+3 −0 util/options_helper.h
+2 −0 util/options_parser.cc
+1 −1 util/sync_point.h
+7 −0 util/testutil.h
+36 −15 util/thread_local.cc
+36 −3 util/thread_local.h
+250 −0 util/thread_posix.cc
+86 −0 util/thread_posix.h
+6 −1 utilities/document/json_document_builder.cc
+266 −0 utilities/env_mirror.cc
+222 −0 utilities/env_mirror_test.cc
+6 −1 utilities/transactions/transaction_base.cc
+1 −1 utilities/write_batch_with_index/write_batch_with_index.cc
6 changes: 3 additions & 3 deletions storage/rocksdb/ha_rocksdb.cc
Original file line number Diff line number Diff line change
Expand Up @@ -2402,9 +2402,6 @@ static int rocksdb_done_func(void *p)
mysql_mutex_lock(&background_mutex);
mysql_mutex_unlock(&background_mutex);

// save remaining stats which might've left unsaved
ddl_manager.persist_stats();

// wait for the drop index thread to finish
mysql_mutex_lock(&drop_index_mutex);
mysql_mutex_unlock(&drop_index_mutex);
Expand Down Expand Up @@ -6987,6 +6984,9 @@ void* background_thread(void*)
}
}

// save remaining stats which might've left unsaved
ddl_manager.persist_stats();

mysql_mutex_unlock(&background_mutex);

return nullptr;
Expand Down

0 comments on commit 786028f

Please sign in to comment.