FB8-54, FB8-55, FB8-70, FB8-101: Expose more information to audit plugin #934

dutow · 2019-01-22T06:44:39Z

Summary:
This commit adds the following fields to the generic event in audit log:

query_id
database
affected_rows
connection_certificate

Reference Patch: 1def6b7
Reference Patch: ce95a09
Reference Patch: 588be34
Reference Patch: ba03c70
Reference Patch: be8c587
Reference Patch: 22b2508

We need some extra info for the shadowing and security logging. This is a
simple first step of info that MariaDB actually also exposes.
Now we would have the query_id and the database name for general events.
Making as few changes as possible to accomplish it, so I'm just taking the
information from the TDH and exposing it through mysql_event_general
struct and as a argument to disconnect.

Forward the connection certificate to the audit plugin. The connection certificate can then be parsed by the audit plugin and handled appropriately. It made more sense for the certificate to live in the connection events, since they generally don't change between every general event, so the move was done.

This is done by caching a BUF_MEM struct on the THD object. Since it's not possible to change certificates on the same connection, this caching should be correct. The BUF_MEM is released on THD::release_resources.

If upstream bumps the MYSQL_AUDIT_INTERFACE_VERSION, we should bump ours to be greater or equal to it.

Expose the port current mysqld is running on for the audit plugin. If no port, 0 is used.

Test Plan:
This commit also introduces changes to the audit_null plugin, which makes
the fields, and further similar changes testable.
The audit_null plugin now has the "extended_log" variable, which can be
turned on. When it's ON, the plugin logs the last generic event log
into the Audit_null_generic_event_response system variable.
This can be easily verified in MTR tests, which is done in the new
audit_null.event_params testcase.

percona-ysorokin · 2019-01-22T11:03:06Z

plugin/audit_null/audit_null.cc

@@ -455,6 +501,9 @@ static int audit_null_notify(MYSQL_THD thd, mysql_event_class_t event_class,
        number_of_calls_general_result++;
        break;
      case MYSQL_AUDIT_GENERAL_STATUS:
+        if (extended_info) {


move extended_info declaration here (inside the case block).

percona-ysorokin · 2019-01-22T11:03:13Z

plugin/audit_null/audit_null.cc

@@ -175,6 +183,10 @@ static MYSQL_THDVAR_INT(event_order_check_exact, PLUGIN_VAR_RQCMDARG,
                        "Plugin checks exact event order.", NULL, NULL, 1, 0, 1,
                        0);

+static MYSQL_THDVAR_INT(extended_log, PLUGIN_VAR_RQCMDARG,


It's better to declare this var as GLOBAL-only (MYSQL_SYSVAR_INT) since we are using a single global buffer for the output anyway.

But doesn't that make thread level control useful? If it's a threadvar, you can turn it on for only one thread, and other threads won't accidentally modify the status variables, breaking tests.

Such change is too visible for user to be discussed only here (unless FB chimes in)

Can this be debug only since it's mainly used for testing? Are there security issues with non-privileged users getting access to see the last logged query.

And is audit_null used outside MTR testing, in production? This is only present in that plugin, that's why I didn't see it as an issue. I can make it debug only, but that will also make all related tests debug only.

It's not used in production, but if it's used as a template for other audit plugins, it might run the risk of it getting out. Let's make it debug-only and having the tests be debug-only would be fine.

percona-ysorokin · 2019-01-22T11:04:52Z

plugin/audit_null/audit_null.cc

@@ -431,6 +476,7 @@ static int audit_null_notify(MYSQL_THD thd, mysql_event_class_t event_class,
  const char *order_str = (const char *)THDVAR(thd, event_order_check);
  int event_order_started = (int)THDVAR(thd, event_order_started);
  int exact_check = (int)THDVAR(thd, event_order_check_exact);
+  int extended_info = (int)THDVAR(thd, extended_log);


const auto extended_info = static_cast<int>(THDVAR(thd, extended_log));

percona-ysorokin · 2019-01-22T11:05:40Z

plugin/audit_null/audit_null.cc

+#undef EVENT_PARAM
+#undef EVENT_PARAM_STR
+
+  std::string str = event_str.str();


const auto str = ...

percona-ysorokin · 2019-01-22T11:10:30Z

plugin/audit_null/audit_null.cc

@@ -455,6 +501,9 @@ static int audit_null_notify(MYSQL_THD thd, mysql_event_class_t event_class,
        number_of_calls_general_result++;
        break;
      case MYSQL_AUDIT_GENERAL_STATUS:
+        if (extended_info) {


if (extended_info !=0)

percona-ysorokin · 2019-01-22T11:35:57Z

mysql-test/suite/audit_null/t/event_params.test

+
+INSERT INTO foo VALUES (1), (2);
+--replace_regex /.*(affected_rows:[^;]*).*/\1/
+SHOW STATUS LIKE "Audit_null_generic_event_response";


percona-ysorokin · 2019-01-22T11:36:03Z

mysql-test/suite/audit_null/t/event_params.test

+
+SELECT * FROM foo;
+--replace_regex /.*(affected_rows:[^;]*).*/\1/
+SHOW STATUS LIKE "Audit_null_generic_event_response";


percona-ysorokin · 2019-01-22T11:36:17Z

mysql-test/suite/audit_null/t/event_params.test

+
+DELETE FROM foo;
+--replace_regex /.*(affected_rows:[^;]*).*/\1/
+SHOW STATUS LIKE "Audit_null_generic_event_response";


percona-ysorokin · 2019-01-22T11:36:31Z

mysql-test/suite/audit_null/t/event_params.test

+let $MYSQLD_PORT= `SELECT @@port`;
+--replace_result $MYSQLD_PORT MYSQLD_PORT
+--replace_regex /.*(port:[^;]*).*/\1/
+SHOW STATUS LIKE "Audit_null_generic_event_response";


percona-ysorokin · 2019-01-22T11:37:05Z

mysql-test/suite/audit_null/r/event_params.result

+Audit_null_generic_event_response	port:MYSQLD_PORT
+UNINSTALL PLUGIN null_audit;
+Warnings:
+Warning	1620	Plugin is busy and will be uninstalled on shutdown


Make sure the test works with --repeat=2

I checked, it works.

percona-ysorokin · 2019-01-22T11:41:12Z

Also, please, add Jira ticket references

dutow · 2019-01-23T16:24:35Z

Updated: alsoadded FB8-55.

percona-ysorokin

Please also add full Jira tickets URLs to the commit message

https://jira.percona.com/browse/FB8-54
https://jira.percona.com/browse/FB8-55
https://jira.percona.com/browse/FB8-70
https://jira.percona.com/browse/FB8-101

percona-ysorokin · 2019-01-24T09:53:17Z

mysql-test/suite/audit_null/t/event_params_cert.test

@@ -0,0 +1,24 @@
+--source include/have_ssl.inc
+


Add --source include/count_sessions.inc as you are establishing/closing a new connection.

percona-ysorokin · 2019-01-24T09:53:53Z

mysql-test/suite/audit_null/t/event_params_cert.test

+DROP TABLE foo;
+
+UNINSTALL PLUGIN null_audit;
+


--source include/wait_until_count_sessions.inc

Still no --source include/wait_until_count_sessions.inc

percona-ysorokin · 2019-01-24T09:58:05Z

plugin/audit_null/audit_null.cc

+  {                                                                            \
+    std::string tmp(event->name.str, event->name.length);                      \
+    boost::replace_all(tmp, "\n", "\\n");                                      \
+    event_str << #name ":" << std::string(event->name.str, event->name.length) \


Should this be

event_str << #name ":" << tmp

?

percona-ysorokin · 2019-01-24T10:01:34Z

sql/sql_class.cc

@@ -893,6 +895,15 @@ void THD::cleanup_connection(void) {
 #endif
 }

+void THD::set_connection_certificate(std::string const &cert) {
+  DBUG_ASSERT(m_connection_certificate == "");


DBUG_ASSERT(m_connection_certificate.empty());

percona-ysorokin · 2019-01-24T10:05:28Z

sql/sql_class.h

+  std::string m_connection_certificate;
+#endif
+
+  std::string const &connection_certificate() const;


Technically, the std::string copy constructor could throw a bad_alloc.

No, no copy constructor here, the result is returned by const reference
std::string const &

Uh, right, I completely missed the & aligned to the function name, fixed.

percona-ysorokin · 2019-01-24T10:08:45Z

sql/sql_class.h

@@ -3323,6 +3323,13 @@ class THD : public MDL_context_owner,
  Gtid_set owned_gtid_set;
 #endif

+#ifdef HAVE_OPENSSL


Should connection_certificate() / set_connection_certificate() also be wrapped with #ifdef HAVE_OPENSSL?
Because otherwise, this does not seem to compile without HAVE_OPENSSL defined

That's a trivial setter / getter, works without SSL.

Actually, my main concern here was that when HAVE_OPENSSL is not defined, the class won't have m_connection_certificate member and therefore connection_certificate() / set_connection_certificate() will fail to compile as m_connection_certificate = cert; / return m_connection_certificate; will be an invalid code.

percona-ysorokin · 2019-01-24T10:11:45Z

sql/sql_class.h

@@ -4032,6 +4039,7 @@ class THD : public MDL_context_owner,
    m_persist_variables_init = is_init;
  }
  bool is_persist_variables_init() { return m_persist_variables_init; }
+


unnecessary change

I think this was clang-format, but I'll recheck what happens if I remove it.

Yes, it was indeed clang-format.

percona-ysorokin · 2019-01-24T10:45:43Z

plugin/audit_null/audit_null.cc

@@ -130,6 +131,11 @@ static char *g_record_buffer;

 #undef AUDIT_NULL_VAR

+static const constexpr size_t event_response_buffer_len = 1000;


Is 1000 enough now when we have certificates also included in GENERAL STATUS event?

For now yes, as all other fields are short.

percona-ysorokin

LGTM

facebook-github-bot

@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

hermanlee · 2019-01-30T20:14:22Z

plugin/audit_null/audit_null.cc

@@ -175,6 +183,10 @@ static MYSQL_THDVAR_INT(event_order_check_exact, PLUGIN_VAR_RQCMDARG,
                        "Plugin checks exact event order.", NULL, NULL, 1, 0, 1,
                        0);

+static MYSQL_THDVAR_INT(extended_log, PLUGIN_VAR_RQCMDARG,


Can this be debug only since it's mainly used for testing? Are there security issues with non-privileged users getting access to see the last logged query.

hermanlee · 2019-01-30T21:45:33Z

sql/sql_class.cc

@@ -863,6 +863,8 @@ void THD::cleanup_connection(void) {
  sp_cache_clear(&sp_proc_cache);
  sp_cache_clear(&sp_func_cache);

+  m_connection_certificate = "";


Does this need to be wrapped in #ifdef HAVE_OPENSSL?

Probably the opposite: any existing HAVE_OPENSSL uses should be removed from patches. E.g. Now CMakeLists.txt contains
ADD_DEFINITIONS(-DHAVE_OPENSSL) # TODO: remove #ifdef from C++ code

I guess it is assumed that WolfSSL is sufficiently similar to OpenSSL, and any code specific for it can be guarded with HAVE_WOLFSSL.

hermanlee · 2019-01-30T21:45:49Z

sql/sql_class.cc

@@ -1058,6 +1071,8 @@ void THD::release_resources() {

  if (current_thd == this) restore_globals();

+  m_connection_certificate = "";


Same, does this need to be wrapped in #ifdef OPENSSL?

hermanlee · 2019-01-30T22:01:36Z

plugin/audit_null/audit_null.cc

+    boost::replace_all(tmp, "\n", "\\n");                 \
+    event_str << #name ":" << tmp << ";";                 \
+  }
+  std::stringstream event_str;


I'm also seeing an internal error compiling this:

../../../plugin/audit_null/audit_null.cc: In function ‘void log_event(const mysql_event_general*)’:
../../../plugin/audit_null/audit_null.cc:438:21: error: aggregate ‘std::stringstream event_str’ has incomplete type and cannot be defined
std::stringstream event_str;

facebook-github-bot · 2019-01-31T19:25:09Z

@dutow has updated the pull request. Re-import the pull request

facebook-github-bot

@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2019-02-04T07:39:15Z

@dutow has updated the pull request. Re-import the pull request

percona-ysorokin · 2019-02-04T12:39:58Z

plugin/audit_null/audit_null.cc

@@ -130,6 +132,11 @@ static char *g_record_buffer;

 #undef AUDIT_NULL_VAR

+static const constexpr size_t event_response_buffer_len = 1000;


Should event_response_buffer_len and generic_event_response definitions be also wrapped in #ifndef DBUG_OFF?

percona-ysorokin · 2019-02-04T12:40:59Z

plugin/audit_null/audit_null.cc

@@ -144,6 +151,9 @@ static SHOW_VAR simple_status[] = {

 #undef AUDIT_NULL_VAR

+    {"Audit_null_generic_event_response", (char *)generic_event_response,


Should this also be wrapped in #ifndef DBUG_OFF?

percona-ysorokin · 2019-02-04T12:42:36Z

plugin/audit_null/audit_null.cc

+/*
+ * Exposes a generic audit log event in a status variable
+ */
+static void log_event(const mysql_event_general *event) {


Should this function also be wrapped in #ifndef DBUG_OFF?

Jira issue: https://jira.percona.com/browse/FB8-54 Jira issue: https://jira.percona.com/browse/FB8-55 Jira issue: https://jira.percona.com/browse/FB8-70 Jira issue: https://jira.percona.com/browse/FB8-101 Reference Patch: 1def6b7 Reference Patch: ba03c70 Reference Patch: ce95a09 Reference Patch: be8c587 Reference Patch: 22b2508 Reference Patch: 588be34 Summary: This commit adds the following fields to the generic event in audit log: * query_id * database * affected_rows * connection_certificate Test Plan: This commit also introduces changes to the audit_null plugin, which makes the fields, and further similar changes testable. The audit_null plugin now has the "extended_log" variable, which can be turned on. When it's ON, the plugin logs the last generic event log into the Audit_null_generic_event_response system variable. This can be easily verified in MTR tests, which is done in the new audit_null.event_params and audit_null.event_params_cert testcases.

facebook-github-bot · 2019-02-04T13:38:45Z

@dutow has updated the pull request. Re-import the pull request

facebook-github-bot · 2019-02-04T13:39:35Z

@dutow has updated the pull request. Re-import the pull request

facebook-github-bot · 2019-02-04T15:39:16Z

@dutow has updated the pull request. Re-import the pull request

facebook-github-bot

@hermanlee has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

…gin (#934) Summary: JIRA: https://jira.percona.com/browse/FB8-54 JIRA: https://jira.percona.com/browse/FB8-55 JIRA: https://jira.percona.com/browse/FB8-70 JIRA: https://jira.percona.com/browse/FB8-101 This commit adds the following fields to the generic event in audit log: * query_id * database * affected_rows * connection_certificate Reference Patch: 1def6b7 Reference Patch: ce95a09 Reference Patch: 588be34 Reference Patch: ba03c70 Reference Patch: be8c587 Reference Patch: 22b2508 We need some extra info for the shadowing and security logging. This is a simple first step of info that MariaDB actually also exposes. Now we would have the `query_id` and the database name for general events. Making as few changes as possible to accomplish it, so I'm just taking the information from the TDH and exposing it through `mysql_event_general` struct and as a argument to disconnect. Forward the connection certificate to the audit plugin. The connection certificate can then be parsed by the audit plugin and handled appropriately. It made more sense for the certificate to live in the connection events, since they generally don't change between every general event, so the move was done. This is done by caching a BUF_MEM struct on the THD object. Since it's not possible to change certificates on the same connection, this caching should be correct. The BUF_MEM is released on THD::release_resources. If upstream bumps the MYSQL_AUDIT_INTERFACE_VERSION, we should bump ours to be greater or equal to it. Expose the port current mysqld is running on for the audit plugin. If no port, 0 is used. Pull Request resolved: #934 Reviewed By: lth Differential Revision: D13874133 Pulled By: lth fbshipit-source-id: 889398c

…gin (facebook#934) (facebook#934) Summary: JIRA: https://jira.percona.com/browse/FB8-54 JIRA: https://jira.percona.com/browse/FB8-55 JIRA: https://jira.percona.com/browse/FB8-70 JIRA: https://jira.percona.com/browse/FB8-101 This commit adds the following fields to the generic event in audit log: * query_id * database * affected_rows * connection_certificate Reference Patch: facebook@1def6b7 Reference Patch: facebook@ce95a09 Reference Patch: facebook@588be34 Reference Patch: facebook@ba03c70 Reference Patch: facebook@be8c587 Reference Patch: facebook@22b2508 We need some extra info for the shadowing and security logging. This is a simple first step of info that MariaDB actually also exposes. Now we would have the `query_id` and the database name for general events. Making as few changes as possible to accomplish it, so I'm just taking the information from the TDH and exposing it through `mysql_event_general` struct and as a argument to disconnect. Forward the connection certificate to the audit plugin. The connection certificate can then be parsed by the audit plugin and handled appropriately. It made more sense for the certificate to live in the connection events, since they generally don't change between every general event, so the move was done. This is done by caching a BUF_MEM struct on the THD object. Since it's not possible to change certificates on the same connection, this caching should be correct. The BUF_MEM is released on THD::release_resources. If upstream bumps the MYSQL_AUDIT_INTERFACE_VERSION, we should bump ours to be greater or equal to it. Expose the port current mysqld is running on for the audit plugin. If no port, 0 is used. Pull Request resolved: facebook#934 Reviewed By: lloyd Differential Revision: D13874133 Pulled By: lth

…gin (percona#934) (percona#934) Summary: JIRA: https://jira.percona.com/browse/FB8-54 JIRA: https://jira.percona.com/browse/FB8-55 JIRA: https://jira.percona.com/browse/FB8-70 JIRA: https://jira.percona.com/browse/FB8-101 This commit adds the following fields to the generic event in audit log: * query_id * database * affected_rows * connection_certificate Reference Patch: facebook/mysql-5.6@1def6b7 Reference Patch: facebook/mysql-5.6@ce95a09 Reference Patch: facebook/mysql-5.6@588be34 Reference Patch: facebook/mysql-5.6@ba03c70 Reference Patch: facebook/mysql-5.6@be8c587 Reference Patch: facebook/mysql-5.6@22b2508 We need some extra info for the shadowing and security logging. This is a simple first step of info that MariaDB actually also exposes. Now we would have the `query_id` and the database name for general events. Making as few changes as possible to accomplish it, so I'm just taking the information from the TDH and exposing it through `mysql_event_general` struct and as a argument to disconnect. Forward the connection certificate to the audit plugin. The connection certificate can then be parsed by the audit plugin and handled appropriately. It made more sense for the certificate to live in the connection events, since they generally don't change between every general event, so the move was done. This is done by caching a BUF_MEM struct on the THD object. Since it's not possible to change certificates on the same connection, this caching should be correct. The BUF_MEM is released on THD::release_resources. If upstream bumps the MYSQL_AUDIT_INTERFACE_VERSION, we should bump ours to be greater or equal to it. Expose the port current mysqld is running on for the audit plugin. If no port, 0 is used. Pull Request resolved: facebook/mysql-5.6#934 Reviewed By: lloyd Differential Revision: D13874133 Pulled By: lth

…gin (facebook#934) (facebook#934) Summary: JIRA: https://jira.percona.com/browse/FB8-54 JIRA: https://jira.percona.com/browse/FB8-55 JIRA: https://jira.percona.com/browse/FB8-70 JIRA: https://jira.percona.com/browse/FB8-101 This commit adds the following fields to the generic event in audit log: * query_id * database * affected_rows * connection_certificate Reference Patch: facebook@1def6b7 Reference Patch: facebook@ce95a09 Reference Patch: facebook@588be34 Reference Patch: facebook@ba03c70 Reference Patch: facebook@be8c587 Reference Patch: facebook@22b2508 We need some extra info for the shadowing and security logging. This is a simple first step of info that MariaDB actually also exposes. Now we would have the `query_id` and the database name for general events. Making as few changes as possible to accomplish it, so I'm just taking the information from the TDH and exposing it through `mysql_event_general` struct and as a argument to disconnect. Forward the connection certificate to the audit plugin. The connection certificate can then be parsed by the audit plugin and handled appropriately. It made more sense for the certificate to live in the connection events, since they generally don't change between every general event, so the move was done. This is done by caching a BUF_MEM struct on the THD object. Since it's not possible to change certificates on the same connection, this caching should be correct. The BUF_MEM is released on THD::release_resources. If upstream bumps the MYSQL_AUDIT_INTERFACE_VERSION, we should bump ours to be greater or equal to it. Expose the port current mysqld is running on for the audit plugin. If no port, 0 is used. Pull Request resolved: facebook#934 Reviewed By: lloyd Differential Revision: D13874133 Pulled By: lth

facebook-github-bot added the CLA Signed label Jan 22, 2019

percona-ysorokin suggested changes Jan 22, 2019

View reviewed changes

dutow changed the title ~~WIP FB8-54, FB8-70, FB8-101: Expose more information to audit plugin~~ WIP FB8-54, FB8-55, FB8-70, FB8-101: Expose more information to audit plugin Jan 23, 2019

percona-ysorokin suggested changes Jan 24, 2019

View reviewed changes

percona-ysorokin reviewed Jan 24, 2019

View reviewed changes

percona-ysorokin approved these changes Jan 29, 2019

View reviewed changes

dutow changed the title ~~WIP FB8-54, FB8-55, FB8-70, FB8-101: Expose more information to audit plugin~~ FB8-54, FB8-55, FB8-70, FB8-101: Expose more information to audit plugin Jan 29, 2019

facebook-github-bot reviewed Jan 30, 2019

View reviewed changes

hermanlee changed the title ~~FB8-54, FB8-55, FB8-70, FB8-101: Expose more information to audit plugin~~ FB8-54, FB8-55, FB8-70, FB8-101, FB8-205: Expose more information to audit plugin Jan 30, 2019

hermanlee changed the title ~~FB8-54, FB8-55, FB8-70, FB8-101, FB8-205: Expose more information to audit plugin~~ FB8-54, FB8-55, FB8-70, FB8-101: Expose more information to audit plugin Jan 30, 2019

hermanlee requested changes Jan 30, 2019

View reviewed changes

hermanlee reviewed Jan 30, 2019

View reviewed changes

facebook-github-bot reviewed Feb 1, 2019

View reviewed changes

percona-ysorokin suggested changes Feb 4, 2019

View reviewed changes

facebook-github-bot reviewed Feb 5, 2019

View reviewed changes

hermanlee approved these changes Feb 5, 2019

View reviewed changes

hermanlee closed this Feb 11, 2019

dutow mentioned this pull request Feb 12, 2019

FB8-53: Expose user certificate details to command line #959

Closed

		@@ -130,6 +131,11 @@ static char *g_record_buffer;

		#undef AUDIT_NULL_VAR

		static const constexpr size_t event_response_buffer_len = 1000;

		@@ -1058,6 +1071,8 @@ void THD::release_resources() {

		if (current_thd == this) restore_globals();

		m_connection_certificate = "";

		@@ -130,6 +132,11 @@ static char *g_record_buffer;

		#undef AUDIT_NULL_VAR

		static const constexpr size_t event_response_buffer_len = 1000;

		@@ -144,6 +151,9 @@ static SHOW_VAR simple_status[] = {

		#undef AUDIT_NULL_VAR

		{"Audit_null_generic_event_response", (char *)generic_event_response,

FB8-54, FB8-55, FB8-70, FB8-101: Expose more information to audit plugin #934

FB8-54, FB8-55, FB8-70, FB8-101: Expose more information to audit plugin #934

Conversation

dutow commented Jan 22, 2019 • edited by hermanlee Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

percona-ysorokin commented Jan 22, 2019

dutow commented Jan 23, 2019

percona-ysorokin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dutow Jan 29, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

percona-ysorokin left a comment

Choose a reason for hiding this comment

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented Jan 31, 2019

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Feb 4, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented Feb 4, 2019

facebook-github-bot commented Feb 4, 2019

facebook-github-bot commented Feb 4, 2019

facebook-github-bot left a comment

Choose a reason for hiding this comment

dutow commented Jan 22, 2019 •

edited by hermanlee

Loading

dutow Jan 29, 2019 •

edited

Loading