new debian mysql does not load plugin #72

vincentkersten · 2014-01-28T12:04:40Z

Hello,
yet another issue with the plugin not working :(

ERROR 1123 (HY000): Can't initialize function 'AUDIT'; Plugin initialization function failed.

when issuing "INSTALL PLUGIN AUDIT SONAME 'libaudit_plugin.so'; "

I have tried:

install of the 'regular' version.
audit-plugin-mysql-5.5-1.0.3-371-linux-x86_64.zip
install of the snapshot version.
audit-plugin-mysql-5.5-1.0.4-456-linux-x86_64.zip
compile of the source against the deb-source package.
apt-get source mysql-server
cd mysql-5.5-5.5.35+dfsg/
cmake .
cd extra/
make
cd ../libservices/
make
cd ../../

wget https://github.com/mcafee/mysql-audit/archive/master.zip
unzip && cd mysql-audit-master/
./configure --with-mysql=../mysql-5.5-5.5.35+dfsg --with-mysql-libservices=../mysql-5.5-5.5.35+dfsg/libservices/libmysqlservices.a
make

scp src/.libs/libaudit_plugin.so.0.0.0 root@sqlserver:/usr/lib/mysql/plugin/libaudit_plugin.so

... of course with mysql restarts in-between, but to no avail. I would like to end up with a recipe for recreating the plugin when upgrades break this, since we obviously want to audit stuff :) and it worked very well so far.

Please let me know if I can do something else or provide you with more information.

Vincent

glicht · 2014-01-28T14:30:53Z

Can you send out the mysql error log. Also can you specify the exact
version of MySQL you are testing. Best is to provide a link from where the
binary distribution was downloaded. There are alot of distributions and I
want to be sure we are talking about the same one.

On Tue, Jan 28, 2014 at 2:04 PM, vincentkersten notifications@github.comwrote:

Hello,
yet another issue with the plugin not working :( ERROR 1123 (HY000):
Can't initialize function 'AUDIT'; Plugin initialization function failed.

when issuing "INSTALL PLUGIN AUDIT SONAME 'libaudit_plugin.so'; "

I have tried:

install of the 'regular' version.
audit-plugin-mysql-5.5-1.0.3-371-linux-x86_64.zip

install of the snapshot version.
audit-plugin-mysql-5.5-1.0.4-456-linux-x86_64.zip
2.

compile of the source against the deb-source package.
apt-get source mysql-server
cd mysql-5.5-5.5.35+dfsg/
cmake .
cd extra/
make
cd ../libservices/
make
cd ../../

wget https://github.com/mcafee/mysql-audit/archive/master.zip
unzip && cd mysql-audit-master/
./configure --with-mysql=../mysql-5.5-5.5.35+dfsg
--with-mysql-libservices=../mysql-5.5-5.5.35+dfsg/libservices/libmysqlservices.a
make

scp src/.libs/libaudit_plugin.so.0.0.0 root@sqlserver
:/usr/lib/mysql/plugin/libaudit_plugin.so

... of course with mysql restarts in-between, but to no avail. I would
like to end up with a recipe for recreating the plugin when upgrades break
this, since we obviously want to audit stuff :) and it worked very well so
far.

Please let me know if I can do something else or provide you with more
information.

Vincent

Reply to this email directly or view it on GitHubhttps://github.com//issues/72
.

vincentkersten · 2014-01-28T16:37:43Z

Dear Guy,

apt-get source mysql-server

Get:1 http://security.debian.org/ wheezy/updates/main mysql-5.5 5.5.35+dfsg-0+wheezy1 (dsc) [2,990 B]
Get:2 http://security.debian.org/ wheezy/updates/main mysql-5.5 5.5.35+dfsg-0+wheezy1 (tar) [21.7 MB]
Get:3 http://security.debian.org/ wheezy/updates/main mysql-5.5 5.5.35+dfsg-0+wheezy1 (diff) [375 kB]

(this is from adding
deb-src http://security.debian.org/ wheezy/updates main contrib non-free
to /etc/apt/sources.list).
So a fairly standard debian (x86_64) install.

it then carries on and applies debian-specific patches:
dpkg-source: info: extracting mysql-5.5 in mysql-5.5-5.5.35+dfsg
dpkg-source: info: unpacking mysql-5.5_5.5.35+dfsg.orig.tar.gz
dpkg-source: info: unpacking mysql-5.5_5.5.35+dfsg-0+wheezy1.debian.tar.gz
dpkg-source: info: applying 1_hurd.patch
dpkg-source: info: applying 33_scripts__mysql_create_system_tables__no_test.patch
dpkg-source: info: applying 38_scripts__mysqld_safe.sh__signals.patch
dpkg-source: info: applying 41_scripts__mysql_install_db.sh__no_test.patch
dpkg-source: info: applying 50_mysql-test__db_test.patch
dpkg-source: info: applying 62_disable_tests.patch
dpkg-source: info: applying 70_mysql_va_list.patch
dpkg-source: info: applying 71_disable_rpl_tests.patch
dpkg-source: info: applying 72_fix_standalone_tests.patch
dpkg-source: info: applying 73_fix_rpl_deadlock_innodb_test.patch
dpkg-source: info: applying 93_kfreebsd_tests.patch
dpkg-source: info: applying 94_spelling.patch
dpkg-source: info: applying regex_plus.patch
dpkg-source: info: applying versioned_symbols.patch

the error from the logs is

Jan 28 17:21:51 SERVER mysqld: 140128 17:21:51 [Note] Audit Plugin: Set interface version to: 12936960 (50535)
Jan 28 17:21:51 SERVER mysqld: 140128 17:21:51 [Warning] Using unique option prefix audit-record-cmd instead of audit-record-cmds is deprecated and will be removed in a future release. Please use the full name instead.
Jan 28 17:21:51 SERVER mysqld: 140128 17:21:51 [Note] Audit Plugin: starting up. Version: 1.0.3 , Revision: 371 (64bit). AUDIT plugin interface version: 50535. MySQL Server version: 5.5.35-0+wheezy1.
Jan 28 17:21:51 SERVER mysqld: 140128 17:21:51 [Note] Audit Plugin: setup_offsets audit_offsets: (null) validate_checksum: 1 offsets_by_version: 1
Jan 28 17:21:51 SERVER mysqld: 140128 17:21:51 [Note] Audit Plugin: mysqld: /usr/sbin/mysqld (5950afa6e6b895df4aedcf6f47ac1ee0)  
Jan 28 17:21:51 SERVER mysqld: 140128 17:21:51 [Note] Audit Plugin: Couldn't find proper THD offsets for: 5.5.35-0+wheezy1
Jan 28 17:21:51 SERVER mysqld: 140128 17:21:51 [ERROR] Plugin 'AUDIT' init function returned error.
Jan 28 17:21:51 SERVER mysqld: 140128 17:21:51 [Note] Audit Plugin: deinit

my config is in /etc/mysql/conf.d/mysqld_AUDIT_plugin.cnf:

[mysqld]
###
# Audit plugin
# http://github.com/mcafee/mysql-audit/wiki/installation
###


## mysqld --help --verbose for quick config reference

plugin-load=AUDIT=libaudit_plugin.so

# Die if we cannot load the plugin
#audit = FORCE

# Enable the log file via JSON
audit-json-file = 1

# Where we log to
audit-json-log-file = /home/audit/mysql_audit/mysql_audit.log

# only audit DB we are interested in
audit-record-objs = DB1.*,DB2.*,DB3.*

# only audit changes
audit-record-cmd = create,alter,drop,update,delete,insert

atze234 · 2014-01-29T10:34:02Z

Hi,
same Problem here with the latest debian mysqld. I tried to extract the new offsets and build a debug package with sources from http://packages.debian.org/source/wheezy/mysql-5.5 to find a general recipe for new mysql Versions.

dpkg-source -x mysql-5.5_5.5.35+dfsg-0+wheezy1.dsc
added -DWITH_DEBUG=ON to debian/rules
dpkg-buildpackage -b -uc

$ ./offset-extract.sh /root/debug-build/mysqld 
140129 11:16:17 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
//offsets for: /root/debug-build/mysqld (5.5.35-0)
{"5.5.35-0","1ffb49db2e21d2634880421251a6392e", 6344, 6392, 4016, 4520, 88, 2592, 96, 0, 32, 104},

but at last:

140129 10:26:10 [Note] Audit Plugin: starting up. Version: 1.0.4 , Revision: 456 (64bit). AUDIT plugin interface version: 50535 (0xc567). MySQL Server version: 5.5.35-0+wheezy1-log.
140129 10:26:10 [Note] Audit Plugin: setup_offsets audit_offsets: 6344,6392,4016,4520,88,2592,96,0,32,104 validate_checksum: 0 offsets_by_version: 1
140129 10:26:10 [Note] Audit Plugin: mysqld: /usr/sbin/mysqld (5950afa6e6b895df4aedcf6f47ac1ee0)
140129 10:26:10 [Note] Audit Plugin: setup_offsets Audit_formatter::thd_offsets values: 6344 6392 4016 4520 88 2592 96 0 32 104
140129 10:26:10 [ERROR] Audit Plugin: Offsets: (null) ((null)) match thread validation check fails with value: 0. Skipping offest.
140129 10:26:10 [ERROR] Audit Plugin: Offsets set didn't pass validation. audit_offsets: 6344,6392,4016,4520,88,2592,96,0,32,104 .
140129 10:26:10 [ERROR] Plugin 'AUDIT' init function returned error.

Im not sure thats the right way to get those offsets, isnt it? But maybe this approach could help with this Problem too.

Jörg

Error with github master and no configured offsets:

140129 14:04:54 [Note] Audit Plugin: starting up. Version: 1.0.0 , Revision: 99999 (64bit). AUDIT plugin interface version: 50535 (0xc567). MySQL Server version: 5.5.35-0+wheezy1-log.
140129 14:04:54 [Note] Audit Plugin: setup_offsets audit_offsets: (null) validate_checksum: 0 offsets_by_version: 1
140129 14:04:54 [Note] Audit Plugin: mysqld: /usr/sbin/mysqld (5950afa6e6b895df4aedcf6f47ac1ee0)
140129 14:04:54 [ERROR] Audit Plugin: Offsets: 5.5.35 (09c5971f9df91d9fde18e969f66d9ff7) match thread validation check fails with value: 0. Skipping offest.
140129 14:04:54 [Note] Audit Plugin: extended offsets validate res: MySQL thread id 123456, OS thread handle 0x0, query id 789 aud_tusr
140129 14:04:54 [Note] Audit Plugin: Using decrement (24) offsets from offset version: 5.5.35 (09c5971f9df91d9fde18e969f66d9ff7) values: 6112 6160 3792 4288 88 2592 96 0 32 104
140129 14:04:54 [Note] Audit Plugin: mem func addr: 0x7f0fa44e4b00 mem start addr: 0x7f0fa44e5000 page size: 4096
140129 14:04:54 [Note] Audit Plugin: hot patching function: 0x7f0fc6f59aa0, trampolineFunction: 0x7f0fa44e5000 trampolinePage: 0x7f0fa44e5000
140129 14:04:54 [ERROR] Audit Plugin: unable to disassemble at address: 0x0x7f0fc6f59aaa. Found relative addressing for instruction: [jnz 0x7f0fc6f59ab5]. Aborting.
140129 14:04:54 [ERROR] Audit Plugin: unable to hot patch log_slow_statement (0x7f0fc6f59aa0). res: -1. Aborting.
140129 14:04:54 [ERROR] Plugin 'AUDIT' init function returned error.

glicht · 2014-01-29T15:17:54Z

Hi Jorg,

Thanks for reporting this problem. The offsets already exist in the latest dev snapshot (1.0.4). 1,0.3 fails as it doesn't have the offsets.

Problem is we fail hot patching the function log_slow_statement on debian. I think this function is compiled differently on debian compared to other distributions. Log message regarding this failure:

Audit Plugin: unable to hot patch log_slow_statement

Anyway, hot patch log_slow_statement is not critical for monitoring. I am going to commit a fix for this.

…non critical

glicht · 2014-01-29T16:19:09Z

New build is availabe here: https://bintray.com/mcafee/mysql-audit-plugin/dev-snapshot/1.0.4-459/files .

I am closing the issue. If you encounter any issues or have questions feel free to re-open.

atze234 · 2014-01-29T16:36:22Z

Seems to work like a charm. Thank you!
I am still wondering how to get those offsets out of the debian source. Mine still wont work. Do you have a hint?

vincentkersten · 2014-01-30T09:20:28Z

Guy,
excellent work! Patch the Patchers :) Works great here as well.
Thank you again for the quick fix

glicht · 2014-01-30T13:34:49Z

atze234,

Not sure how you are compiling. But to get the offsets you need to compile with the exact same cmake command used by the debian build system. I think the problem is that you added -DWITH_DEBUG=ON to debian/rules .

atze234 · 2014-01-30T15:59:31Z

Ah, i thought i have to activate it to get the symbols, but its just an

export DEB_BUILD_OPTIONS="debug"

Now there are symbols in it, but the offset Extract says:

//offsets for: /root/mysqld (5.5.35-0)
offsets.gdb:2: Error in sourced command file:
No symbol "size_t" in current context.,

almost there :(

vincentkersten · 2014-01-31T07:29:28Z

Hello AZ
as per manual:
$ DEB_BUILD_OPTIONS="nostrip noopt" fakeroot apt-get -b source hello
https://wiki.debian.org/HowToGetABacktrace

Regards,

Vincent

atze234 · 2014-01-31T14:33:42Z

Hi Vincent,
this is what i've done. I event tried it with 5.5.31, but also there for example:

(gdb) symbol-file /usr/sbin/mysqld
Reading symbols from /usr/sbin/mysqld...done.
(gdb) printf "%d", ((size_t)&((THD *)log_slow_statement)->query_id)
No symbol "size_t" in current context.
(gdb) printf "%d", log_slow_statement
3287488(gdb)

Jörg

glicht · 2014-01-31T19:44:06Z

The nostrip option is needed to avoid stripping but you also need to add "-g". I took a look at the rules file and you need to modify the file and add "-g" to CFLAGS and CXXFLAGS passed to cmake.

So modify the line:

CFLAGS=$${MYSQL_BUILD_CFLAGS:-"-O2 -DBIG_JOINS=1 ${FORCE_FPIC_CFLAGS} -fno-strict-aliasing"}

To:

CFLAGS=$${MYSQL_BUILD_CFLAGS:-"-g -O2 -DBIG_JOINS=1 ${FORCE_FPIC_CFLAGS} -fno-strict-aliasing"}

And:

CXXFLAGS=$${MYSQL_BUILD_CXXFLAGS:-"-O3 -DBIG_JOINS=1 -felide-constructors -fno-exceptions -fno-rtti ${FORCE_FPIC_CFLAGS} -fno-strict-aliasing"}

to:

CXXFLAGS=$${MYSQL_BUILD_CXXFLAGS:-"-g -O3 -DBIG_JOINS=1 -felide-constructors -fno-exceptions -fno-rtti ${FORCE_FPIC_CFLAGS} -fno-strict-aliasing"}

glicht · 2014-01-31T20:10:29Z

I've added instructions for Debian build on the wiki page: https://github.com/mcafee/mysql-audit/wiki/Troubleshooting . Any comments are welcome.

vincentkersten · 2014-02-01T19:26:09Z

Well done, Guy and Jorg, instructions look good and all is well again :)
Thank you for all the time and effort!

Vincent

atze234 · 2014-02-03T11:32:14Z

Works very well with the "-g" build offsets.
Thank you.

glicht pushed a commit that referenced this issue Jan 29, 2014

fix for issue #72 . treat failure to hot patch log_slow_statement as …

95a8bec

…non critical

glicht closed this as completed Jan 29, 2014

lurdan mentioned this issue Jan 6, 2016

fails hot patching mysql_execute_command for debian mysql #132

Closed

kubo mentioned this issue Jun 12, 2016

Reduce the minimum required hot patching size from 14 to 6 on x86_64 #141

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

new debian mysql does not load plugin #72

new debian mysql does not load plugin #72

vincentkersten commented Jan 28, 2014

glicht commented Jan 28, 2014

vincentkersten commented Jan 28, 2014

atze234 commented Jan 29, 2014

glicht commented Jan 29, 2014

glicht commented Jan 29, 2014

atze234 commented Jan 29, 2014

vincentkersten commented Jan 30, 2014

glicht commented Jan 30, 2014

atze234 commented Jan 30, 2014

vincentkersten commented Jan 31, 2014

atze234 commented Jan 31, 2014

glicht commented Jan 31, 2014

glicht commented Jan 31, 2014

vincentkersten commented Feb 1, 2014

atze234 commented Feb 3, 2014

new debian mysql does not load plugin #72

new debian mysql does not load plugin #72

Comments

vincentkersten commented Jan 28, 2014

glicht commented Jan 28, 2014

vincentkersten commented Jan 28, 2014

atze234 commented Jan 29, 2014

glicht commented Jan 29, 2014

glicht commented Jan 29, 2014

atze234 commented Jan 29, 2014

vincentkersten commented Jan 30, 2014

glicht commented Jan 30, 2014

atze234 commented Jan 30, 2014

vincentkersten commented Jan 31, 2014

atze234 commented Jan 31, 2014

glicht commented Jan 31, 2014

glicht commented Jan 31, 2014

vincentkersten commented Feb 1, 2014

atze234 commented Feb 3, 2014