Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[master] MySQL module fixes #56174

Merged
merged 6 commits into from
Mar 10, 2020

Conversation

garethgreenaway
Copy link
Contributor

@garethgreenaway garethgreenaway commented Feb 15, 2020

What does this PR do?

Various fixes to the mysql module to break out the handling of user management into different functions based on MySQL variant.

What issues does this PR fix or reference?

#56124
#56177
#56170

Tests written?

[NOTICE] Bug fixes or features added to Salt require tests.
Please review the test documentation for details on how to implement tests into Salt's test suite.
Yes. Existing tests.

Commits signed with GPG?

Yes

Please review Salt's Contributing Guide for best practices.

See GitHub's page on GPG signing for more information about signing commits with GPG.

…anagement into different functions based on MySQL variant.
@garethgreenaway garethgreenaway requested a review from a team as a code owner February 15, 2020 06:07
@ghost ghost requested a review from dwoz February 15, 2020 06:07
Copy link

@poofyteddy poofyteddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not very good with python, but i had to fix issue with auth socket in earlier version, since you are splitting user management between mysql and mariadb, i feel it's a good time to fix those to.

salt/modules/mysql.py Outdated Show resolved Hide resolved
salt/modules/mysql.py Outdated Show resolved Hide resolved
…QL and MariaDB. Adding some functions to install, remove, and check the status of plugins which we can then use when adding users which will use the unix_socket & auth_socket plugins. Adding additional tests for these new functions as well as test to ensure the correct SQL is being generated when using passwordless and unix_socket options.
@garethgreenaway
Copy link
Contributor Author

I updated the PR to include some fixes for unix_socket support, turns out it's different between MySQL and MariaDB. Also added some additional functions to manage and check plugins, as the unix_socket and auth_socket plugins aren't enabled by default so we want to check for those before attempting to use them, makes for a nicer user experience to handle the error rather than get a traceback. If anyone is able to test out the PR that would be greatly appreciated. Thanks!

@carsten-AEI
Copy link

carsten-AEI commented Feb 17, 2020

If anyone is able to test out the PR that would be greatly appreciated. Thanks!

I just tested the diff by directly applying it onto a salt-minion installed from upstream's Debian repo (3000+ds-1) - obviously, I ignored the unit tests for it. Afterwards, the initial problems AttributeError: 'str' object has no attribute 'items' or TypeError: an integer is required (got type str) (with conv set to None) have vanished and mysql states seem to work again. 👍

For completeness:

Salt Version:
Salt: 3000
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.7.3
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: 1.3.10
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.7.3 (default, Dec 20 2019, 18:57:59)
python-gnupg: Not Installed
PyYAML: 3.13
PyZMQ: 17.1.2
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.1
System Versions:
dist: debian 10.3
locale: UTF-8
machine: x86_64
release: 4.19.0-6-amd64
system: Linux
version: debian 10.3

@scambra
Copy link

scambra commented Feb 17, 2020

I have tested in same way, running mariadb, and mysql_user.present with allow_passwordless and unix_socket writes auth_socket to plugin column, then login with root fails, as I said in a comment.

root_unix_socket:
  mysql_user.present:
    - name: root
    - allow_passwordless: true
    - unix_socket: true
    - connection_host: 'localhost'
    - connection_user: '{{ mysql_salt_user }}'
  {%- if mysql_salt_pass %}
    - connection_pass: '{{ mysql_salt_pass }}'
  {%- endif %}
Salt Version:
           Salt: 3000
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: 1.3.10
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.3 (default, Dec 20 2019, 18:57:59)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1
 
System Versions:
           dist: debian 10.3 
         locale: utf-8
        machine: x86_64
        release: 5.3.10-1-pve
         system: Linux
        version: debian 10.3 

@scambra
Copy link

scambra commented Feb 17, 2020

Changing auth_socket to unix_socket in line 1780 of modules/mysql.py fixes this issue.

@poofyteddy
Copy link

user_create look great !

The auth for user_exists [L1305 && L1344] and user_chpass [L1728 && L1780] need to work the same way you've fixed user_create

@garethgreenaway
Copy link
Contributor Author

@carsten-AEI Thanks for testing.

@scambra Thanks. What version of MariaDB were you testing with?

@poofyteddy Thanks. I'll update those functions as well.

…riadb chpass functions to ensure that the respective plugins are enabled before attempting to use them.
@garethgreenaway
Copy link
Contributor Author

I've updated the PR with the requested changes.
@scambra I found the issue that you were referring to and it's been updated.

@poofyteddy I updated the chpass functions to ensure that the respective plugin is available before attempting to use it. I'm not entirely sure we need that logic for the exists functions as it's just doing a query to see if they're there, can you elaborate a bit on that request?

Thanks!

@poofyteddy
Copy link

I'm not entirely sure we need that logic for the exists functions as it's just doing a query to see if they're there, can you elaborate a bit on that request?

Oh ... i really didn't think about that....
maybe if module are called individually (not in state) but that would be an odd use-case.
Sorry for making you work for nothing :(

L1345,L1491,L1732,L1788 you use an args, for the socket. useful if you build your commend from condition, but with this rewrite you no longer use them.
In user_create your write them directly in the request. It look ok to me with it to be clear.

I'm having issue that prevent me test the code. i can only offer a read today.

@garethgreenaway
Copy link
Contributor Author

garethgreenaway commented Feb 17, 2020

@poofyteddy No worries! I just wanted to make sure I was clear about what your idea there was.
The user_exists function can be called outside of a state, it's a remote execution function in the module. Even then we're just looking to see if the value in the user table matches what we think the auth plugin should be.

In the other functions, we're still using args. They are returned from the database specific functions along with the built up query.

@scambra
Copy link

scambra commented Feb 18, 2020

I'm using mariadb 10.3. It's failing to detect unix_socket plugin is enabled. I had plugin enabled in mysql config:

[mysqld]
plugin-load-add = auth_socket.so

Plugin library is auth_socket.so, but plugin name is unix_socket.

MariaDB [mysql]> SELECT PLUGIN_STATUS FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_NAME = "unix_socket";
+---------------+
| PLUGIN_STATUS |
+---------------+
| ACTIVE        |
+---------------+

When plugin_status is not enabled, or salt thinks is not enabled, user_chpass state raise exception instead of displaying error. _mariadb_user_chpass (and _mysql_user_chpass) return false for qry variable, then _execute is called with qry, instead of checking qry

local:
----------
          ID: root_unix_socket
    Function: mysql_user.present
        Name: root
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 238, in execute                                                                                                                      
                  query = query % args                                                                                                                                                                              
              TypeError: unsupported operand type(s) for %: 'bool' and 'dict'                                                                                                                                       
                                                                                                                                                                                                                    
              During handling of the above exception, another exception occurred:                                                                                                                                   
                                                                                                                                                                                                                    
              Traceback (most recent call last):                                                                                                                                                                    
                File "/usr/lib/python3/dist-packages/salt/state.py", line 1981, in call                                                                                                                             
                  **cdata['kwargs'])                                                                                                                                                                                
                File "/usr/lib/python3/dist-packages/salt/loader.py", line 1977, in wrapper                                                                                                                         
                  return f(*args, **kwargs)                                                                                                                                                                         
                File "/usr/lib/python3/dist-packages/salt/states/mysql_user.py", line 193, in present                                                                                                               
                  **connection_args):                                                                                                                                                                               
                File "/usr/lib/python3/dist-packages/salt/modules/mysql.py", line 1890, in user_chpass                                                                                                              
                  result = _execute(cur, qry, args)                                                                                                                                                                 
                File "/usr/lib/python3/dist-packages/salt/modules/mysql.py", line 647, in _execute                                                                                                                  
                  return cur.execute(qry, args)                                                                                                                                                                     
                File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 240, in execute                                                                                                                      
                  self.errorhandler(self, ProgrammingError, str(m))                                                                                                                                                 
                File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 52, in defaulterrorhandler                                                                                                       
                  raise errorclass(errorvalue)                                                                                                                                                                      
              _mysql_exceptions.ProgrammingError: unsupported operand type(s) for %: 'bool' and 'dict'                                                                                                              
     Started: 11:19:54.762890
    Duration: 6.509 ms
     Changes:   

@scambra
Copy link

scambra commented Feb 18, 2020

salt-call mysql.plugin_status unix_socket returned nothing:

local:

Removing double quotes from qry fixes issue detecting plugin status:

-    qry = 'SELECT PLUGIN_STATUS FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_NAME = "%(name)s"'
+    qry = 'SELECT PLUGIN_STATUS FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_NAME = %(name)s'

With fixed query:

local:
    ACTIVE

plugin_remove and plugin_add are wrong too:
Error for plugin_remove:
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''unix_socket'' at line 1")
Error for plugin_add:
_mysql_exceptions.ProgrammingError: (1064, 'You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''unix_socket' SONAME "'auth_socket.so'"' at line 1')

mariadb doesn't want plugin name to be quoted, changing lines 2745 and 2782 fixes it in mariadb, I don't know about mysql:

-        _execute(cur, qry, args)
+        _execute(cur, qry % args)
$ salt-call mysql.plugin_remove unix_socket
local:
    True
$ sudo salt-call mysql.plugin_status unix_socket                        
[ERROR   ] MySQL Error 1524: Plugin 'unix_socket' is not loaded
local:
$ sudo salt-call mysql.plugin_add unix_socket soname='auth_socket.so' connection_default_file=/etc/mysql/debian.cnf
local:
    True
$ sudo salt-call mysql.plugin_status unix_socket 
local:
    ACTIVE

@garethgreenaway
Copy link
Contributor Author

@scambra Pushed some more updates to the PR, can you check again and see if this last push fixes the remaining issues you've identified?

@scambra
Copy link

scambra commented Feb 19, 2020

mysql.plugin_status still fails, query in mysql.plugin_status hasn't been changed, double quotes should be removed in line 2824, _execute will take care of it. I don't know about test, but I guess previous test was right if double quotes were removed too. And add args into 2829 line.

Or use "{0}"'.format(name) in line 2824, as uninstall plugin query.

@garethgreenaway
Copy link
Contributor Author

@scambra I tested without the quotes and it errored out for me. Can you provide an example state file or usage for when you see the double quotes in the plugin_status function causing an issue? Thanks!

@scambra
Copy link

scambra commented Feb 19, 2020

$ sudo salt-call mysql.plugin_status unix_socket 
local:
$ sudo mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 29833
Server version: 10.3.22-MariaDB-1:10.3.22+maria~buster mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SELECT PLUGIN_STATUS FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_NAME = "unix_socket";
+---------------+
| PLUGIN_STATUS |
+---------------+
| ACTIVE        |
+---------------+
1 row in set (0.000 sec)

Current code is querying with SELECT PLUGIN_STATUS FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_NAME = "%(name)s", without replacing %(name)s, because it calls _execute(cur, qry), without args. If I remove double quotes, it raises error, which fixes when args is added to _execute. I have changed modules/mysql.py like this (first line is 2823):

    cur = dbc.cursor()
    qry = 'SELECT PLUGIN_STATUS FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_NAME = %(name)s'
    args = {}
    args['name'] = name

    try:
        _execute(cur, qry, args)
$ sudo salt-call mysql.plugin_status unix_socket
local:
    ACTIVE

@garethgreenaway
Copy link
Contributor Author

@scambra As soon as I posted this I tested again and saw the scenario you were talking about. I've updated the PR and believe it's in the correct state now. Thanks!

@scambra
Copy link

scambra commented Feb 20, 2020

@garethgreenaway everything works now

@garethgreenaway
Copy link
Contributor Author

@scambra excellent! Thanks for all the testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
v3000.1 vulnerable version
Projects
None yet
Development

Successfully merging this pull request may close these issues.

mariadb socket access must be enabled before highstate - salt 3K still tries empty password
6 participants