[DPE-4266] Strip passwords from command execute output and tracebacks

[shayancanonical]

Issue

We are leaking passwords when there's an issue creating a backup. For instance:

unit-mysql-k8s-0: 19:14:27 ERROR unit.mysql-k8s/0.juju-log Failed command: commands=['bash', '-c', 'set -o pipefail; xxtrabackup --defaults-file=/etc/mysql/my.cnf --defaults-group=mysqld --no-version-check --par
allel=12 --user=backups --password=vrEsWPEgo93NN8cv1a12nnZu --socket=/var/run/mysqld/mysqld.sock --lock-ddl --backup --stream=xbstream --xtrabackup-plugin-dir=/usr/lib64/xtrabackup/plugin --target-dir=/var/lib/m
ysql/xtra_backup_Eq2D --no-server-version-check | xbcloud put --curl-retriable-errors=7 --insecure --parallel=10 --md5 --storage=S3 --s3-region=us-east-1 --s3-bucket=mysql-backups-development --s3-endpoint=https
://s3.amazonaws.com --s3-api-version=auto --s3-bucket-lookup=auto mysql-k8s-test/2024-07-31T19:14:25Z'], user='mysql', group='mysql'                                                                               
Traceback (most recent call last):                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/mysql_k8s_helpers.py", line 572, in _execute_commands                                                                                                      
    stdout, stderr = process.wait_output()                                                                                                                                                                         
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/pebble.py", line 1771, in wait_output                                                                                                                 
    raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)                                                                                                                                        
ops.pebble.ExecError: non-zero exit code 1 executing ['bash', '-c', 'set -o pipefail; xxtrabackup --defaults-file=/etc/mysql/my.cnf --defaults-group=mysqld --no-version-check --parallel=12 --user=backups --passw
ord=vrEsWPEgo93NN8cv1a12nnZu --socket=/var/run/mysqld/mysqld.sock --lock-ddl --backup --stream=xbstream --xtrabackup-plugin-dir=/usr/lib64/xtrabackup/plugin --target-dir=/var/lib/mysql/xtra_backup_Eq2D --no-serv
er-version-check | xbcloud put --curl-retriable-errors=7 --insecure --parallel=10 --md5 --storage=S3 --s3-region=us-east-1 --s3-bucket=mysql-backups-development --s3-endpoint=https://s3.amazonaws.com --s3-api-ve
rsion=auto --s3-bucket-lookup=auto mysql-k8s-test/2024-07-31T19:14:25Z'], stdout='', stderr=''                                                                                                                     
unit-mysql-k8s-0: 19:14:27 ERROR unit.mysql-k8s/0.juju-log Failed to execute backup commands                                                                                                                       
Traceback (most recent call last):                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/mysql_k8s_helpers.py", line 572, in _execute_commands                                                                                                      
    stdout, stderr = process.wait_output()                                                                                                                                                                         
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/pebble.py", line 1771, in wait_output                                                                                                                 
    raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)                                                                                                                                        
ops.pebble.ExecError: non-zero exit code 1 executing ['bash', '-c', 'set -o pipefail; xxtrabackup --defaults-file=/etc/mysql/my.cnf --defaults-group=mysqld --no-version-check --parallel=12 --user=backups --passw
ord=vrEsWPEgo93NN8cv1a12nnZu --socket=/var/run/mysqld/mysqld.sock --lock-ddl --backup --stream=xbstream --xtrabackup-plugin-dir=/usr/lib64/xtrabackup/plugin --target-dir=/var/lib/mysql/xtra_backup_Eq2D --no-serv
er-version-check | xbcloud put --curl-retriable-errors=7 --insecure --parallel=10 --md5 --storage=S3 --s3-region=us-east-1 --s3-bucket=mysql-backups-development --s3-endpoint=https://s3.amazonaws.com --s3-api-ve
rsion=auto --s3-bucket-lookup=auto mysql-k8s-test/2024-07-31T19:14:25Z'], stdout='', stderr=''                                                                                                                     
                                                                                                                                                                                                                   
During handling of the above exception, another exception occurred:                                                                                                                                                
                                                                                                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/lib/charms/mysql/v0/mysql.py", line 2452, in execute_backup_commands                                                                                           
    return self._execute_commands(                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 647, in wrapped_function                                                                                       
    return callable(*args, **kwargs)  # type: ignore                                                                                                                                                               
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/mysql_k8s_helpers.py", line 576, in _execute_commands                                                                                                      
    raise MySQLExecError                                                                                                                                                                                           
charms.mysql.v0.mysql.MySQLExecError                                                                                                                                                                               

Solution

Modify code to avoid leaks of any passwords when _execute_command or _run_mysqlcli_script is called. For instance, the above stack trace becomes:

unit-mysql-k8s-0: 20:45:38 ERROR unit.mysql-k8s/0.juju-log Failed command: commands=bash -c set -o pipefail; xxtrabackup --defaults-file=/etc/mysql/my.cnf --defaults-group=mysqld --no-version-check --parallel=12 --user=backups --password=xxxxxxxxxxxx --socket=/var/run/mysqld/mysqld.sock --lock-ddl --backup --stream=xbstream --xtrabackup-plugin-dir=/usr/lib64/xtrabackup/plugin --target-dir=/var/lib/mysql/xtra_backup_g7WB --no-server-version-check | xbcloud put --curl-retriable-errors=7 --insecure --parallel=10 --md5 --storage=S3 --s3-region=us-east-1 --s3-bucket=mysql-backups-development --s3-endpoint=https://s3.amazonaws.com --s3-api-version=auto --s3-bucket-lookup=auto mysql-k8s-test/2024-07-31T20:45:36Z, user='mysql', group='mysql'
unit-mysql-k8s-0: 20:45:38 ERROR unit.mysql-k8s/0.juju-log Failed to execute backup commands
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/lib/charms/mysql/v0/mysql.py", line 2460, in execute_backup_commands
    return self._execute_commands(
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 647, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/mysql_k8s_helpers.py", line 578, in _execute_commands
    raise MySQLExecError from None
charms.mysql.v0.mysql.MySQLExecError

[taurus-forever]

Tnx for taking care!

[carlcsaposs-canonical]

if possible, might be safer to pass password via env variable

if that's not possible, might be safer to never include password in string that will get logged in the first place then to rely on substitution (e.g. https://github.com/canonical/mysql-router-k8s-operator/blob/ae2e8eb634a57807192cbe3946509f434e47dde8/src/mysql_shell/__init__.py#L77-L80)

but definitely an improvement as-is