mc alias list sometimes return the configured alias, sometimes not

[kevinh-canva]

Expected behavior

mcli alias list should always discover a configured alias with mcli alias set.

Actual behavior

Sometimes, mcli alias list works correctly, but sometimes it doesn't show the new configured alias.

Steps to reproduce the behavior

• mcli alias set test-minio http://localhost:9000 <some-root-user> <some-root-password>
• mcli alias list the 1st time

"\"gcs       \\n  URL       : https://storage.googleapis.com\\n  AccessKey : YOUR-ACCESS-KEY-HERE\\n  SecretKey : YOUR-SECRET-KEY-HERE\\n  API       : S3v2\\n  Path      : dns\\nlocal     \\n  URL       : http://localhost:9000\\n  AccessKey : \\n  SecretKey : \\n  API       : \\n  Path      : auto\\nplay      \\n  URL       : https://play.min.io\\n  AccessKey : Q3AM3UQ867SPQQA43P2F\\n  SecretKey : zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG\\n  API       : S3v4\\n  Path      : auto\\ns3        \\n  URL       : https://s3.amazonaws.com\\n  AccessKey : YOUR-ACCESS-KEY-HERE\\n  SecretKey : YOUR-SECRET-KEY-HERE\\n  API       : S3v4\\n  Path      : dns\\ntest-minio\\n  URL       : http://localhost:9000\\n  AccessKey : <REDACTED>\\n  SecretKey : <REDACTED>\\n  API       : s3v4\\n  Path      : auto\"": "{{\"gcs       \\n  URL       : https://storage.googleapis.com\\n  AccessKey : YOUR-ACCESS-KEY-HERE\\n  SecretKey : YOUR-SECRET-KEY-HERE\\n  API       : S3v2\\n  Path      : dns\\nlocal     \\n  URL       : http://localhost:9000\\n  AccessKey : \\n  SecretKey : \\n  API       : \\n  Path      : auto\\nplay      \\n  URL       : https://play.min.io\\n  AccessKey : Q3AM3UQ867SPQQA43P2F\\n  SecretKey : zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG\\n  API       : S3v4\\n  Path      : auto\\ns3        \\n  URL       : https://s3.amazonaws.com\\n  AccessKey : YOUR-ACCESS-KEY-HERE\\n  SecretKey : YOUR-SECRET-KEY-HERE\\n  API       : S3v4\\n  Path      : dns\\ntest-minio\\n  URL       : http://localhost:9000\\n  AccessKey : <REDACTED>\\n  SecretKey : <REDACTED>\\n  API       : s3v4\\n  Path      : auto\"}}"

• mcli alias list the 2nd time

"\"gcs       \\n  URL       : https://storage.googleapis.com\\n  AccessKey : YOUR-ACCESS-KEY-HERE\\n  SecretKey : YOUR-SECRET-KEY-HERE\\n  API       : S3v2\\n  Path      : dns\\nlocal     \\n  URL       : http://localhost:9000\\n  AccessKey : \\n  SecretKey : \\n  API       : \\n  Path      : auto\\nplay      \\n  URL       : https://play.min.io\\n  AccessKey : Q3AM3UQ867SPQQA43P2F\\n  SecretKey : zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG\\n  API       : S3v4\\n  Path      : auto\\ns3        \\n  URL       : https://s3.amazonaws.com\\n  AccessKey : YOUR-ACCESS-KEY-HERE\\n  SecretKey : YOUR-SECRET-KEY-HERE\\n  API       : S3v4\\n  Path      : dns\\ntest-minio\\n  URL       : http://localhost:9000\\n  AccessKey : coder-minio\\n  SecretKey : coder-minio\\n  API       : s3v4\\n  Path      : auto\"": "{{\"gcs       \\n  URL       : https://storage.googleapis.com\\n  AccessKey : YOUR-ACCESS-KEY-HERE\\n  SecretKey : YOUR-SECRET-KEY-HERE\\n  API       : S3v2\\n  Path      : dns\\nlocal     \\n  URL       : http://localhost:9000\\n  AccessKey : \\n  SecretKey : \\n  API       : \\n  Path      : auto\\nplay      \\n  URL       : https://play.min.io\\n  AccessKey : Q3AM3UQ867SPQQA43P2F\\n  SecretKey : zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG\\n  API       : S3v4\\n  Path      : auto\\ns3        \\n  URL       : https://s3.amazonaws.com\\n  AccessKey : YOUR-ACCESS-KEY-HERE\\n  SecretKey : YOUR-SECRET-KEY-HERE\\n  API       : S3v4\\n  Path      : dns\\ntest-minio\\n  URL       : http://localhost:9000\\n  AccessKey : coder-minio\\n  SecretKey : coder-minio\\n  API       : s3v4\\n  Path      : auto\"}}"

NOTE: Sorry in advance for the hard-to-read output, as this is spitted out by Ansible and/or I must have done something wrong. However, you can see in the 1st output, test-minio config is there, but it's no longer in the 2nd one. Also, for some reason the aliases seem to be duplicated in each output (though it could also be that I messes up the way it's formatted somehow).

Sometimes, the 2nd mcli alias list works though. I have verified that both commands run under the same user, and hence should read the config from the same ~/.mcli/config.json (and the fact that sometimes this works suggest it's the same user, otherwise it'd fail everytime).

The result of a wrong mcli alias list, is if I proceed to create a bucket like so: mcli mb --ignore-existing --with-versioning test-minio/test-s3-upload-bucket, it doesn't recognize test-minio as an alias, and instead try to create a directory, which is not what I want.

mc --version

• (paste output of mc --version)

$ mcli --version
mcli version RELEASE.2024-03-30T15-29-52Z (commit-id=9f8147bf0e037730077a1b3baef25e53181099b0)
Runtime: go1.21.8 linux/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>

$ minio --version
minio version RELEASE.2024-03-30T09-41-56Z (commit-id=cb577835d945dbe47f873be96d25caa5a8a858f8)
Runtime: go1.21.8 linux/amd64
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Copyright: 2015-2024 MinIO, Inc.

System information

$ uname -a
Linux test-sync 6.5.0-1020-aws #20~22.04.1-Ubuntu SMP Wed May  1 16:10:50 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

[vadmeste]

@kevinh-canva mostly what is happening is that the server is not ready when your ansible script executes this mcli alias set test-minio http://localhost:9000 <some-root-user> <some-root-password>. mc will probe the server when setting the alias to get some information. You can avoid this by adding --api=s3v4 to mcli alias set command

[kevinh-canva]

Hello @vadmeste 👋 Thanks for responding, but I'm not sure that's the case.mcli alias set is indeed run in the same ansible role that installs the minio server, so what you said maybe true, but the first mcli alias list command is run right after that, and this always show test-minio to be configured successfully (at least in the mcli alias list output, I'm not sure if it's persisted to ~/.mcli/config.json).

The 2nd mcli alias list is run before I attempt to create a bucket with mcli mb, during a systemd ExecStop that fires upon shutdown. That one is the one that doesn't show the test-minio alias

[kevinh-canva]

What you said make me wondering though, is it possible by the time the 2nd mcli alias list and mcli mb is run, minio systemd service is already stopped, and hence these commands failed?

I can understand if that fails the 2nd command, but not sure about mcli alias list, because I thought it will just print out what's in ~/.mcli/config.json. But if it does fire some checks to the endpoint, and if the endpoint is down, it doesn't return the alias, this does make sense to me finally. Can you confirm?

[kevinh-canva]

Ok, I've managed to print out ~/.mcli/config.json. After the mcli alias set command, this is the output

{
    "aliases": {
        "gcs": {
            "accessKey": "YOUR-ACCESS-KEY-HERE",
            "api": "S3v2",
            "path": "dns",
            "secretKey": "YOUR-SECRET-KEY-HERE",
            "url": "https://storage.googleapis.com"
        },
        "local": {
            "accessKey": "",
            "api": "S3v4",
            "path": "auto",
            "secretKey": "",
            "url": "http://localhost:9000"
        },
        "play": {
            "accessKey": "Q3AM3UQ867SPQQA43P2F",
            "api": "S3v4",
            "path": "auto",
            "secretKey": "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG",
            "url": "https://play.min.io"
        },
        "s3": {
            "accessKey": "YOUR-ACCESS-KEY-HERE",
            "api": "S3v4",
            "path": "dns",
            "secretKey": "YOUR-SECRET-KEY-HERE",
            "url": "https://s3.amazonaws.com"
        },
        "test-minio": {
            "accessKey": "<REDACTED>",
            "api": "s3v4",
            "path": "auto",
            "secretKey": "<REDACTED>",
            "url": "http://localhost:9000"
        }
    },
    "version": "10"
}

And then, during the shutdown sequence, this is the output

{
    "aliases": {
        "gcs": {
            "accessKey": "YOUR-ACCESS-KEY-HERE",
            "api": "S3v2",
            "path": "dns",
            "secretKey": "YOUR-SECRET-KEY-HERE",
            "url": "https://storage.googleapis.com"
        },
        "local": {
            "accessKey": "",
            "api": "S3v4",
            "path": "auto",
            "secretKey": "",
            "url": "http://localhost:9000"
        },
        "play": {
            "accessKey": "Q3AM3UQ867SPQQA43P2F",
            "api": "S3v4",
            "path": "auto",
            "secretKey": "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG",
            "url": "https://play.min.io"
        },
        "s3": {
            "accessKey": "YOUR-ACCESS-KEY-HERE",
            "api": "S3v4",
            "path": "dns",
            "secretKey": "YOUR-SECRET-KEY-HERE",
            "url": "https://s3.amazonaws.com"
        }
    },
    "version": "10"
}

test-minio alias has gone AWOL. I already made sure to add Before= and WantedBy= to minio systemd service, so that it starts before, and stops after our ansible playbook systemd service, yet this still happens.

Can we please re-open this issue?

[harshavardhana]

test-minio alias has gone AWOL. I already made sure to add Before= and WantedBy= to minio systemd service, so that it starts before, and stops after our ansible playbook systemd service, yet this still happens.

That doesn't mean the service has started; just doing a systemctl start minio doesn't mean the cluster is up and running.

You need to use mc ready or do health checks to see if mc alias set can be set; otherwise, it will fail.

Your ansible automation is racy, and you are not following what @vadmeste wrote.

mc is a dumb CLI tool that nothing magical writes to config.json if mc alias set gets a 200 OK (success by making an actual API call to the service endpoint)

If you want it to persist, always provide --api s3v4 flag

Can we please re-open this issue?

No

[kevinh-canva]

@harshavardhana

You need to use mc ready or do health checks to see if mc alias set can be set; otherwise, it will fail.

It doesn't fail. mc alias set is successful, a subsequent mc alias list, as well as the output of ~/.mcli/config.json all indicates that it's successful. I can see my alias in all the outputs.

Your ansible automation is racy, and you are not following what @vadmeste wrote.

Forgive me, but at what point did I not follow what he wrote? The suggestion was to add --api so mc alias set is successful, but it already is, so why would I need to add that?

I don't mean to be rude, but I think you don't really follow what I wrote either. What fails is the 2nd mc alias list that was invoked during the shutdown sequence, as it (and ~/.mcli/config.json) no longer shows my test-minio alias. It was never mc alias set that fails.

I just found out that despite Before= and WantedBy=, minio service somehow still stops before my systemd service, and that would explain why mcli mb fail. That is definitely on me and not on the tool, so I will investigate more on my side.

However, for this last run, mcli alias list actually shows test-minio alias, which makes me thought it works this time, but it's not (because as explained above, the service was already dead). It's another inconsistent result.

mc is a dumb CLI tool that nothing magical writes to config.json if mc alias set gets a 200 OK (success by making an actual API call to the service endpoint)

I don't think I follow your wording here at all. Do you mean unless mc alias set gets a 200?

Either way, thanks for the response. I don't understand what actually modifies ~/.mcli/config.json behind the scenes, that leads to inconsistent results in my several test attempts. I thought that once the config is written, it will never change, but maybe I'm wrong here.

[kevinh-canva]

@harshavardhana Please re-open this issue. Adding --api to mc alias set doesn't work, and the alias still disappears from time to time.
Check this out:

devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]: TASK [Print configured aliases] ************************************************
--
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]: Friday 07 June 2024  10:10:09 +1000 (0:00:00.367)       0:00:05.641 ***********
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]: ok: [localhost] => {
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:     "minio_aliases.stdout": "gcs  \n  URL       : https://storage.googleapis.com\n  AccessKey : YOUR-ACCESS-KEY-HERE\n  SecretKey : YOUR-SECRET-KEY-HERE\n  API       : S3v2\n  Path      : dns\nlocal\n  URL       : http://localhost:9000\n  AccessKey : \n  SecretKey : \n  API       : \n  Path      : auto\nplay \n  URL       : https://play.min.io\n  AccessKey : Q3AM3UQ867SPQQA43P2F\n  SecretKey : zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG\n  API       : S3v4\n  Path      : auto\ns3   \n  URL       : https://s3.amazonaws.com\n  AccessKey : YOUR-ACCESS-KEY-HERE\n  SecretKey : YOUR-SECRET-KEY-HERE\n  API       : S3v4\n  Path      : dns"
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]: }
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]: TASK [Read mcli config.json] ***************************************************
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]: Friday 07 June 2024  10:10:09 +1000 (0:00:00.024)       0:00:05.665 ***********
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]: ok: [localhost]
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]: TASK [Print mcli config.json] **************************************************
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]: Friday 07 June 2024  10:10:09 +1000 (0:00:00.042)       0:00:05.707 ***********
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]: ok: [localhost] => {
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:     "mcli_config_json": {
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:         "aliases": {
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:             "gcs": {
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "accessKey": "YOUR-ACCESS-KEY-HERE",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "api": "S3v2",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "path": "dns",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "secretKey": "YOUR-SECRET-KEY-HERE",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "url": "https://storage.googleapis.com"
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:             },
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:             "local": {
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "accessKey": "",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "api": "S3v4",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "path": "auto",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "secretKey": "",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "url": "http://localhost:9000"
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:             },
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:             "play": {
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "accessKey": "Q3AM3UQ867SPQQA43P2F",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "api": "S3v4",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "path": "auto",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "secretKey": "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "url": "https://play.min.io"
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:             },
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:             "s3": {
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "accessKey": "YOUR-ACCESS-KEY-HERE",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "api": "S3v4",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "path": "dns",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "secretKey": "YOUR-SECRET-KEY-HERE",
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:                 "url": "https://s3.amazonaws.com"
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:             }
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:         },
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:         "version": "10"
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]:     }
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]: }
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]: TASK [Get minio service status] ************************************************
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:09 ip-10-20-10-130 sync.sh[5754]: Friday 07 June 2024  10:10:09 +1000 (0:00:00.025)       0:00:05.732 ***********
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:10 ip-10-20-10-130 ansible-ansible.builtin.systemd[6338]: Invoked with name=minio daemon_reload=False daemon_reexec=False scope=system no_block=False state=None enabled=None force=None masked=None
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:10 ip-10-20-10-130 sync.sh[5754]: ok: [localhost]
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:10 ip-10-20-10-130 sync.sh[5754]: TASK [Print minio service status] **********************************************
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:10 ip-10-20-10-130 sync.sh[5754]: Friday 07 June 2024  10:10:10 +1000 (0:00:00.632)       0:00:06.365 ***********
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:10 ip-10-20-10-130 sync.sh[5754]: ok: [localhost] => {
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:10 ip-10-20-10-130 sync.sh[5754]:     "msg": "active (running)"
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:10 ip-10-20-10-130 sync.sh[5754]: }
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:10 ip-10-20-10-130 sync.sh[5754]: TASK [Create minio bucket] *****************************************************
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:10 ip-10-20-10-130 sync.sh[5754]: Friday 07 June 2024  10:10:10 +1000 (0:00:00.022)       0:00:06.388 ***********
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:10 ip-10-20-10-130 ansible-ansible.legacy.command[6365]: Invoked with _raw_params=mcli mb --ignore-existing --with-versioning test-minio/test-s3-upload-bucket _uses_shell=False expand_argument_vars=True stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
  | devbox-validate.amazon-ebs.stable: Jun 07 10:10:10 ip-10-20-10-130 sync.sh[5754]: fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["mcli", "mb", "--ignore-existing", "--with-versioning", "test-minio/test-s3-upload-bucket"], "delta": "0:00:00.032264", "end": "2024-06-07 10:10:10.454462", "msg": "non-zero return code", "rc": 1, "start": "2024-06-07 10:10:10.422198", "stderr": "mcli: <ERROR> Unable to make bucket `test-minio/test-s3-upload-bucket`. mkdir /etc/ansible/test/test-minio: permission denied", "stderr_lines": ["mcli: <ERROR> Unable to make bucket `test-minio/test-s3-upload-bucket`. mkdir /etc/ansible/test/test-minio: permission denied"], "stdout": "", "stdout_lines": []}

The minio.service is still running (as you can see in the check service status output). However, mc alias list and ~/.mcli/config.json doesn't show the alias anymore, even though it was showing the first time.