SFTP upload blocked by "ssh: disconnect, reason 2: too many authentication failures" #1354

mskyttner · 2024-09-25T09:34:41Z

This worked with quay.io/minio/minio:RELEASE.2024-01-16T16-07-38Z but after upgrade to quay.io/minio/minio:RELEASE.2024-09-22T00-33-43Z the following message appears when attempting upload of a file to a bucket using sftp:

minio-1  | MinIO Object Storage Server
minio-1  | Copyright: 2015-2024 MinIO, Inc.
minio-1  | License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
minio-1  | Version: RELEASE.2024-09-22T00-33-43Z (go1.22.7 linux/amd64)
minio-1  | 
minio-1  | API: http://172.19.0.11:9000  http://127.0.0.1:9000 
minio-1  | WebUI: https://data.bibliometrics.lib.kth.se/console/ 
minio-1  | 
minio-1  | Docs: https://docs.min.io
minio-1  | 
minio-1  | API: SYSTEM.sftp
minio-1  | Time: 09:04:22 UTC 09/25/2024
minio-1  | DeploymentID: 7784c0bc-41ad-4eac-9a71-7400a6a97c26
minio-1  | Error: ssh: disconnect, reason 2: too many authentication failures (*ssh.disconnectMsg)
minio-1  |        5: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
minio-1  |        4: internal/logger/logonce.go:149:logger.LogOnceIf()
minio-1  |        3: cmd/logging.go:196:cmd.sftpLogOnceIf()
minio-1  |        2: cmd/sftp-server.go:358:cmd.(*sftpLogger).Error()
minio-1  |        1: github.com/minio/pkg/v3@v3.0.20/sftp/sftp.go:212:sftp.(*Server).handleConnection()

Expected Behavior

Same as when using quay.io/minio/minio:RELEASE.2024-01-16T16-07-38Z, ie not reporting "ssh: too many authentication failures"

Current Behavior

Not possible to upload file using sftp due to "ssh: too many authentication failures". Note that password authentication is used, so we're not attempting to use certificates in line with the new "TrustedCACert"-variant (which honestly in my opinion feels a bit awkward instead of just reading "authorized_keys").

Possible Solution

It looks like the issue has been introduced by any of the PRs related to the "pubkey access" feature , since downgrading to RELEASE.2024-05-01T01-11-10Z (the version just before the one with "pubkey access via SFTP") appears to resolve the issue.

Steps to Reproduce (for bugs)

Use quay.io/minio/minio:RELEASE.2024-05-01T01-11-10Z with password based authentication for SFTP transfers - it works
Upgrade to quay.io/minio/minio:RELEASE.2024-09-22T00-33-43Z - it doesn't work without "ssh: disconnect, reason 2: too many authentication failures"

Context

I am currently stuck at using an earlier version of minio which supports this.

If I upgrade to the latest version, I will run into what looks like a regression starting with versions after the "pubkey access via SFTP"-features were introduced.

Regression

It appears so.

See above.

Your Environment

Containerized minio, information about versions were provided above.

Running on linux.

The text was updated successfully, but these errors were encountered:

zveinn · 2024-09-26T08:27:46Z

Can you share your full upload command ?

mskyttner · 2024-09-26T08:33:51Z

SSHPASS="users_access_key_secret_goes_here" sshpass -e sftp -P 7071 user_account@data.some.minio.server.org:some-bucket

klauspost · 2024-09-26T08:41:44Z

Testing on master

$ SSHPASS="minio123" sshpass -e sftp -P 60021 minio@127.0.0.1:testbucket
Connected to 127.0.0.1.
Changing to: /testbucket
sftp> ls
0.txt   ab.txt  abc
sftp> put minio.exe
Uploading minio.exe to /testbucket/minio.exe
minio.exe                                                                                                                           100%  131MB  39.5MB/s   00:03
sftp> ls
0.txt      ab.txt     abc        minio.exe
sftp>

What happens if you do a manual log-in, with sftp -P 7071 user_account@data.some.minio.server.org:some-bucket - and typing in the password?

On my first try I has to accept the host cert.. maybe that is it?

mskyttner · 2024-09-26T09:48:19Z

I brought up the latest minio using this compose.yaml:

services:
  minio:
    image: quay.io/minio/minio:latest
    command: server /data --console-address ":9001" --sftp="address=:8022" --sftp="ssh-private-key=/home/miniouser/.ssh
/id_rsa"
    environment:
      - MINIO_ROOT_USER=minioadmin
      - MINIO_ROOT_PASSWORD=minioadmin
      - TZ=Europe/Stockholm
    volumes:
      - ./minio_key:/home/miniouser/.ssh/id_rsa:ro
    ports:
      - 9000:9000
      - 9001:9001
      - 8022:8022

First minio_key was generated with ssh-keygen -f minio_key -t rsa and then the service was started.

I created a bucket "test" and uploaded a file there:

mc alias set sftptest http://127.0.0.1:9000 minioadmin minioadmin
mc mb sftptest/test
touch sftptest.txt
mc cp sftptest.txt sftptest/test

No problems so far. Then I attempted to connect using sftp:

sftp -P 8022 -o StrictHostKeyChecking=accept-new miniouser@127.0.0.1:test

Result:

Received disconnect from 127.0.0.1 port 8022:2: too many authentication failures

The log from minio:

minio-1  | ERRO: Calculating checksum failed: open minio: no such file or directory
minio-1  | MinIO Object Storage Server
minio-1  | Copyright: 2015-2024 MinIO, Inc.
minio-1  | License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
minio-1  | Version: RELEASE.2024-09-22T00-33-43Z (go1.22.7 linux/amd64)
minio-1  | 
minio-1  | API: http://172.26.0.2:9000  http://127.0.0.1:9000 
minio-1  | WebUI: http://172.26.0.2:9001 http://127.0.0.1:9001  
minio-1  | 
minio-1  | Docs: https://docs.min.io
minio-1  | WARN: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables
minio-1  | INFO: SFTP Server listening on :8022
minio-1  | 
minio-1  | API: SYSTEM.sftp
minio-1  | Time: 09:34:27 UTC 09/26/2024
minio-1  | DeploymentID: 4d2eb81b-4c9c-45fe-918c-dd68dc668fed
minio-1  | Error: EOF (*errors.errorString)
minio-1  |        5: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
minio-1  |        4: internal/logger/logonce.go:149:logger.LogOnceIf()
minio-1  |        3: cmd/logging.go:196:cmd.sftpLogOnceIf()
minio-1  |        2: cmd/sftp-server.go:358:cmd.(*sftpLogger).Error()
minio-1  |        1: github.com/minio/pkg/v3@v3.0.20/sftp/sftp.go:212:sftp.(*Server).handleConnection()
minio-1  | 
minio-1  | API: SYSTEM.sftp
minio-1  | Time: 09:35:40 UTC 09/26/2024
minio-1  | DeploymentID: 4d2eb81b-4c9c-45fe-918c-dd68dc668fed
minio-1  | Error: ssh: disconnect, reason 2: too many authentication failures (*ssh.disconnectMsg)
minio-1  |        5: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
minio-1  |        4: internal/logger/logonce.go:149:logger.LogOnceIf()
minio-1  |        3: cmd/logging.go:196:cmd.sftpLogOnceIf()
minio-1  |        2: cmd/sftp-server.go:358:cmd.(*sftpLogger).Error()
minio-1  |        1: github.com/minio/pkg/v3@v3.0.20/sftp/sftp.go:212:sftp.(*Server).handleConnection()

Apart from the "ssh: disconnect, reason 2: too many authentication failures" there is also an initial error "ERRO: Calculating checksum failed: open minio: no such file or directory" which I am not sure if I need to act on.

zveinn · 2024-09-26T09:53:27Z

yes, you need to set an absolute path to the minio binary inside the pod. (regarding the checksum error). Let's fix that first and then try again

zveinn · 2024-09-26T09:58:13Z

Also, I'm not an expert in docker services but won't :ro set the key to be read-only from the containers perspective ? does the user which runs minio also have the proper permissions on the key ?

mskyttner · 2024-09-26T10:05:58Z

I think the container does not need any rights to modify the private key file, since key files often have too open permissions, I just from habit put it to be readonly, this is a rootful container anyways, I think?

How can I set an absolute path to the minio binary inside the container? I haven't seen this error message with earlier versions and this is a stock image being used. Some new moves needed for that now?

I can absolutely try to relax the permissions on the key file and fix the absolute path to the minio binary inside the container, if you can provide some clues for how to do the latter. And try again.

zveinn · 2024-09-26T10:17:50Z

yea, I'm trying to set this up myself and whenever I mount my id_rsa key using the same method it thinks it's a directory

services:
  minio:
    image: quay.io/minio/minio:latest
    command: server /data --console-address ":9001"
    environment:
      - MINIO_ROOT_USER=minioadmin
      - MINIO_ROOT_PASSWORD=minioadmin
      - TZ=Europe/Stockholm
    volumes:
      - /home/zveinn/.ssh/id_rsa:/home/miniouser/.ssh/id_rsa:ro
    ports:
      - 9000:9000
      - 9001:9001
      - 8022:8022

bash-5.1# ls -la
total 12
drwxr-xr-x 3 root root 4096 Sep 26 10:03 .
drwxr-xr-x 3 root root 4096 Sep 26 10:15 ..
drwxr-xr-x 2 root root 4096 Sep 26 10:03 id_rsa
bash-5.1# cat id_rsa/
cat: id_rsa/: Is a directory
bash-5.1# pwd
/home/miniouser/.ssh
bash-5.1#

I'm out of my depth when it comes to debugging stuff like this, I'm going to poke someone with more ducker/kubernetes experience for you.

mskyttner · 2024-09-26T10:45:55Z

Using docker compose up -d (or podman) when starting the minio service, the mounted paths are local to the current directory (for the volumes key in the yaml) so for me this mounts the keyfile properly:

    volumes:
      - ./minio_key:/home/miniouser/.ssh/id_rsa:ro

as long as the keyfile is located in the same directory as the compose.yaml file when the service is started.

But I thought absolute paths would be fine too. Perhaps there is some difference between linux and macos when using "docker compose" or "podman compose" with regards to mounting files like that (ie not using "docker volume mounts" but bind mounts directly).

In any case, I checked that this command actually returns the content of the private keyfile, so that shouldn't be an issue:

docker compose exec minio bash -c "cat /home/miniouser/.ssh/id_rsa"

zveinn · 2024-09-26T10:54:03Z

@mskyttner
I just noticed:
sftp -P 8022 -o StrictHostKeyChecking=accept-new miniouser@127.0.0.1:test

miniouser is the wrong username, it's minioadmin ?

mskyttner · 2024-09-26T11:00:00Z

Gah! Yes, so I tried to clean up with docker compose down and recycle with docker compose up -d, but after recreating the test bucket again and uploading a file, I see the same message in the minio log though when using sftp -P 8022 -o StrictHostKeyChecking=accept-new minioadmin@127.0.0.1:test to connect.

zveinn · 2024-09-26T11:09:55Z

zveinn · 2024-09-26T11:14:50Z

I can't seem to reproduce your problem

mskyttner · 2024-09-26T11:15:45Z

It works for me too if using an earlier version of the image:

quay.io/minio/minio                                      latest                         683daefd75a4   2 days ago      165MB
quay.io/minio/minio                                      RELEASE.2024-05-01T01-11-10Z   7da2dcb8bf57   4 months ago    159MB

Are you using the image with the checksum/id 683daefd75a4?

zveinn · 2024-09-26T11:16:56Z

b3e6f75ad212 quay.io/minio/minio:latest "/usr/bin/docker-ent…" 8 minutes ago Exited (0) 9 seconds ago

zveinn · 2024-09-26T11:17:14Z

Pulled the image just a few minute ago

mskyttner · 2024-09-26T11:17:39Z

Ok, strange... not sure then what is going on here.

zveinn · 2024-09-26T11:18:49Z

try deleting the image and pulling it again

mskyttner · 2024-09-26T11:23:16Z

I seem to be on the latest:

$ docker images | grep -P "minio\s+latest"

quay.io/minio/minio                                      latest                         683daefd75a4   2 days ago      165MB

$ docker pull quay.io/minio/minio:latest

latest: Pulling from minio/minio
Digest: sha256:7d80fd232a2f7108aa6f133fcfe5fade3f1626d92d31ae1318076e7aa61928a2
Status: Image is up to date for quay.io/minio/minio:latest
quay.io/minio/minio:latest

zveinn · 2024-09-26T11:27:48Z

you do indeed, then I have no clue.. but it's not an issue with our image at least, I've tripple checked.

zveinn · 2024-09-26T11:28:27Z

I'll leave this issue open for today incase you find something new, but I'll be closing it by the end of day since it's not a bug

mskyttner · 2024-09-26T11:40:16Z

Any hints about how to get rid of the "minio-1 | ERRO: Calculating checksum failed: open minio: no such file or directory"?

zveinn · 2024-09-26T11:43:29Z

I don't think it affects the sftp server.. but it's an issue we fixed internally. Just ignore it for now.
minio/minio#20487

mskyttner · 2024-09-26T12:19:08Z

I can get sftp transfers using password authentication to work non-interactively (on latest minio) by explicitly disabling pubkeyauthentication like this:

# this used to work for earlier versions of minio server, but not any longer...
$ SSHPASS="minioadmin" sshpass -e sftp -P 8022 minioadmin@127.0.0.1:test <<< 'put sftptest.txt'

Received disconnect from 127.0.0.1 port 8022:2: too many authentication failures
Disconnected from 127.0.0.1 port 8022
Connection closed.  
Connection closed

# ... but it works if I explicitly switch off pubkey authentication attempts and forces password authentication only
$ SSHPASS="minioadmin" sshpass -e sftp -P 8022 -o PubkeyAuthentication=no -o PreferredAuthentications=password minioadmin@127.0.0.1:test <<< 'put sftptest.txt'

Connected to 127.0.0.1.
Changing to: /test
sftp> put sftptest.txt
Uploading sftptest.txt to /test/sftptest.txt

So it is maybe not a bug but a difference in behavior.

I think the "too many authentication failures" comes from sftp/ssh attempting all existing keys on the system before it moves on to password authentication and those keys can be many so it never got all the way to the password authentication step as it was hitting "too many authentication failures".

In any case I have a workaround and even if the new behavior is different than before, it does not block me from upgrading to the latest version.

zveinn · 2024-09-26T12:53:04Z

This seems to be a common configuration issue in core sftp, not much we can do here: https://serverfault.com/questions/36291/how-to-recover-from-too-many-authentication-failures-for-user-root

Also, -o PubkeyAuthentication=no should probably be enough to prevent the problem.

Closing this issue.

suside · 2024-10-22T09:46:58Z

Since this issue is the top result for minio sftp authorized_keys, a quick howto for those poor souls who just want to add their pubkey to some ~/.ssh/authorized_keys.
Well in minio you can't do that 😃 Here is what you have to do instead:

Generate keypair for the server: ssh-keygen -f ./ca_user_key
Generate keypair for the user: ssh-keygen -f ./minioadmin
Start the minio with ... --sftp="address=:8022" --sftp="ssh-private-key=ca_user_key" --sftp="trusted-user-ca-key=ca_user_key.pub"
Sign user keypair with server keys: ssh-keygen -s ca_user_key -I minioadmin -n minioadmin -V +9999d -z 1 minioadmin.pub
And now you have minioadmin-cert.pub, this file if crucial and is - sort of - equivalent of ~/.ssh/authorized_keys
From now on having minioadmin and minioadmin-cert.pub in one directory you can just:

$ sftp -i ./minioadmin -oPort=8022 minioadmin@localhost
Connected to localhost.
sftp>

zveinn · 2024-10-22T13:49:32Z

Than you, our docs team will have a look and potentially update the docs.

Adds missing config settings throughout storage class page. Closes #1349 Adds instructions for adding certificates for connecting by sftp. Closes #1354 Corrects information about which key to use for self-signed CAs. Closes #1358 Moves `context` to optional for batch key rotate jobs. Closes #1356

- Adds missing config settings throughout storage class page. Closes #1349 - Adds instructions for adding certificates for connecting by sftp. Closes #1354 - Corrects information about which key to use for self-signed CAs. Closes #1358 - Moves `context` to optional for batch key rotate jobs. Closes #1356 - clean up missing ref links causing build warnings.

mskyttner added community triage Needs triage and scheduling labels Sep 25, 2024

harshavardhana assigned zveinn Sep 26, 2024

harshavardhana added priority: medium and removed triage Needs triage and scheduling labels Sep 26, 2024

zveinn closed this as completed Sep 26, 2024

ravindk89 transferred this issue from minio/minio Oct 22, 2024

ravindk89 reopened this Oct 22, 2024

ravindk89 unassigned zveinn Oct 22, 2024

ravindk89 added priority: low external-contributor and removed priority: medium labels Oct 22, 2024

djwfyi self-assigned this Dec 3, 2024

djwfyi mentioned this issue Dec 3, 2024

Minor fixes for several smaller issues #1380

Merged

djwfyi closed this as completed in #1380 Dec 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SFTP upload blocked by "ssh: disconnect, reason 2: too many authentication failures" #1354

SFTP upload blocked by "ssh: disconnect, reason 2: too many authentication failures" #1354

mskyttner commented Sep 25, 2024

zveinn commented Sep 26, 2024 •

edited

Loading

mskyttner commented Sep 26, 2024

klauspost commented Sep 26, 2024 •

edited

Loading

mskyttner commented Sep 26, 2024

zveinn commented Sep 26, 2024 •

edited

Loading

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024 •

edited

Loading

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024 •

edited

Loading

zveinn commented Sep 26, 2024

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024

zveinn commented Sep 26, 2024

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024

zveinn commented Sep 26, 2024

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024

zveinn commented Sep 26, 2024

suside commented Oct 22, 2024

zveinn commented Oct 22, 2024

SFTP upload blocked by "ssh: disconnect, reason 2: too many authentication failures" #1354

SFTP upload blocked by "ssh: disconnect, reason 2: too many authentication failures" #1354

Comments

mskyttner commented Sep 25, 2024

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Context

Regression

Your Environment

zveinn commented Sep 26, 2024 • edited Loading

mskyttner commented Sep 26, 2024

klauspost commented Sep 26, 2024 • edited Loading

mskyttner commented Sep 26, 2024

zveinn commented Sep 26, 2024 • edited Loading

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024 • edited Loading

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024 • edited Loading

zveinn commented Sep 26, 2024

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024

zveinn commented Sep 26, 2024

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024

zveinn commented Sep 26, 2024

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024

zveinn commented Sep 26, 2024

mskyttner commented Sep 26, 2024

zveinn commented Sep 26, 2024

suside commented Oct 22, 2024

zveinn commented Oct 22, 2024

zveinn commented Sep 26, 2024 •

edited

Loading

klauspost commented Sep 26, 2024 •

edited

Loading

zveinn commented Sep 26, 2024 •

edited

Loading

mskyttner commented Sep 26, 2024 •

edited

Loading

mskyttner commented Sep 26, 2024 •

edited

Loading