sshd-sftp - Not able to get files from windows server

[learncourse2024]

Version

2.11.0

Bug description

sshd-sftp 2.11.0 version - cannot list files in a directory when connecting to windows sftp server(windows server 2019)

Actual behavior

We are using sshd-sftp 2.11.0 version to connect to SFTP servers and do transfer of files from SFTP to AWS S3
It works fine when connecting to linux based SFTP servers. But we are facing issues when connecting to windows based sftp servers.
SftpClientDirectoryScanner.scandir() does not list files in the directory.
On debugging we observed that the code - attrs.isRegularFile() returns false as it returns getPermissions() as 438 even when the file permissions are rw-rw-rw-

Expected behavior

The files should be listed from the respective directory

Relevant log output

No response

Other information

No response

[tomaswolf]

Decimal 438 is octal 666 (hex 1B6) and thus reflects the rw-rw-rw- permissions correctly. But it looks as if the server does not return the file type flags. It's supposed to return in bits 12-15 (mask 0xf000) bit combinations that tell what kind of file (regular file, link, directory, ...) it is.

Granted, for SFTP version 3 the draft RFC is a bit unclear about this as it only says

The `permissions' field contains a bit mask of file permissions as defined by posix.

From SFTP version 4 on, the draft RFCs have an extra field for the file type.

What kind of SFTP server is running on that Windows Server?

You could also try setting a breakpoint where we parse these flags from the server's directory listing and then stepping into readAttributes. Perhaps there's a bug there.

[learncourse2024]

Its SFTP version 3
On debugging i see it is set to SftpConstants.SSH_FILEXFER_TYPE_UNKNOWN

mina-sshd/sshd-sftp/src/main/java/org/apache/sshd/sftp/common/SftpHelper.java

Line 549 in 69b64da

return SftpConstants.SSH_FILEXFER_TYPE_UNKNOWN;

[tomaswolf]

Yes, if the server doesn't send any of these flags then the file type is unknown.

What kind of SFTP server is this?

[learncourse2024]

SolarWinds Sftp Server
File transfer protocol = SFTP-3
Cryptographic protocol = SSH-2
SSH implementation = WeOnlyDo 2.5.5
Compression = No

[tomaswolf]

Possibly that SFTP server has a bug?

It's a bit unclear what we could do. Probably the directory scanner should not return only directories and regular files but also do something meaningful for the other types. Then we'd at least list something.

Remains the question what to do with "unknown" file types. What do other SFTP clients display? If you have on that server a directory that contains subdirectories and files, do FileZilla or WinSCP correctly show the items as directories or files? If so, we could look what those programs do to figure it out. One option might be to parse the "longname", even if the draft RFC says

However, clients SHOULD NOT attempt to parse the longname field for file attributes; they SHOULD use the attrs field instead.

Provided that SFTP server fills in the "longname" field reasonably. What are the complete buffer contents (raw bytes) at a breakpoint at the location I had indicated? (From the buffer's reading position rpos on, up to its wpos.)

[learncourse2024]

So here are the details which i found while debugging
LongName - -rw-rw-rw 1 root root 1944 Apr 24 14:58 demo.csv

ByteArrayBuffer[rpos=288, wpos=320, size=384]

[0, 0, 1, 60, 104, 0, 0, 0, 104, 0, 0, 0, 3, 0, 0, 0, 1, 46, 0, 0, 0, 56, 100, 114, 119, 45, 114, 119, 45, 114, 119, 32, 32, 32, 49, 32, 32, 32, 32, 32, 114, 111, 111, 116, 32, 32, 32, 32, 32, 114, 111, 111, 116, 32, 32, 32, 32, 32, 32, 32, 32, 32, 48, 32, 65, 112, 114, 32, 50, 52, 32, 49, 52, 58, 53, 56, 32, 46, 0, 0, 0, 15, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 65, -74, 102, 41, 82, -9, 102, 41, 86, 57, 0, 0, 0, 2, 46, 46, 0, 0, 0, 57, 100, 114, 119, 45, 114, 119, 45, 114, 119, 32, 32, 32, 49, 32, 32, 32, 32, 32, 114, 111, 111, 116, 32, 32, 32, 32, 32, 114, 111, 111, 116, 32, 32, 32, 32, 32, 32, 32, 32, 32, 48, 32, 65, 112, 114, 32, 50, 52, 32, 49, 52, 58, 53, 56, 32, 46, 46, 0, 0, 0, 15, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 65, -74, 102, 41, 82, -9, 102, 41, 86, 57, 0, 0, 0, 8, 100, 101, 109, 111, 46, 99, 115, 118, 0, 0, 0, 63, 45, 114, 119, 45, 114, 119, 45, 114, 119, 32, 32, 32, 49, 32, 32, 32, 32, 32, 114, 111, 111, 116, 32, 32, 32, 32, 32, 114, 111, 111, 116, 32, 32, 32, 32, 32, 32, 49, 57, 52, 52, 32, 65, 112, 114, 32, 50, 52, 32, 49, 52, 58, 53, 56, 32, 100, 101, 109, 111, 46, 99, 115, 118, 0, 0, 0, 15, 0, 0, 0, 0, 0, 0, 7, -104, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, -74, 102, 41, 86, 57, 102, 41, 86, 57, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

I tried to see on filezilla too

[tomaswolf]

OK. Findings:

1. This server sends longname as -rw-rw-rw 1 root root 1944 Apr 24 14:58 demo.csv (just before offset 288)
2. This is then followed by flags 0x00_00_00_0F: we have size, uid/gid, permissions, and access and modification time.
   • Size is 0x00_00_00_00_00_00_07_98, i.e. 1944 bytes
   • UID = 0x00_00_00_00, GID = 0x00_00_00_00
   • Permissions = 0x00_00_01_B6 (438, rw-rw-rw-)
   • Access and modification times (both are equal, two 4-byte unsigned ints, seconds since Unix Epoch)

So far, so bad. The server indeed does not send the "file type" flags in "permissions". And moreover, its "longname" looks strange. I would have expected it to start with "-rw-rw-rw-", not with "-rw-rw-rw ".

WinSCP indeed parses "longname". It assumes that the first character of the longname gives the item type: '-' for a regular file, 'l' for a symbolic link, and 'd' for a directory (case insensitive).

I suppose we should do so, too, at least if it's SFTP V3 and the permissions don't give the file type.

[tomaswolf]

... its "longname" looks strange. I would have expected it to start with "-rw-rw-rw-", not with "-rw-rw-rw ". ...

This was mentioned in a SolarWinds forum back in 2019 already. Apparently it was not fixed. It doesn't matter for us since the attributes do give the correct permission bits. They just lack the file type, which we could get from the first character of "longname",

[learncourse2024]

Thank you @tomaswolf for explaining it.
Will there be any fix for this issue which will be worked on in MINA

[tomaswolf]

@learncourse2024 : please try the snapshot release from the Apache Snapshots repository. It should be able to handle this case better.

[learncourse2024]

Thank you @tomaswolf for the fix. We verified with snapshot version. It works!!
Any approximate date when will next release be?