exceptionCaught(ServerSessionImpl[user@/10.x.x.x:23232])[state=Opened] IllegalStateException: Bad length (32796) for cmd=SSH_MSG_CHANNEL_DATA - max. allowed=32768 #403
Comments
|
You can increase the maximum packet size for channel windows by changing property |
|
Ty @tomaswolf I did bump up the packet size and it seemed to have worked increasing the limit on server. But client is still sending 28 bytes more than the increased limits. Wondering if additional bytes are related to certain openssh extension/s. I will be debugging more with the server setup config for extensions. |
|
Its been happening with other clients as well. eg. maverick_legacy_1.6.0 |
|
I suppose you mean not the whole mailing list but just the thread starting at https://www.mail-archive.com/users@mina.apache.org/msg06936.html . Yes, that's the same issue. If that client consistently sends 28 bytes more it's likely a client-side bug. From the SSH message structures as defined by the SSH RFC and the SFTP draft RFCs I can't readily see why it should send 28 extra bytes, though. 41 or 4 I could understand. |
|
Since you mentioned maverick_legacy, I've tried to reproduce this. Simple SFTP test server using Apache MINA sshd, and a simple SFTP client using maverick uploading a 100KB file. I cannot reproduce this. I tried both j2ssh-maverick:1.5.5 and maverick-legacy:1.6.0 (with maverick-common:1.0.1) against both Apache MINA 2.9.2 and 2.10.0. Apart from the fact that I had to enable some long deprecated KEX algorithms and allow small keys in my test server for maverick-legacy: the SFTP file upload worked fine in all combinations. (All running on localhost on OS X.) |
|
Related:
reported in 2022 for WS_FTP 12.6, and in 2023 for 12.7 and 12.8. Seems to be specific to WS_FTP, which I don't have. @dragonknight88 : if you can live debug the server, a breakpoint where that exception is raised in Also, it would be interesting to know when exactly it occurs. What messages were exchanged over the channel already when the exception occurs? A debug log of the server would help. |
|
hi @tomaswolf it seem to be failing in Surprisingly client works without an issue with OpenSSH server. I have been testing server behavior with different packet limits for below props on server init. Unsure if mina buffer validation is active by default or this enforces the buffer size validation check. |
|
Sorry, this log doesn't help. It seems to report stuff only after the exception has already occurred and the channel has been closed. Does this occur also with Apache MINA sshd 2.10.0? How big is the file being sent? (Or if you don't know: how much data was already received?) What's the log before that? From the start of the SFTP connection. |
|
Occurs on 2.10.0 as well. Lmk if this helps or if you have a sftp server running over the internet; which I could use to connect with the sftp client and test user for the whole logs on server side. |
|
Thanks, that helps. I was a bit confused for a moment by the log being chronologically descending, but that's OK. The interesting bit are these two lines: This shows the SSH message itself is internally consistent. We have
Next comes the data itself, which is an SFTP write request:
Also visible from the log: it's the first SSH_FXP_WRITE request for this file (there's an SSH_FXP_OPEN request just before). So whatever that client is doing, it really is sending 28 bytes too many. I do notice that 28 = 32 - 4. OpenSSH returns handles that have only 4 bytes. Apache MINA sshd uses by default handles of 16 bytes AFAIK, and it stringifies these 16 bytes, so we end up with 32 bytes (one byte per hex character). The SFTP draft RFCs say that SFTP file handles may be up to 256 bytes long. If you set the handle size in the Apache MINA sshd server to 8, we should end up with 16 hex chars. Does that client then send 12 extra bytes? And if the handle size is 20, we should have 40 bytes, does it then send 36 extra bytes? (For the configuration, see If so, we'd have established that this SFTP client can only handle SFTP file handles of 4 bytes. For larger handles, its length logic for SFTP write requests (SSH_FXP_WRITE) is wrong. This would also explain why it works with OpenSSH: an OpenSSH server will use a handle size of 4 bytes. |
|
All that said: I find the file handle implementation in Apache MINA sshd a little bit strange. I don't know why it is done the way it is done; it strikes me as overly complicated. I see absolutely no reason why a file handle should be longer than 4 bytes, be it in memory or in the network packet. So even if I think we may have found a bug in WS_FTP, I also think the Apache MINA sshd implementation could be simplified and use by default a handle size of 4 bytes. |
|
Ack! I was able to successfully test by setting FILE_HANDLE_SIZE to 4; and testing it for 16 and 40. Other thing I noticed is server dint have it configured to 32 before but mina was somehow defaulting file-handle to be 32. Going through code it is still unclear where it defaults to 32 vs 16 bytes. file-handle-limits-default Initially I suspected client may be sending corrupted file-handle back; but that dint seem to be the case. |
|
Yes, that was my mistake when reading the code. File handle size 16 means it uses 16 bytes, but then converts these to a hex string, which gives 32 characters, and thus we have with Now that this is confirmed:
|
It was reported by me :) I will add more info on RFC (file handle, 4-258 byte limits and buffer calc.)
Meanwhile, a possible quick fix could be made on Validation condition - AbstractChannel.java#L856 as it appears +4L confusion here is related to fileHandle * 2 byte count you mentioned above. And partly the reason why file handle size of 4 succeeds without validation failure. Thank you so much Tomas for taking the time and helping out on the issue. |
Yes, I'll also do something like that. Except that this validation is in the connection protocol (RFC 4254), and fileHandleSize is known only in the higher-layer SFTP protocol, so a bit more is needed to implement this in a proper way without layer breaks. We'd want to accept this extra size only in SFTP channels, and only if it's exactly this expected extra amount of bytes. |
|
BTW, the "+4 confusion" is this: https://mailarchive.ietf.org/arch/msg/secsh/mIvfsnrukzaIvUBah5RJjDb3yyQ/ It's basically unrelated to this handle size problem. But it makes the SFTP write from WS_FTP work if the handle size in Apache MINA sshd is set to 4: then we get 8 bytes of actual handle, and WS_FTP will send exactly 4 bytes too many, which Apache MINA sshd will accept, but for the wrong reason as it's not this "+4 confusion" but a bug at another level that just happens to also cause exactly 4 extra bytes. |
If the file handle size in Apache MINA sshd is > 4, WS_FTP client <= 12.9 sends (fileHandleSize - 4) too many bytes in SSH_FXP_WRITE requests. If that exceeds the maximum packet size of the channel's local window, the channel packet is invalid and Apache MINA sshd correctly throws an exception and closes the channel. Work around this by allowing for this special case on the server side in channels used for SFTP. Bug: apache#403
Change the file handle implementation: use raw bytes, not hexified bytes. Representing file handles internally as hexified strings doubled the size, so with the default setting of 16, an Apache MINA file handle was actually 32 bytes on the network. SftpModuleProperties.FILE_HANDLE_SIZE specified that its the size in bytes of the file handle; so the final handle as sent to the client should not be twice the configured size. Because many APIs in Apache MINA sshd use String as the type to pass around such file handles, we cannot easily change that without potentially breaking a lot of user code. So keep storing file handles as strings, but use ISO-8859-1 to convert: this lets us store any byte value as a character. When printing file handles, for instance in log messages, hexify them explicitly. Bug: apache#403
Change the default size for SFTP file handles to 4 bytes, and change the implementation such that it produces collision-free 4-byte handles. The algorithm for longer handles is left unchanged. Using smaller SFTP file handles wastes less space in the SSH packets, which can thus be used to send actual data. OpenSSH also uses 4-byte SFTP file handles; using the same size also helps avoid strange bugs in SFTP clients that might not always handle larger file handles correctly. Bug: apache#403
Change the file handle implementation: use raw bytes, not hexified bytes. Representing file handles internally as hexified strings doubled the size, so with the default setting of 16, an Apache MINA file handle was actually 32 bytes on the network. SftpModuleProperties.FILE_HANDLE_SIZE specified that its the size in bytes of the file handle; so the final handle as sent to the client should not be twice the configured size. Because many APIs in Apache MINA sshd use String as the type to pass around such file handles, we cannot easily change that without potentially breaking a lot of user code. So keep storing file handles as strings, but use ISO-8859-1 to convert: this lets us store any byte value as a character. When printing file handles, for instance in log messages, hexify them explicitly. Bug: apache#403
Change the default size for SFTP file handles to 4 bytes, and change the implementation such that it produces collision-free 4-byte handles. The algorithm for longer handles is left unchanged. Using smaller SFTP file handles wastes less space in the SSH packets, which can thus be used to send actual data. OpenSSH also uses 4-byte SFTP file handles; using the same size also helps avoid strange bugs in SFTP clients that might not always handle larger file handles correctly. Bug: apache#403
If the file handle size in Apache MINA sshd is > 4, WS_FTP client <= 12.9 sends (fileHandleSize - 4) too many bytes in SSH_FXP_WRITE requests. If that exceeds the maximum packet size of the channel's local window, the channel packet is invalid and Apache MINA sshd correctly throws an exception and closes the channel. Work around this by allowing for this special case on the server side in channels used for SFTP. Bug: apache#403
Change the file handle implementation: use raw bytes, not hexified bytes. Representing file handles internally as hexified strings doubled the size, so with the default setting of 16, an Apache MINA file handle was actually 32 bytes on the network. SftpModuleProperties.FILE_HANDLE_SIZE specified that its the size in bytes of the file handle; so the final handle as sent to the client should not be twice the configured size. Because many APIs in Apache MINA sshd use String as the type to pass around such file handles, we cannot easily change that without potentially breaking a lot of user code. So keep storing file handles as strings, but use ISO-8859-1 to convert: this lets us store any byte value as a character. When printing file handles, for instance in log messages, hexify them explicitly. Bug: apache#403
Change the default size for SFTP file handles to 4 bytes, and change the implementation such that it produces collision-free 4-byte handles. The algorithm for longer handles is left unchanged. Using smaller SFTP file handles wastes less space in the SSH packets, which can thus be used to send actual data. OpenSSH also uses 4-byte SFTP file handles; using the same size also helps avoid strange bugs in SFTP clients that might not always handle larger file handles correctly. Bug: apache#403
|
@dragonknight88 : PR #405 has been merged. Using the 2.10.1-SNAPSHOT release from the Apache Snapshot maven repository you could test whether it really works with WS_FTP with different handle sizes. If not, feel free to re-open this. Ultimately it would be good if the vendor of WS_FTP fixed that bug in their SFTP client. |
Client: WS_FTP 1.26/1.27
Server: SFTP server (mina 2.9.2)
Operation: Uploading a file
Mina version - 2.9.2
As sftp client tries to upload a file channel is closed as below exception is caught.
2023-08-10 10:16:31 2023-08-10T17:16:31.119Z WARN 337 --- [] [] [] [)-nio2-thread-3] o.a.s.server.session.ServerSessionImpl : exceptionCaught(ServerSessionImpl[user@/10.x.x.x:23232])[state=Opened] IllegalStateException: Bad length (32796) for cmd=SSH_MSG_CHANNEL_DATA - max. allowed=32768Is there a way or config via core module props to increase the channel packet size limit?
The text was updated successfully, but these errors were encountered: