Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

openwrt dropbear ssh timeout #193

Closed
yuhongwei380 opened this issue Nov 1, 2022 · 21 comments
Closed

openwrt dropbear ssh timeout #193

yuhongwei380 opened this issue Nov 1, 2022 · 21 comments

Comments

@yuhongwei380
Copy link

last time,we use dropbear
2020.81-3 version,we can ssh to my remote openwrt across ipsec-vpn; but this time ,i upgrade my openwrt, and fine the dropbear version is update to 2022.82 ;we can't ssh to my remote openwrt ;but between remote sever and remote openwrt , it can work ;so i think maybe IPQos influences this ?

@mkj
Copy link
Owner

mkj commented Nov 1, 2022

I haven't had any reports of this, though it's possible. As far as I know AF21 (used by dropbear) is fairly normal though.
Can you send logs of the ssh client getting stuck?

You should be able to disable the qos if you run on the openwrt device:
nft add rule inet filter output ip dscp set cs0

A packet capture with wireshark/tcpdump might have some clues too.

@yuhongwei380
Copy link
Author

yuhongwei380 commented Nov 1, 2022

image
like this ; when i want to ssh my remote openwrt(dropbear 22.82) over ipsec-vpn; it is hanged and noticed time out ; i try to switch my old version (20.81-3) ,it is work well , i can ssh my remote openwrt over ipsec-vpn ;
Looking forward to your reply,Thank you!

@yuhongwei380
Copy link
Author

i am sure it is AF21 influenced this ssh problem samed as openssh 8.0 ; In OpenSSH version 8.1+ a DSCP value of AF21 is appended by default. This seems to break communication in some environments.
so how can i revise the default dscp setting on openwrt ;
image
it does not work ,may this command is wrong?

@yuhongwei380
Copy link
Author

image
i find it AF21 in the dropbear 2022.82 release-note

@mkj
Copy link
Owner

mkj commented Nov 2, 2022

OK yep, seems likely AF21 is the cause if OpenSSH 8.1 shows the same thing.
Do you know which ipsec vpn software/hardware is being used? It might be worth seeing if it can be fixed there, given newer OpenSSH will have trouble too.

AF21 is hardcoded for Dropbear, you can disable it by commenting out (or setting to 0 instead)

val = 0x48; /* IPTOS_DSCP_AF21 */

I'll make it a setting for localoptions.h in future.

I'm not sure why that nft line didn't work, I tested the commandline here on my laptop. Is your openwrt system using iptables or nft? If it's iptables it will need a different command.

@yuhongwei380
Copy link
Author

my openwrt may used iptables ; i can find iptables;

@yuhongwei380
Copy link
Author

my ipsec-vpn is on the firewall device ; two huawei USG device ; i did not setting any qos in the ipsec-vpn

@yuhongwei380
Copy link
Author

yeap, i try to find any problem or setting in my firewall device ; but in remote ubuntu ; ssh version is 8.0; but it works well . So i am so confused ; last time we met this problem is in my remote centos8 linux server ; we add IPQos=thoughout in sshd_config .

@mkj
Copy link
Owner

mkj commented Nov 2, 2022

Debian (which Ubuntu follows) patched their OpenSSH to use the old IPQoS=cs1 setting , see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923879#17

Perhaps it's a bug in the Huawei USG device, a few other things like vmware were also incorrectly handling DSCP.

For iptables this might work?
iptables -t mangle -A OUTPUT -j DSCP --set-dscp 0

@yuhongwei380
Copy link
Author

image

@mkj
Copy link
Owner

mkj commented Nov 2, 2022 via email

@ynezz
Copy link
Contributor

ynezz commented Nov 2, 2022

I'll make it a setting for localoptions.h in future.

FYI for openwrt/openwrt#10405 I've created following workaround.

@mkj
Copy link
Owner

mkj commented Nov 2, 2022

FYI for openwrt/openwrt#10405 I've created following workaround.

Thanks @ynezz , I hadn't seen those problems - I'd assumed OpenSSH had already shaken out any wonky network equipment!

@yuhongwei380
Copy link
Author

so i suggest next dropbear version should be the default ?

@ynezz
Copy link
Contributor

ynezz commented Nov 2, 2022

so i suggest next dropbear version should be the default ?

Well, this is not an issue in dropbear, its just exposing broken QoS support on your device, so IMO you've following options:

  • disable the QoS in dropbear by using above suggested workarounds
  • fix QoS support on your target/device

@mkj
Copy link
Owner

mkj commented Nov 2, 2022 via email

@ynezz
Copy link
Contributor

ynezz commented Nov 2, 2022

I'll add the runtime option to let it be disabled in problematic networks/devices.

Its your call :) I've not prepared PR for this simply because we've decided, that dropbear is critical element in the network infra, so any such bloat might has negative influence on its security.

In the case of mt76 it looks like it was an unrelated driver bug that happened to trigger with AF21?

Correct.

@mkj
Copy link
Owner

mkj commented Nov 2, 2022

Its your call :) I've not prepared PR for this simply because we've decided, that dropbear is critical element in the network infra, so any such bloat might has negative influence on its security.

Yeah, normally I'd be reluctant to add workarounds for broken things, but the history of TOS/DSCP flag handling is messy and there are probably a few other broken network devices that are harder to change. It shouldn't add too much size.

@yuhongwei380
Copy link
Author

I think I'll leave the default as AF21, that is the proper way so cake shaper etc can prioritise interactive SSH sessions when traffic is heavy. I'll add the runtime option to let it be disabled in problematic networks/devices. In the case of mt76 it looks like it was an unrelated driver bug that happened to trigger with AF21? openwrt/mt76@5ec78e1

so ,thats mean , you can provide a option to setting default when we met those ipqos problem?

@mkj
Copy link
Owner

mkj commented Nov 2, 2022

Yes, I'll add a flag that can be added to dropbear and dbclient.

@yuhongwei380
Copy link
Author

Thank you very much!Looking forward the next version of dropbear~ Now we can use the old version

@mkj mkj closed this as completed in 86ac841 Nov 10, 2022
freebsd-git pushed a commit to freebsd/freebsd-ports that referenced this issue Nov 15, 2022
Features and Changes:
  Note >> for compatibility/configuration changes

- >> Disable DROPBEAR_DSS by default
  It is only 1024 bit and uses sha1, most distros disable it by default already.

- Added DROPBEAR_RSA_SHA1 option to allow disabling sha1 rsa signatures.
  >> RSA with sha1 will be disabled in a future release (rsa keys will continue
  to work OK, with sha256 signatures used instead).

- Add option for requiring both password and pubkey (-t)
  Patch from Jackkal

- Add 'no-touch-required' and 'verify-required' options for sk keys
  Patch from Egor Duda

  - >> DROPBEAR_SK_KEYS config option now replaces separate DROPBEAR_SK_ECDSA
  and DROPBEAR_SK_ED25519 options.

- Add 'permitopen' option for authorized_keys to restrict forwarded ports
  Patch from Tuomas Haikarainen

- >> Added LTM_CFLAGS configure argument to set flags for building
  bundled libtommath. This also restores the previous arguments used
  in 2020.81 (-O3 -funroll-loops). That gives a big speedup for RSA
  key generation, which regressed in 2022.82.
  There is a tradeoff with code size, so -Os can be used if required.
  mkj/dropbear#174
  Reported by David Bernard

- Add '-z' flag to disable setting QoS traffic class. This may be necessary
  to work with broken networks or network drivers, exposed after changes to use
  AF21 in 2022.82
  mkj/dropbear#193
  Reported by yuhongwei380, patch from Petr Štetiar

- Allow overriding user shells with COMPAT_USER_SHELLS
  Based on a patch from Matt Robinson

- Improve permission error message
  Patch from k-kurematsu

- >> Remove HMAC_MD5 entirely

Regression fixes from 2022.82:

- Fix X11 build

- Fix build warning

- Fix compilation when disabling pubkey authentication
  Patch from MaxMougg

- Fix MAX_UNAUTH_CLIENTS regression
  Reported by ptpt52

- Avoid using slower prime testing in bundled libtomcrypt when DSS is disabled
  mkj/dropbear#174
  Suggested by Steffen Jaeckel

- Fix Dropbear plugin support
  mkj/dropbear#194
  Reported by Struan Bartlett

Other fixes:

- Fix long standing incorrect compression size check. Dropbear
  (client or server) would erroneously exit with
  "bad packet, oversized decompressed"
  when receiving a compressed packet of exactly the maximum size.

- Fix missing setsid() removed in 2020.79
  mkj/dropbear#180
  Reported and debugged by m5jt and David Bernard

- Try keyboard-interactive auth before password, in dbclient.
  This was unintentionally changed back in 2013
  mkj/dropbear#190
  Patch from Michele Giacomoli

- Drain the terminal when reading the fingerprint confirmation response
  mkj/dropbear#191
  Patch from Michele Giacomoli

- Fix utx wtmp variable typo. This has been wrong for a long time but
  only recently became a problem when wtmp was detected.
  mkj/dropbear#189
  Patch from Michele Giacomoli

- Improve configure test for hardening options.
  Fixes building on AIX
  mkj/dropbear#158

- Fix debian/dropbear.init newline
  From wulei-student

Infrastructure:

- Test off-by-default compile options
- Set -Wundef to catch typos in #if statements
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Dec 20, 2023
Include terrapin fix and bump PKGREVISION to make clear this
is not 2022.83.

2022.83 - 14 November 2022

Features and Changes:
  Note >> for compatibility/configuration changes

- >> Disable DROPBEAR_DSS by default
  It is only 1024 bit and uses sha1, most distros disable it by default already.

- Added DROPBEAR_RSA_SHA1 option to allow disabling sha1 rsa signatures.
  >> RSA with sha1 will be disabled in a future release (rsa keys will continue
  to work OK, with sha256 signatures used instead).

- Add option for requiring both password and pubkey (-t)
  Patch from Jackkal

- Add 'no-touch-required' and 'verify-required' options for sk keys
  Patch from Egor Duda

  - >> DROPBEAR_SK_KEYS config option now replaces separate DROPBEAR_SK_ECDSA
  and DROPBEAR_SK_ED25519 options.

- Add 'permitopen' option for authorized_keys to restrict forwarded ports
  Patch from Tuomas Haikarainen

- >> Added LTM_CFLAGS configure argument to set flags for building
  bundled libtommath. This also restores the previous arguments used
  in 2020.81 (-O3 -funroll-loops). That gives a big speedup for RSA
  key generation, which regressed in 2022.82.
  There is a tradeoff with code size, so -Os can be used if required.
  mkj/dropbear#174
  Reported by David Bernard

- Add '-z' flag to disable setting QoS traffic class. This may be necessary
  to work with broken networks or network drivers, exposed after changes to use
  AF21 in 2022.82
  mkj/dropbear#193
  Reported by yuhongwei380, patch from Petr Štetiar

- Allow overriding user shells with COMPAT_USER_SHELLS
  Based on a patch from Matt Robinson

- Improve permission error message
  Patch from k-kurematsu

- >> Remove HMAC_MD5 entirely

Regression fixes from 2022.82:

- Fix X11 build

- Fix build warning

- Fix compilation when disabling pubkey authentication
  Patch from MaxMougg

- Fix MAX_UNAUTH_CLIENTS regression
  Reported by ptpt52

- Avoid using slower prime testing in bundled libtomcrypt when DSS is disabled
  mkj/dropbear#174
  Suggested by Steffen Jaeckel

- Fix Dropbear plugin support
  mkj/dropbear#194
  Reported by Struan Bartlett

Other fixes:

- Fix long standing incorrect compression size check. Dropbear
  (client or server) would erroneously exit with
  "bad packet, oversized decompressed"
  when receiving a compressed packet of exactly the maximum size.

- Fix missing setsid() removed in 2020.79
  mkj/dropbear#180
  Reported and debugged by m5jt and David Bernard

- Try keyboard-interactive auth before password, in dbclient.
  This was unintentionally changed back in 2013
  mkj/dropbear#190
  Patch from Michele Giacomoli

- Drain the terminal when reading the fingerprint confirmation response
  mkj/dropbear#191
  Patch from Michele Giacomoli

- Fix utx wtmp variable typo. This has been wrong for a long time but
  only recently became a problem when wtmp was detected.
  mkj/dropbear#189
  Patch from Michele Giacomoli

- Improve configure test for hardening options.
  Fixes building on AIX
  mkj/dropbear#158

- Fix debian/dropbear.init newline
  From wulei-student

Infrastructure:

- Test off-by-default compile options

- Set -Wundef to catch typos in #if statements


2022.82 - 1 April 2022

Features and Changes:
  Note >> for compatibility/configuration changes

- Implemented OpenSSH format private key handling for dropbearconvert.
  Keys can be read in OpenSSH format or the old PEM format.
  >> Keys are now written in OpenSSH format rather than PEM.
  ED25519 support is now correct. DSS keys are still PEM format.

- Use SHA256 for key fingerprints

- >> Reworked -v verbose printing, specifying multiple times will increase
  verbosity. -vvvv is equivalent to the old DEBUG_TRACE -v level, it
  can be configured at compile time in localoptions.h (see default_options.h)
  Lower -v options can be used to check connection progress or algorithm
  negotiation.
  Thanks to Hans Harder for the implementation

  localoptions.h DEBUG_TRACE should be set to 4 for the same result as the
  previous DEBUG_TRACE 1.

- Added server support for U2F/FIDO keys (ecdsa-sk and ed25519-sk) in
  authorized_keys. no-touch-required option isn't allowed yet.
  Thanks to Egor Duda for the implementation

- autoconf output (configure script etc) is now committed to version control.
  >> It isn't necessary to run "autoconf" any more on a checkout.

- sha1 will be omitted from the build if KEX/signing/MAC algorithms don't
  require it. Instead sha256 is used for random number generation.
  See sysoptions.h to see which algorithms require which hashes.

- Set SSH_PUBKEYINFO environment variable based on the authorized_keys
  entry used for auth. The first word of the comment after the key is used
  (must only have characters a-z A-Z 0-9 .,_-+@)
  Patch from Hans Harder, modified by Matt Johnston

- Let dbclient multihop mode be used with '-J'.
  Patch from Hans Harder

- Allow home-directory relative paths ~/path for various settings
  and command line options.
  *_PRIV_FILENAME DROPBEAR_PIDFILE SFTPSERVER_PATH MOTD_FILENAME
  Thanks to Begley Brothers Inc

  >> The default DROPBEAR_DEFAULT_CLI_AUTHKEY has now changed, it now needs
  a tilde prefix.

- LANG environment variable is carried over from the Dropbear server process
  From Maxim Kochetkov

- Add /usr/sbin and /sbin to $PATH when logging in as root.
  Patch from Raphaël Hertzog
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903403

- Added client option "-o DisableTrivialAuth". It disallows a server immediately
  giving successful authentication (without presenting any password/pubkey prompt).
  This avoids a UI confusion issue where it may appear that the user is accepting
  a SSH agent prompt from their local machine, but are actually accepting a prompt
  sent immediately by the remote server.
  CVE-2021-36369 though the description there is a bit confused. It only applies
  to Dropbear as a client.
  Thanks to Manfred Kaiser from Austrian MilCERT

- Add -q client option to hide remote banner, from Hans Harder

- Add -e option to pass all server environment variables to child processes.
  This should be used with caution.
  Patch from Roland Vollgraf (github #118)

- >> Use DSCP for QoS traffic classes. Priority (tty) traffic is now set to
  AF21 "interactive". Previously TOS classes were used, they are not used by
  modern traffic classifiers. Non-tty traffic is left at default priority.

- >> Disable dh-group1 key exchange by default. It has been disabled server
  side by default since 2018.

- >> Removed Twofish cipher

Fixes:

- Fix flushing channel data when pty was allocated (github #85)
  Data wasn't completely transmitted at channel close.
  Reported and initial patch thanks to Yousong Zhou

- Dropbear now re-executes itself rather than just forking for each connection
  (only on Linux). This allows ASLR to randomise address space for each
  connection as a security mitigation. It should not have any visible impact
  - if there are any performance impacts in the wild please report it.

- Check authorized_keys permissions as the user, fixes NFS squash root.
  Patch from Chris Dragan (github #107)

- A missing home directory is now non-fatal, starting in / instead

- Fixed IPv6 [address]:port parsing for dbclient -b
  Reported by Fabio Molinari

- Improve error logging so that they are logged on the server rather than being
  sent to the client over the connection

- Max window size is increased to 10MB, more graceful fallback if it's invalid.

- Fix correctness of Dropbear's handling of global requests.
  Patch from Dirkjan Bussink

- Fix some small bugs found by fuzzers, null pointer dereference crash and leaks
  (post authentication)

- $HOME variable is used before /etc/passwd when expanding paths such as
  ~/.ssh/id_dropbear (for the client). Patch from Matt Robinson

- C89 build fixes from Guillaume Picquet

Infrastructure:

- Improvements to fuzzers. Added post-auth fuzzer, and a mutator that can
  handle the structure of SSH packet streams. Added cifuzz to run on commits
  and pull requests.
  Thanks to OSS-Fuzz for the tools/clusters and reward funding.

- Dropbear source tarballs generated by release.sh are now reproducible from a
  Git or Mercurial checkout, they will be identical on any system. Tested
  on ubuntu and macos.

- Added some integration testing using pytest. Currently this has tests
  for various channel handling edge cases, ASLR fork randomisation,
  dropbearconvert, and SSH_PUBKEYINFO

- Set up github actions. This runs the pytest suite and other checks.
  - build matrix includes c89, dropbearmulti, bundled libtom, macos, DEBUG_TRACE
  - test for configure script regeneration
  - build a tarball for external reproducibility
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants