insufficient security 71

[soon-leong]

When the p and q values of a Diffie-Hellman server key exchange is not found in the groups supported by bouncy castle, an insufficent security 71 is encountered. It seems like custom group is not supported. Is this true?

[peterdettman]

It would be an expensive operation to validate a custom group, so by default we accept only a few well-known DH groups (from RFC 3526 and RFC 7919). Because of attacks like "logjam" we also (by default) require at minimum a 2048 bit prime.

This default behaviour is captured in org.bouncycastle.tls.DefaultTlsDHGroupVerifier . If you are using the low-level TLS API you can replace this behaviour by overriding AbstractTlsClient#getDHGroupVerifier to return your own implementation of TlsDHGroupVerifier (e.g. to accept also the particular DH group a server uses if you know it's valid) .

If using BCJSSE, the integer system property "org.bouncycastle.jsse.client.dh.minimumPrimeBits" can be set to control the minimum prime bits and the boolean system property "org.bouncycastle.jsse.client.dh.unrestrictedGroups" can be set to "true" to allow unrestricted groups. Unfortunately no finer grain control is available.

We don't recommend loosening these restrictions; in general if you have any control over the server it would be preferable to change it to use non-DH cipher suites, or at least to use a well-known DH group from RFC 7919. It's probably a bad idea to accept custom groups from arbitrary internet servers.

[soon-leong]

That DH group came from a Windows machine. Today, the same p value was observed for a Ubuntu machine. Hence, this should not be a custom DH group.

Here is the p values from the DH group.

0000 0e f8 69 0b 35 2f 62 59 f7 af 4e 19 b5 9b d2 eb
0010 33 78 1d 43 1d b6 e4 a3 63 47 6a d4 a8 28 11 8c
0020 3f c8 f1 32 2b 5d 9f f8 a6 ca 21 71 de 30 d7 b5
0030 d6 a4 c2 ee c0 49 30 e7 8c 9b 1a 5a 08 2a 11 84
0040 e2 c8 36 6c dc 06 79 59 51 a4 a0 8f e1 20 94 80
0050 ac 6d fd 3b a6 a6 70 51 93 59 28 51 54 a3 c5 15
0060 44 2c 12 e7 95 62 0e 65 2f 8c 0d f8 63 52 00 2a
0070 a5 d7 59 ef 13 18 33 25 bc ad c8 0a 72 8d 26 63
0080 d5 b3 bc 43 35 0b 5d c7 ca 45 17 06 24 71 ca 20
0090 73 e8 18 d3 8e e9 e9 8f 67 c0 2c 14 7e 41 18 6c
00a0 74 72 56 34 c0 db dd 85 8b e0 99 e8 5e c8 f7 d1
00b0 0c f8 83 34 37 9e 01 df 1c d9 e9 95 c1 4c 64 37
00c0 9b f5 8f 99 97 55 68 2e 23 b0 35 f3 a5 97 92 a0
00d0 6d b4 f8 d8 47 ce 3f 0b 36 0e eb 13 15 fd 4f 98
00e0 4f 14 26 e2 ac d9 42 c6 43 8a 95 6b 2b 44 38 7f
00f0 60 97 77 d8 7c 6f 5d 62 7c e1 c8 83 12 8b 5e 5e
0100 c7 5e d5 60 f3 2f fc fe 70 ac 58 3a 3c 18 15 54
0110 84 a8 aa 41 26 7b e0 a3

To my best effort, I did not find this in the standard DH groups so I was under the assumption that this is a custom group.

Does this p value belong to any standard DH group? I know it did not match the ones supported by bouncy castle.

[peterdettman]

Windows/Ubuntu doesn't have much to do with TLS DH group selection. Most likely the group in use is the default group for some particular server software (you didn't mention anything about what you are connecting to).

[soon-leong]

We are connecting using RDP protocol to Windows and Ubuntu.

The connection is to xrdp running on Ubuntu and RDP server on Windows.

The RDP connection is done over TLS and the DH group came from the cipher selected in the server hello packet during the TLS handshake. The cipher would come from the Windows server and openssl for xrdp. Hence, how could the TLS DH group selection on two OS be using the same custom DH group?

[peterdettman]

OK, xrdp, so now we can see where this group is defined: https://github.com/neutrinolabs/xrdp/blob/441f427694bf04bd7a7d900eafe75d9c44d375b0/common/ssl_calls.c#L890 .

The OS isn't usually relevant to the choice of DH group for a TLS server. It's either negotiated with the client from a list of named groups, or else the server is free to choose an explicit group (pre-TLS 1.3 only). There's no reason two servers can't use the same group. Perhaps the Windows server is also running xrdp, or perhaps one of the implementations copied the DH group from the other, to minimize interoperability problems.

[soon-leong]

We understand the choice of DH groups and the decision to accept standard DH groups in bouncy castle.

At our end, we are dealing with systems where their security cipher implementation varies and spans over the years. While we wish we could deal with standard DH groups, there are many reasons from different angles that imposes constraints and restriction at the integration points.