Fix test_ssl.rb in FIPS. #937

junaruga · 2025-09-01T16:44:06Z

This PR is working in progress. Right now there are 4 test failures below on the OpenSSL FIPS cases.

https://github.com/junaruga/ruby-openssl/actions/runs/17331123966/job/49207153277#step:11:419

$ OPENSSL_CONF=$OPENSSL_DIR/ssl/openssl_fips.cnf \
  bundle exec rake debug test_fips TEST=test/openssl/test_ssl.rb
...
F
===============================================================================
Failure: test_get_ephemeral_key(OpenSSL::TestSSL):
  exceptions on 1 threads:
  #<Thread:0x00007fc06e32d6e8 /home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:251 dead>:
  /home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:2321:in 'OpenSSL::SSL::SSLSocket#connect': SSL_connect returned=1 errno=0 peeraddr=127.0.0.1:45383 state=error: EVP lib (OpenSSL::SSL::SSLError)
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:2321:in 'OpenSSL::TestSSL#server_connect'
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:1760:in 'block in OpenSSL::TestSSL#test_get_ephemeral_key'
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:255:in 'block (2 levels) in OpenSSL::SSLTestCase#start_server'
/home/jaruga/var/git/ruby/openssl/bundle/ruby/3.5.0+2/gems/test-unit-ruby-core-1.0.7/lib/core_assertions.rb:762:in 'Test::Unit::CoreAssertions#assert_join_threads'
/home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:277:in 'block in OpenSSL::SSLTestCase#start_server'
/home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:203:in 'IO.pipe'
/home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:203:in 'OpenSSL::SSLTestCase#start_server'
/home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:1755:in 'OpenSSL::TestSSL#test_get_ephemeral_key'
     1752:       ctx.max_version = OpenSSL::SSL::TLS1_2_VERSION
     1753:       ctx.ciphers = "kRSA"
     1754:     }
  => 1755:     start_server(ctx_proc: ctx_proc1, ignore_listener_error: true) do |port|
     1756:       ctx = OpenSSL::SSL::SSLContext.new
     1757:       ctx.max_version = OpenSSL::SSL::TLS1_2_VERSION
     1758:       ctx.ciphers = "kRSA"
===============================================================================
F
===============================================================================
Failure: test_post_connect_check_with_anon_ciphers(OpenSSL::TestSSL):
  exceptions on 1 threads:
  #<Thread:0x00007fc06e246720 /home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:251 dead>:
  /home/jaruga/var/git/ruby/openssl/lib/openssl/ssl.rb:551:in 'OpenSSL::SSL::SSLSocket#accept': SSL_accept returned=1 errno=0 peeraddr=127.0.0.1:33730 state=error: internal error (OpenSSL::SSL::SSLError)
  	from /home/jaruga/var/git/ruby/openssl/lib/openssl/ssl.rb:551:in 'OpenSSL::SSL::SSLServer#accept'
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:226:in 'block (3 levels) in OpenSSL::SSLTestCase#start_server'
  	from <internal:kernel>:168:in 'Kernel#loop'
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:222:in 'block (2 levels) in OpenSSL::SSLTestCase#start_server'
  /home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:2321:in 'OpenSSL::SSL::SSLSocket#connect': SSL_connect returned=1 errno=0 peeraddr=127.0.0.1:34315 state=error: tlsv1 alert internal error (SSL alert number 80) (OpenSSL::SSL::SSLError)
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:2321:in 'OpenSSL::TestSSL#server_connect'
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:702:in 'block in OpenSSL::TestSSL#test_post_connect_check_with_anon_ciphers'
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:255:in 'block (2 levels) in OpenSSL::SSLTestCase#start_server'
/home/jaruga/var/git/ruby/openssl/bundle/ruby/3.5.0+2/gems/test-unit-ruby-core-1.0.7/lib/core_assertions.rb:762:in 'Test::Unit::CoreAssertions#assert_join_threads'
/home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:277:in 'block in OpenSSL::SSLTestCase#start_server'
/home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:203:in 'IO.pipe'
/home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:203:in 'OpenSSL::SSLTestCase#start_server'
/home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:697:in 'OpenSSL::TestSSL#test_post_connect_check_with_anon_ciphers'
     694:       ctx.security_level = 0
     695:     }
     696:
  => 697:     start_server(ctx_proc: ctx_proc) { |port|
     698:       ctx = OpenSSL::SSL::SSLContext.new
     699:       ctx.max_version = OpenSSL::SSL::TLS1_2_VERSION
     700:       ctx.ciphers = "aNULL"
===============================================================================
F
===============================================================================
Failure: test_tmp_dh(OpenSSL::TestSSL):
  exceptions on 1 threads:
  #<Thread:0x00007fc06d3c2460 /home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:251 dead>:
  /home/jaruga/var/git/ruby/openssl/lib/openssl/ssl.rb:551:in 'OpenSSL::SSL::SSLSocket#accept': SSL_accept returned=1 errno=0 peeraddr=127.0.0.1:58706 state=error: internal error (OpenSSL::SSL::SSLError)
  	from /home/jaruga/var/git/ruby/openssl/lib/openssl/ssl.rb:551:in 'OpenSSL::SSL::SSLServer#accept'
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:226:in 'block (3 levels) in OpenSSL::SSLTestCase#start_server'
  	from <internal:kernel>:168:in 'Kernel#loop'
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:222:in 'block (2 levels) in OpenSSL::SSLTestCase#start_server'
  /home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:2321:in 'OpenSSL::SSL::SSLSocket#connect': SSL_connect returned=1 errno=0 peeraddr=127.0.0.1:33911 state=error: tlsv1 alert internal error (SSL alert number 80) (OpenSSL::SSL::SSLError)
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:2321:in 'OpenSSL::TestSSL#server_connect'
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:2143:in 'block in OpenSSL::TestSSL#test_tmp_dh'
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:255:in 'block (2 levels) in OpenSSL::SSLTestCase#start_server'
/home/jaruga/var/git/ruby/openssl/bundle/ruby/3.5.0+2/gems/test-unit-ruby-core-1.0.7/lib/core_assertions.rb:762:in 'Test::Unit::CoreAssertions#assert_join_threads'
/home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:277:in 'block in OpenSSL::SSLTestCase#start_server'
/home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:203:in 'IO.pipe'
/home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:203:in 'OpenSSL::SSLTestCase#start_server'
/home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:2142:in 'OpenSSL::TestSSL#test_tmp_dh'
     2139:       ctx.ciphers = "DH:!NULL" # use DH
     2140:       ctx.tmp_dh = dh
     2141:     }
  => 2142:     start_server(ctx_proc: ctx_proc) do |port|
     2143:       server_connect(port) { |ssl|
     2144:         assert_equal dh.to_der, ssl.tmp_key.to_der
     2145:       }
===============================================================================
F
===============================================================================
Failure: test_tmp_dh_callback(OpenSSL::TestSSL):
  exceptions on 1 threads:
  #<Thread:0x00007fc06d330740 /home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:251 dead>:
  /home/jaruga/var/git/ruby/openssl/lib/openssl/ssl.rb:551:in 'OpenSSL::SSL::SSLSocket#accept': SSL_accept returned=1 errno=0 peeraddr=127.0.0.1:55890 state=error: internal error (OpenSSL::SSL::SSLError)
  	from /home/jaruga/var/git/ruby/openssl/lib/openssl/ssl.rb:551:in 'OpenSSL::SSL::SSLServer#accept'
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:226:in 'block (3 levels) in OpenSSL::SSLTestCase#start_server'
  	from <internal:kernel>:168:in 'Kernel#loop'
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:222:in 'block (2 levels) in OpenSSL::SSLTestCase#start_server'
  /home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:2321:in 'OpenSSL::SSL::SSLSocket#connect': SSL_connect returned=1 errno=0 peeraddr=127.0.0.1:45871 state=error: tlsv1 alert internal error (SSL alert number 80) (OpenSSL::SSL::SSLError)
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:2321:in 'OpenSSL::TestSSL#server_connect'
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:1879:in 'block in OpenSSL::TestSSL#test_tmp_dh_callback'
  	from /home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:255:in 'block (2 levels) in OpenSSL::SSLTestCase#start_server'
/home/jaruga/var/git/ruby/openssl/bundle/ruby/3.5.0+2/gems/test-unit-ruby-core-1.0.7/lib/core_assertions.rb:762:in 'Test::Unit::CoreAssertions#assert_join_threads'
/home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:277:in 'block in OpenSSL::SSLTestCase#start_server'
/home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:203:in 'IO.pipe'
/home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:203:in 'OpenSSL::SSLTestCase#start_server'
/home/jaruga/var/git/ruby/openssl/test/openssl/test_ssl.rb:1878:in 'OpenSSL::TestSSL#test_tmp_dh_callback'
     1875:         dh
     1876:       }
     1877:     }
  => 1878:     start_server(ctx_proc: ctx_proc) do |port|
     1879:       server_connect(port) { |ssl|
     1880:         assert called, "dh callback should be called"
     1881:         assert_equal dh.to_der, ssl.tmp_key.to_der
===============================================================================
Finished in 11.583246314 seconds.
-------------------------------------------------------------------------------
88 tests, 559 assertions, 4 failures, 0 errors, 0 pendings, 1 omissions, 0 notifications
95.4023% passed
-------------------------------------------------------------------------------
...

I am debugging ruby openssl's latest master branch on the OpenSSL relatively latest commit openssl/openssl@0b091c8.

The cause of the 4 test failures looks similar. For example, the failure of the test_post_connect_check_with_anon_ciphers happened at the

openssl/test/openssl/test_ssl.rb

Line 2321 in d377a34

ssl.connect

(SSLSocket#connect).

$ OPENSSL_CONF=$OPENSSL_DIR/ssl/openssl_fips.cnf \
  ruby -I ./lib test/openssl/test_ssl.rb \
  -v --name=test_post_connect_check_with_anon_ciphers

I debugged the issue on gdb like this.

$ gdb --args \
  ruby -I ./lib test/openssl/test_ssl.rb \
  -v --name=test_post_connect_check_with_anon_ciphers
(gdb) set environment OPENSSL_CONF=/home/jaruga/.local/openssl-3.6.0-dev-fips-debug-0b091c88d7/ssl/openssl_fips.cnf
(gdb) b ossl_ssl_connect
(gdb) b SSL_connect

Then I found the following line is a part of the happening error.

https://github.com/openssl/openssl/blob/0b091c88d7d50c542ee393ed31ef5a1b92eea476/ssl/statem/statem.c#L489

(gdb) f
#0  state_machine (s=0x7fffc000b640, server=0) at ssl/statem/statem.c:489
489>                goto end;

This is the backtrace.

(gdb) bt
#0  state_machine (s=0x7fffc000b640, server=0) at ssl/statem/statem.c:489
#1  0x00007fffce314187 in ossl_statem_connect (s=0x7fffc000b640) at ssl/statem/statem.c:301
#2  0x00007fffce277794 in SSL_do_handshake (s=0x7fffc000b640) at ssl/ssl_lib.c:5014
#3  0x00007fffce270e03 in SSL_connect (s=0x7fffc000b640) at ssl/ssl_lib.c:2250
#4  0x00007fffce3d9551 in ossl_start_ssl (self=140736638532320, func=0x7fffce270d49 <SSL_connect>, funcname=0x7fffce406f95 "SSL_connect", opts=0) at ../../../../ext/openssl/ossl_ssl.c:1790
#5  0x00007fffce3d97c8 in ossl_ssl_connect (self=140736638532320) at ../../../../ext/openssl/ossl_ssl.c:1864
#6  0x00007ffff7ae7c92 in ractor_safe_call_cfunc_0 (recv=140736638532320, argc=0, argv=0x7fffcc9ac0b0, func=0x7fffce3d978e <ossl_ssl_connect>) at vm_insnhelper.c:3587
#7  0x00007ffff7ae885b in vm_call_cfunc_with_frame_ (ec=0xd0b030, reg_cfp=0x7fffccaabf30, calling=0x7fffcc9a99e0, argc=0, argv=0x7fffcc9ac0b0, stack_bottom=0x7fffcc9ac0a8) at vm_insnhelper.c:3771
#8  0x00007ffff7ae8ac4 in vm_call_cfunc_with_frame (ec=0xd0b030, reg_cfp=0x7fffccaabf30, calling=0x7fffcc9a99e0) at vm_insnhelper.c:3817
#9  0x00007ffff7ae8bed in vm_call_cfunc_other (ec=0xd0b030, reg_cfp=0x7fffccaabf30, calling=0x7fffcc9a99e0) at vm_insnhelper.c:3843
#10 0x00007ffff7ae9028 in vm_call_cfunc (ec=0xd0b030, reg_cfp=0x7fffccaabf30, calling=0x7fffcc9a99e0) at vm_insnhelper.c:3925
#11 0x00007ffff7aebc13 in vm_call_method_each_type (ec=0xd0b030, cfp=0x7fffccaabf30, calling=0x7fffcc9a99e0) at vm_insnhelper.c:4750
#12 0x00007ffff7aec6c7 in vm_call_method (ec=0xd0b030, cfp=0x7fffccaabf30, calling=0x7fffcc9a99e0) at vm_insnhelper.c:4876
#13 0x00007ffff7aec8c5 in vm_call_general (ec=0xd0b030, reg_cfp=0x7fffccaabf30, calling=0x7fffcc9a99e0) at vm_insnhelper.c:4920
#14 0x00007ffff7aef00b in vm_sendish (ec=0xd0b030, reg_cfp=0x7fffccaabf30, cd=0x812770, block_handler=0, method_explorer=mexp_search_method) at vm_insnhelper.c:5969
#15 0x00007ffff7af6b6e in vm_exec_core (ec=0xd0b030) at insns.def:899
#16 0x00007ffff7b0ef92 in rb_vm_exec (ec=0xd0b030) at vm.c:2625
#17 0x00007ffff7b0c2c9 in invoke_iseq_block_from_c (ec=0xd0b030, captured=0xd05c10, self=140736651740080, argc=0, argv=0x7fffcc9aaa20, kw_splat=0, passed_block_handler=0, cref=0x0, is_lambda=0, me=0x0) at vm.c:1655
#18 invoke_block_from_c_proc (ec=0xd0b030, proc=0xd05c10, self=140736651740080, argc=0, argv=0x7fffcc9aaa20, kw_splat=0, passed_block_handler=0, is_lambda=0, me=0x0) at vm.c:1749
#19 vm_invoke_proc (ec=0xd0b030, proc=0xd05c10, self=140736651740080, argc=0, argv=0x7fffcc9aaa20, kw_splat=0, passed_block_handler=0) at vm.c:1779
#20 0x00007ffff7b0ca37 in rb_vm_invoke_proc (ec=0xd0b030, proc=0xd05c10, argc=0, argv=0x7fffcc9aaa20, kw_splat=0, passed_block_handler=0) at vm.c:1800
#21 0x00007ffff7a9d62f in thread_do_start_proc (th=0x72fc10) at thread.c:604
#22 0x00007ffff7a9d6a0 in thread_do_start (th=0x72fc10) at thread.c:621
#23 0x00007ffff7a9d98d in thread_start_func_2 (th=0x72fc10, stack_start=0x7fffcc9aabe8) at thread.c:676
#24 0x00007ffff7a96eb5 in call_thread_start_func_2 (th=0x72fc10) at thread_pthread.c:2218
#25 0x00007ffff7a96fd5 in nt_start (ptr=0xd069c0) at thread_pthread.c:2263
#26 0x00007ffff767f1d4 in start_thread (arg=<optimized out>) at pthread_create.c:448
#27 0x00007ffff7701cec in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

I need to debug more. But I assume the error in the connection process just is happening in the FIPS specific part, of course. What do you think about how to fix this failure?

junaruga · 2025-09-02T17:58:36Z

I need to debug more. But I assume the error in the connection process just is happening in the FIPS specific part, of course. What do you think about how to fix this failure?

I debugged more with the openssl/openssl@0b091c8. The error is happening at the following FIPS-specific logic, when the dh->params.q is NULL.

https://github.com/openssl/openssl/blob/0b091c88d7d50c542ee393ed31ef5a1b92eea476/crypto/dh/dh_key.c#L325-L328

crypto/dh/dh_key.c

...
325 #ifdef FIPS_MODULE
326             if (dh->params.q == NULL)
327                 goto err;
328 #else
...

The GDB stacktrace is below.

(gdb) f
#0  generate_key (dh=0x7fffc800e640) at crypto/dh/dh_key.c:327
327	                goto err;
(gdb) bt
#0  generate_key (dh=0x7fffc800e640) at crypto/dh/dh_key.c:327
#1  0x00007fffcd6a239c in DH_generate_key (dh=0x7fffc800e640) at crypto/dh/dh_key.c:223
#2  0x00007fffcd6337cf in dh_gen (genctx=0x7fffc800e5c0, osslcb=0x7fffcdb92ce1 <ossl_callback_to_pkey_gencb>, cbarg=0x7fffc800e500) at providers/implementations/keymgmt/dh_kmgmt.c:811
#3  0x00007fffcdb84ba8 in evp_keymgmt_gen (keymgmt=0x889fe0, genctx=0x7fffc800e5c0, cb=0x7fffcdb92ce1 <ossl_callback_to_pkey_gencb>, cbarg=0x7fffc800e500) at crypto/evp/keymgmt_meth.c:464
#4  0x00007fffcdb83580 in evp_keymgmt_util_gen (target=0x7fffc8000ba0, keymgmt=0x889fe0, genctx=0x7fffc800e5c0, cb=0x7fffcdb92ce1 <ossl_callback_to_pkey_gencb>, cbarg=0x7fffc800e500) at crypto/evp/keymgmt_lib.c:522
#5  0x00007fffcdb92f68 in EVP_PKEY_generate (ctx=0x7fffc800e500, ppkey=0x7fffccbab1c8) at crypto/evp/pmeth_gn.c:189
#6  0x00007fffcdb93234 in EVP_PKEY_keygen (ctx=0x7fffc800e500, ppkey=0x7fffccbab1c8) at crypto/evp/pmeth_gn.c:274
#7  0x00007fffce25d88f in ssl_generate_pkey (s=0x7fffc80032b0, pm=0xb1a5a0) at ssl/s3_lib.c:4965
#8  0x00007fffce33138b in tls_construct_server_key_exchange (s=0x7fffc80032b0, pkt=0x7fffccbab330) at ssl/statem/statem_srvr.c:2615
#9  0x00007fffce31550b in write_state_machine (s=0x7fffc80032b0) at ssl/statem/statem.c:902
#10 0x00007fffce314761 in state_machine (s=0x7fffc80032b0, server=1) at ssl/statem/statem.c:492
#11 0x00007fffce314202 in ossl_statem_accept (s=0x7fffc80032b0) at ssl/statem/statem.c:311
#12 0x00007fffce277794 in SSL_do_handshake (s=0x7fffc80032b0) at ssl/ssl_lib.c:5014
#13 0x00007fffce270d47 in SSL_accept (s=0x7fffc80032b0) at ssl/ssl_lib.c:2230
#14 0x00007fffce3d9551 in ossl_start_ssl (self=140736638531520, func=0x7fffce270c8d <SSL_accept>, funcname=0x7fffce406fa4 "SSL_accept", opts=0) at ../../../../ext/openssl/ossl_ssl.c:1790
#15 0x00007fffce3d9873 in ossl_ssl_accept (self=140736638531520) at ../../../../ext/openssl/ossl_ssl.c:1911
#16 0x00007ffff7ae7c92 in ractor_safe_call_cfunc_0 (recv=140736638531520, argc=0, argv=0x7fffccbae0d8, func=0x7fffce3d9839 <ossl_ssl_accept>) at vm_insnhelper.c:3587
#17 0x00007ffff7ae885b in vm_call_cfunc_with_frame_ (ec=0x8fc240, reg_cfp=0x7fffcccadef8, calling=0x7fffccbab9e0, argc=0, argv=0x7fffccbae0d8, stack_bottom=0x7fffccbae0d0) at vm_insnhelper.c:3771
#18 0x00007ffff7ae8ac4 in vm_call_cfunc_with_frame (ec=0x8fc240, reg_cfp=0x7fffcccadef8, calling=0x7fffccbab9e0) at vm_insnhelper.c:3817
#19 0x00007ffff7ae8bed in vm_call_cfunc_other (ec=0x8fc240, reg_cfp=0x7fffcccadef8, calling=0x7fffccbab9e0) at vm_insnhelper.c:3843
#20 0x00007ffff7ae9028 in vm_call_cfunc (ec=0x8fc240, reg_cfp=0x7fffcccadef8, calling=0x7fffccbab9e0) at vm_insnhelper.c:3925
#21 0x00007ffff7aebc13 in vm_call_method_each_type (ec=0x8fc240, cfp=0x7fffcccadef8, calling=0x7fffccbab9e0) at vm_insnhelper.c:4750
#22 0x00007ffff7aec6c7 in vm_call_method (ec=0x8fc240, cfp=0x7fffcccadef8, calling=0x7fffccbab9e0) at vm_insnhelper.c:4876
#23 0x00007ffff7aec8c5 in vm_call_general (ec=0x8fc240, reg_cfp=0x7fffcccadef8, calling=0x7fffccbab9e0) at vm_insnhelper.c:4920
#24 0x00007ffff7aef00b in vm_sendish (ec=0x8fc240, reg_cfp=0x7fffcccadef8, cd=0x8565d0, block_handler=0, method_explorer=mexp_search_method) at vm_insnhelper.c:5969
#25 0x00007ffff7af6b6e in vm_exec_core (ec=0x8fc240) at insns.def:899
#26 0x00007ffff7b0ef92 in rb_vm_exec (ec=0x8fc240) at vm.c:2625
#27 0x00007ffff7b0c2c9 in invoke_iseq_block_from_c (ec=0x8fc240, captured=0xd05470, self=140736651740080, argc=0, argv=0x7fffccbaca20, kw_splat=0, passed_block_handler=0, cref=0x0, is_lambda=0, me=0x0) at vm.c:1655
#28 invoke_block_from_c_proc (ec=0x8fc240, proc=0xd05470, self=140736651740080, argc=0, argv=0x7fffccbaca20, kw_splat=0, passed_block_handler=0, is_lambda=0, me=0x0) at vm.c:1749
#29 vm_invoke_proc (ec=0x8fc240, proc=0xd05470, self=140736651740080, argc=0, argv=0x7fffccbaca20, kw_splat=0, passed_block_handler=0) at vm.c:1779
#30 0x00007ffff7b0ca37 in rb_vm_invoke_proc (ec=0x8fc240, proc=0xd05470, argc=0, argv=0x7fffccbaca20, kw_splat=0, passed_block_handler=0) at vm.c:1800
#31 0x00007ffff7a9d62f in thread_do_start_proc (th=0xd03990) at thread.c:604
#32 0x00007ffff7a9d6a0 in thread_do_start (th=0xd03990) at thread.c:621
#33 0x00007ffff7a9d98d in thread_start_func_2 (th=0xd03990, stack_start=0x7fffccbacbe8) at thread.c:676
#34 0x00007ffff7a96eb5 in call_thread_start_func_2 (th=0xd03990) at thread_pthread.c:2218
#35 0x00007ffff7a96fd5 in nt_start (ptr=0xd02cb0) at thread_pthread.c:2263
#36 0x00007ffff767f1d4 in start_thread (arg=<optimized out>) at pthread_create.c:448
#37 0x00007ffff7701cec in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

The value of the dh->params is below.

(gdb) f
#0  generate_key (dh=0x7fffc800e640) at crypto/dh/dh_key.c:327
327	                goto err;
(gdb) p dh->params
$61 = {p = 0x7fffc8001970, q = 0x0, g = 0x7fffc800f530, j = 0x0, seed = 0x0,
  seedlen = 0, pcounter = -1, nid = 0, gindex = -1, h = 0, flags = 3, mdname = 0x0,
  mdprops = 0x0, keylength = 0}

The dh->params.q means struct ffc_params_st - BIGNUM *q.

https://github.com/openssl/openssl/blob/0b091c88d7d50c542ee393ed31ef5a1b92eea476/include/internal/ffc.h#L93

The change was added by the commit openssl/openssl@f11f86f (this part).

Do you think this is an expected behavior? Is it better to improve an error handling for this error?

rhenium · 2025-09-02T19:07:10Z

I'm not familiar with the requirements by FIPS 140, but the DH instance would not have the "q" value since dh-1.pem is PKCS#3 format, which doesn't include it.

OpenSSL has a special workaround to recover the missing "q" value for known named parameters, which is why other tests that use our defaults in lib/openssl/ssl.rb are working: https://github.com/openssl/openssl/blob/cca9844f0c3d59b1be99ba52b41d5cb922b78138/crypto/dh/dh_group_params.c#L70-L79

After all, DH is mostly only kept around for compatibility. I think the failing tests should simply be skipped.

junaruga · 2025-09-11T17:51:31Z

I'm not familiar with the requirements by FIPS 140, but the DH instance would not have the "q" value since dh-1.pem is PKCS#3 format, which doesn't include it.

OpenSSL has a special workaround to recover the missing "q" value for known named parameters, which is why other tests that use our defaults in lib/openssl/ssl.rb are working: https://github.com/openssl/openssl/blob/cca9844f0c3d59b1be99ba52b41d5cb922b78138/crypto/dh/dh_group_params.c#L70-L79

After all, DH is mostly only kept around for compatibility. I think the failing tests should simply be skipped.

Thank you for your reply.

So, in the case with not-known named parameters (not-known ciphers) such as aNULL in the test_post_connect_check_with_anon_ciphers, the workaround is skipped, right? I will skip the test, and check other failing tests in the test/openssl/test_ssl.rb.

junaruga · 2025-09-16T11:29:56Z

@rhenium I rebased this PR. I added the reasons why we skip the tests in FIPS in the commit message. Could you review the PR including the content of the commit message? Thank you!

Note that I confirmed the the failures came from the following FIPS specific logic checking the q value was NULL in the test_post_connect_check_with_anon_ciphers, test_tmp_dh_callback and test_tmp_dh tests using the GDB.

https://github.com/openssl/openssl/blob/0b091c88d7d50c542ee393ed31ef5a1b92eea476/crypto/dh/dh_key.c#L325-L328

test_post_connect_check_with_anon_ciphers: test_tmp_dh_callback: test_tmp_dh: DH missing the q value on unknown named parameters (ciphers) is not FIPS-approved, according to the FIPS-186-4 APPENDIX B: Key Pair Generation - B.1.1 Key Pair Generation Using Extra Random Bits, the inputs p, q, and g are required. However, TLS doesn't send q. https://csrc.nist.gov/pubs/fips/186-4/final OpenSSL has a special workaround to recover the missing "q" value for known named parameters, which is the reason why other tests that use the default parameters in `lib/openssl/ssl.rb` are working. Note that the test_post_connect_check_with_anon_ciphers test got the following error on `OpenSSL.debug = true` in FIPS. ``` /home/jaruga/var/git/ruby/openssl/lib/openssl/ssl.rb:551: warning: error on stack: error:0A0C0103:SSL routines:tls_construct_server_key_exchange:internal error ``` test_get_ephemeral_key: kRSA (PKCS1-v1_5 padding) is not allowed in FIPS according to the NIST SP 800-131A Rev. 2 - 6 Key Agreement and Key Transport Using RSA - Table 5: Approval Status for the RSA-based Key Agreement and Key Transport Schemes - PKCS1-v1_5 padding - Disallowed after 2023 https://csrc.nist.gov/pubs/sp/800/131/a/r2/final Note that the test_get_ephemeral_key test got the following error on `OpenSSL.debug = true` in FIPS. ``` test/openssl/test_ssl.rb:2326: warning: error on stack: error:1C8000A8:Provider routines:rsa_encrypt:invalid padding mode ```

rhenium

Looks good to me, thanks!

junaruga · 2025-09-17T15:04:05Z

Thank you for review!

junaruga mentioned this pull request Sep 1, 2025

Support OpenSSL 3 FIPS #603

Open

junaruga mentioned this pull request Sep 2, 2025

Add FFC param/key generation openssl/openssl#10909

Closed

2 tasks

junaruga force-pushed the wip/fips-test-ssl branch from 36eeb49 to 41d37c2 Compare September 16, 2025 11:06

junaruga marked this pull request as ready for review September 16, 2025 11:25

junaruga changed the title ~~WIP: Fix test_ssl.rb in FIPS.~~ Fix test_ssl.rb in FIPS. Sep 16, 2025

junaruga force-pushed the wip/fips-test-ssl branch from 41d37c2 to ac3559e Compare September 16, 2025 12:28

rhenium approved these changes Sep 17, 2025

View reviewed changes

rhenium merged commit 0df267e into ruby:master Sep 17, 2025
41 checks passed

junaruga deleted the wip/fips-test-ssl branch September 17, 2025 15:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix test_ssl.rb in FIPS. #937

Fix test_ssl.rb in FIPS. #937

Uh oh!

junaruga commented Sep 1, 2025

Uh oh!

junaruga commented Sep 2, 2025 •

edited

Loading

Uh oh!

rhenium commented Sep 2, 2025

Uh oh!

junaruga commented Sep 11, 2025

Uh oh!

junaruga commented Sep 16, 2025

Uh oh!

rhenium left a comment

Uh oh!

Uh oh!

junaruga commented Sep 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Fix test_ssl.rb in FIPS. #937

Fix test_ssl.rb in FIPS. #937

Uh oh!

Conversation

junaruga commented Sep 1, 2025

Uh oh!

junaruga commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rhenium commented Sep 2, 2025

Uh oh!

junaruga commented Sep 11, 2025

Uh oh!

junaruga commented Sep 16, 2025

Uh oh!

rhenium left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

junaruga commented Sep 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

junaruga commented Sep 2, 2025 •

edited

Loading