SSL Error With Nginx Key Download #268
Comments
|
TLDR; @jamesbjackson has the right approach #268 (comment) Put this recipe at the top of your Setup (for Ubuntu 18.04 not sure if it will work in all distros) remote_file "Copy more recent root certificate into Chef" do
path "/opt/chef/embedded/ssl/certs/cacert.pem"
source "file:///etc/ssl/certs/ca-certificates.crt"
owner 'root'
group 'root'
mode 0644
endPreviously I wrote:I'm seeing this too and was about to open my own issue. The problem is the SSL cert used by nginx.org was updated in a way that's not compatible with You can reproduce this error and find the root cause like this: Note this works fine in Ruby 2.7.4 It also works if you run The work-around I found is to set the Example: 🥳🥳🥳 WORKS! My "very monkey-patched midnight crisis mode" fix to get production running last night was to edit require "tempfile"
require "net/https"
require "uri"
require "chef/http/basic_client"
require "chef/monkey_patches/net_http"
require "chef/config"
require "chef/platform/query_helpers"
require "chef/exceptions"
ENV['SSL_CERT_FILE']="/etc/ssl/certs/ca-certificates.crt" # <=== ADD THIS
class Chef
# == Chef::HTTP
# Basic HTTP client, with support for adding features via middleware
class HTTP
.
.
.I'm working on a better fix and I'll update shortly. In the meantime, I wanted to get this info out there. Editorial comment: TBH this feels like the final straw for me on OpsWorks. My (admittedly raw) feelings last night https://twitter.com/wckoehler/status/1443713784670003205 EDIT: Additional context. OpsWorks has been a great tool and I'm super grateful for all the effort people like @ajgon have put into keeping it going. I have some use cases that will be hard to deploy any other way (like a system that processes large videos with ffmpeg). But the current situation with OpsWorks feels untenable to me. There are too many things out of my control. |
|
This is related to chef/chef#12126. TL;DR... the Let's encrypt root cert expired yesterday and it looks like all/many versions of Chef aren't handling it well. |
|
You can fix this by setting the |
|
This has worked as a clean way on server bring up (Ubuntu 20.04 LTS minimal tested). In case it helps anyone. |
|
I ended up adding As early in my cookbook as I could, and this seems to have fixed stuff for us. YMMV. |
|
This can manually be fixed on an already provisioned server by running sudo apt-get install -y ca-certificates \
&& update-ca-certificates --verbose --fresh \
&& cp /etc/ssl/certs/ca-certificates.crt /opt/chef/embedded/ssl/certs/cacert.pemAnd this is what I had to do in my cookbook # Prevent Chef from using outdated/distrusted CA certificates
# https://github.com/chef/chef/issues/12126
package 'ca-certificates' do
action :purge
end
package 'ca-certificates' do
action :install
end
execute 'Updating CA certificates...' do
command 'update-ca-certificates --verbose --fresh'
end
link '/opt/chef/embedded/ssl/certs/cacert.pem' do
to '/etc/ssl/certs/ca-certificates.crt'
endSource: chef/chef#12126 |
Hello @olbrich! Forgive the silly question. Trying to put out a major inherited fire here and new to Opsworks, very outdated on my Chef skills. When you say as early in your cookbook as you could, where exactly are you adding that |
@jhirn I created a recipe called link '/opt/chef/embedded/ssl/certs/cacert.pem' do
to '/etc/ssl/certs/ca-certificates.crt'
endand added it as the first step of my Setup (see screenshot). Good luck with the 🔥🤞
|
|
@olbrich 🙇♂️ Testing this out now. Will report back but thank you so much this is very helpful!!! Edit: whoops forgot this @willkoehler 🙇♂️ 😉 Update: It worked by simply adding the link to the top of the existing recipes. Not the cleanest but wer're in the proces sof upgrading our process so i'm not too worried about it. Thx everyone! |
|
What need to be added for Windows Instances ? |
|
Thank you very much, for investigating this. I'll add proper info, to Troubleshoot section of README. Edit: Or even better, I'll add it to opsworks ruby as configurable option. |
AWS Opsworks uses old, and for long time deprecated version of Chef (Chef 12). This version includes ROOT CA lists, which are long time expired - causing most of the scripts to fail. To mimic, a new configuration param `node['patches']['chef12_ssl_fix']` is introduced, to include more recent lists. Fixes #268 BREAKING CHANGE: By default new list of SSL certificates is used. It should not affect any of your current deployments, but if you start seeing SSL errors, the first thing you should check, is disabling `node['patches']['chef12_ssl_fix']` option. See #268 for more information.
AWS Opsworks uses old, and for long time deprecated version of Chef (Chef 12). This version includes ROOT CA lists, which are long time expired - causing most of the scripts to fail. To mimic, a new configuration param `node['patches']['chef12_ssl_fix']` is introduced, to include more recent lists. Fixes #268 BREAKING CHANGE: By default new list of SSL certificates is used. It should not affect any of your current deployments, but if you start seeing SSL errors, the first thing you should check, is disabling `node['patches']['chef12_ssl_fix']` option. See #268 for more information.
AWS Opsworks uses old, and for long time deprecated version of Chef (Chef 12). This version includes ROOT CA lists, which are long time expired - causing most of the scripts to fail. To mimic, a new configuration param `node['patches']['chef12_ssl_fix']` is introduced, to include more recent lists. Fixes #268 BREAKING CHANGE: By default new list of SSL certificates is used. It should not affect any of your current deployments, but if you start seeing SSL errors, the first thing you should check, is disabling `node['patches']['chef12_ssl_fix']` option. See #268 for more information.
|
It is very sad that they have closed the original issue leaving people on their own with the problem. I spend hours and hours trying different combinations of commands, debugging the issue in Ruby, diving into the beautiful world of Ruby and OpenSSL setup in Chef, and, after all: Here's the working solution for Ubuntu 16.04.7 LTS Xenial and OpsWorks
shared-cookbook/recipes/chef_cert_fix.rbRun the recipe before other cookbooks or include it in a cookbook that may need it. I personally call it during AMI builds as a separate provisioner before running the main cookbook for the AMI. case node['platform']
when 'ubuntu'
%w(ca-certificates libgnutls-openssl27).each do |pkg|
package pkg do
action :nothing
end.run_action(:install) # run at compile time
end
bash 'update-ca-certificates' do
chef_ca_cert_path = '/opt/chef/embedded/ssl/certs/cacert.pem'
code <<-EOC
sed -i 's/mozilla\/DST_Root_CA_X3.crt/!mozilla\/DST_Root_CA_X3.crt/g' /etc/ca-certificates.conf
update-ca-certificates --fresh # `ls -al /etc/ssl/certs/ | grep DST_Root_CA_X3` must return nothing!
cp -r /etc/ssl/certs/ca-certificates.crt #{chef_ca_cert_path} && chmod 0644 #{chef_ca_cert_path}
# Now let's allow Chef client to continue communicating with OpsWorks after the certs substitution.
# In this example I assume that you somehow preconfigured `~/.chef/knife.rb`
# and Knife knows how to contact to your OpsWorks server
knife ssl fetch && knife ssl check
cat /root/.chef/ca_certs/AWS_OpsWorks_Root_CA.crt >> #{chef_ca_cert_path}
# You can replace the `knife` command output with a static file(s) provided by your CI/CD system:
# cat /etc/chef/trusted_certs/*.pem >> #{chef_ca_cert_path}
EOC
returns 0
group 'root'
user 'root'
action :nothing
end.run_action(:run) # run at compile time
endshared-cookbook/recipes/chef_cert_fix_trigger.rbThis recipe is an additional trick that extends the main #
# The file contains an additional step for chef_cert_fix that checks if anything else
# has restored the mozilla/DST_Root_CA_X3.crt certificate. If the file exists, the recipe
# deletes it and triggers auth-refreshment for Chef certificates located in chef_cert_fix.rb:bash[update-ca-certificates]
#
case node['platform']
when 'ubuntu'
file '/usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt' do
action :delete
only_if { File.exist? '/usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt' }
notifies :run, 'bash[update-ca-certificates]'
end.run_action(:delete) # run at compile time
endTo use it, include them like this: somecookbook/recipes/some-action.rbinclude_recipe 'shared-cookbook::chef_cert_fix' # include the main recipe before everything
include_recipe 'somecookbook::some-other-action' # assuming that something inside restores the broken CA cert
include_recipe 'shared-cookbook::chef_cert_fix_trigger' # include the additional trick to make sure the broken cert is deleted again
include_recipe 'somecookbook::yet-another-step' # a step that used to be complaining about the outdated certUnfortunately, even this solution is limited and there's a risk that the broken cert may be reinstalled at some moment where Chef cannot control it. The only advice here is to upgrade the Ubuntu itself to the latest. Other tipsWhen you connect to the machine (an AMI builder or an instance controlled by Chef where the cookbook is running), you can try debugging SSL issues using the commands: /opt/chef/embedded/bin/irbAnd inside the Ruby shell: require 'net/http'
Net::HTTP.get(URI.parse("https://YOUR-OPSWORKS-URL.AWS-REGION.opsworks-cm.io:443"))
Net::HTTP.get(URI.parse("https://pkg.jenkins.io/debian-stable/jenkins.io.key"))Both commands must be successful. If they are not, try using something like the doctor.rb to check what's going on with the certificates: SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt \
/opt/chef/embedded/bin/ruby \
doctor.rb \
pkg.jenkins.io:443
SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt \
/opt/chef/embedded/bin/ruby \
doctor.rb \
YOUR-OPSWORKS-URL.AWS-REGION.opsworks-cm.io:443
# COMPARE WITH:
SSL_CERT_FILE=/opt/chef/embedded/ssl/certs/cacert.pem \
/opt/chef/embedded/bin/ruby \
doctor.rb \
YOUR-OPSWORKS-URL.AWS-REGION.opsworks-cm.io:443
SSL_CERT_FILE=/opt/chef/embedded/ssl/certs/cacert.pem \
/opt/chef/embedded/bin/ruby \
doctor.rb \
pkg.jenkins.io:443
P.S.: All credit goes to the people who published their recommendations, comments, tools etc. in different places, including:
|
|
Hey Eugene,
Where do you put `mycookbook::chef_cert_fix` on OpsWorks to be called
before anything else?
…On Fri, 15 Oct 2021, 22:19 Eugene Glotov, ***@***.***> wrote:
It is very sad that they have closed the original issue
<chef/chef#12126> leaving people on their own
with the problem. I spend hours and hours trying different combinations of
commands, debugging the issue in Ruby, diving into the beautiful world of
Ruby setup in Chef, and, after all:
Here's the working solution for *Ubuntu 16.04.7 LTS Xenial* and OpsWorks
mycookbook/recipes/chef_cert_fix.rb
Run the recipe before other cookbooks or include it in a cookbook that may
need it. I personally call it during AMI builds as a separate provisioner
before running the main cookbook for the AMI.
case node['platform']when 'ubuntu'
%w(ca-certificates libgnutls-openssl27).each do |pkg|
package pkg do
action :nothing
end.run_action(:install) # run at compile time
end
bash 'update-ca-certificates' do
chef_ca_cert_path = '/opt/chef/embedded/ssl/certs/cacert.pem'
code <<-EOC sed -i 's/mozilla\/DST_Root_CA_X3.crt/!mozilla\/DST_Root_CA_X3.crt/g' /etc/ca-certificates.conf update-ca-certificates --fresh # `ls -al /etc/ssl/certs/ | grep DST_Root_CA_X3` must return nothing! cp -r /etc/ssl/certs/ca-certificates.crt #{chef_ca_cert_path} && chmod 0644 #{chef_ca_cert_path} # Now let's allow Chef client to continue communicating with OpsWorks after the certs substitution. # In this example I assume that you somehow preconfigured `~/.chef/knife.rb` # and Knife knows how to contact to your OpsWorks server knife ssl fetch && knife ssl check cat /root/.chef/ca_certs/AWS_OpsWorks_Root_CA.crt >> #{chef_ca_cert_path} # You can replace the `knife` command output with a static file(s) provided by your CI/CD system: # cat /etc/chef/trusted_certs/*.pem >> #{chef_ca_cert_path} EOC
returns 0
group 'root'
user 'root'
action :nothing
end.run_action(:run) # run at compile timeend
When you connect to the server, you can try debugging SSL issues using the
commands:
/opt/chef/embedded/bin/irb
And inside the Ruby shell:
require 'net/http'Net::HTTP.get(URI.parse("https://YOUR-OPSWORKS-URL.AWS-REGION.opsworks-cm.io:443"))Net::HTTP.get(URI.parse("https://pkg.jenkins.io/debian-stable/jenkins.io.key"))
Both commands must be successful. If they are not, try using something
like the doctor.rb <https://github.com/mislav/ssl-tools/tree/master> to
check what's going on with the certificates:
SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt \
/opt/chef/embedded/bin/ruby \
doctor.rb \
pkg.jenkins.io:443
SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt \
/opt/chef/embedded/bin/ruby \
doctor.rb \
YOUR-OPSWORKS-URL.AWS-REGION.opsworks-cm.io:443
# COMPARE WITH:
SSL_CERT_FILE=/opt/chef/embedded/ssl/certs/cacert.pem \
/opt/chef/embedded/bin/ruby \
doctor.rb \
YOUR-OPSWORKS-URL.AWS-REGION.opsworks-cm.io:443
SSL_CERT_FILE=/opt/chef/embedded/ssl/certs/cacert.pem \
/opt/chef/embedded/bin/ruby \
doctor.rb \
pkg.jenkins.io:443
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#268 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFDGAYPVGKOSQUYD7SN6FDUHCLFDANCNFSM5FEQZOKQ>
.
|
|
@clemblanco, we manage cookbooks differently for historical reasons. When we start an instance, a cookbook run-list is already mentioned in UserData in the launch command. Also, in my specific case, I faced the issue during an AMI build, where Packer starts Chef Solo and the recipe is added to You can probably try doing something like @willkoehler mentioned in this comment above. Or, simply include the recipe into a cookbook:
depends 'shared_cookbook'
include_recipe 'shared_cookbook::chef_cert_fix'
include_recipe 'current_cookbook::some_action'Also, I cannot say yet that the issue has gone completely. I still see that something is restoring back Up: also I added one more recipe |
AWS Opsworks uses old, and for long time deprecated version of Chef (Chef 12). This version includes ROOT CA lists, which are long time expired - causing most of the scripts to fail. To mimic, a new configuration param `node['patches']['chef12_ssl_fix']` is introduced, to include more recent lists. Fixes ajgon#268 BREAKING CHANGE: By default new list of SSL certificates is used. It should not affect any of your current deployments, but if you start seeing SSL errors, the first thing you should check, is disabling `node['patches']['chef12_ssl_fix']` option. See ajgon#268 for more information.
Hi guys, I'm seeing a new error when starting a new server with opsworks_ruby.
Seems the recipe is having trouble connecting to the nginx.org site in order to download the key:
[2021-10-01T11:53:51+00:00] ERROR: SSL Validation failure connecting to host: nginx.org - SSL_connect returned=1 errno=0 state=error: certificate verify failed
Is anybody seeing this behaviour?
The text was updated successfully, but these errors were encountered: