Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Puppet reporting fails with handshake_failure #535

Open
pschyska opened this issue Jun 14, 2019 · 12 comments
Open

Puppet reporting fails with handshake_failure #535

pschyska opened this issue Jun 14, 2019 · 12 comments

Comments

@pschyska
Copy link

The puppet reporting fails on puppetserver 2.8.1. We are using datadog-datadog_agent (v2.6.0). I manually updated the dogapi gem from 1.22 to 1.36.0 to test if that is an issue, but the behaviour didn't change.

Relevant configuration:

# datadog-reports.yaml
### MANAGED BY PUPPET

---
:datadog_api_key: '...'
:api_url: https://api.datadoghq.com

# /etc/puppetlabs/puppet/puppet.conf
[master]
reports = puppetdb,datadog_reports

Stacktrace from puppetserver.log:

2019-06-14 13:32:48,690 ERROR [qtp912129323-65] [puppetserver] Puppet Report processor failed: Received fatal alert: handshake_failure
org/jruby/ext/openssl/SSLSocket.java:215:in `connect'
/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/net/http.rb:800:in `connect'
org/jruby/ext/timeout/Timeout.java:115:in `timeout'
/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/net/http.rb:800:in `connect'
/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/net/http.rb:756:in `do_start'
/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/net/http.rb:745:in `start'
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/dogapi-1.36.0/lib/dogapi/common.rb:96:in `connect'
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/dogapi-1.36.0/lib/dogapi/common.rb:116:in `request'
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/dogapi-1.36.0/lib/dogapi/v1/metric.rb:24:in `upload'
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/dogapi-1.36.0/lib/dogapi/v1/metric.rb:41:in `flush_buffer'
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/dogapi-1.36.0/lib/dogapi/facade.rb:165:in `batch_metrics'
/etc/puppetlabs/code/environments/production/modules/datadog_agent/lib/puppet/reports/datadog_reports.rb:115:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:53:in `processors'
org/jruby/RubyArray.java:1613:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:285:in `save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:176:in `do_save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:48:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:306:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:47:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `process'
org/jruby/RubyArray.java:1613:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:60:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:58:in `process'
file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/master.rb:42:in `handleRequest'
Puppet$$Server$$Master_734706569.gen:13:in `handleRequest'
request_handler_core.clj:273:in `invoke'
jruby_request.clj:46:in `invoke'
jruby_request.clj:31:in `invoke'
request_handler_service.clj:34:in `handle_request'
request_handler.clj:3:in `invoke'
request_handler.clj:3:in `invoke'
core.clj:2515:in `invoke'
core.clj:211:in `invoke'
core.clj:45:in `invoke'
core.clj:343:in `invoke'
core.clj:51:in `invoke'
ringutils.clj:83:in `invoke'
master_core.clj:430:in `invoke'
ring.clj:21:in `invoke'
ring.clj:12:in `invoke'
comidi.clj:249:in `invoke'
jetty9_core.clj:427:in `invoke'
normalized_uri_helpers.clj:81:in `invoke'
@pschyska
Copy link
Author

I'm getting the same error in puppetserver irb. I guess you are using a certificate for api.datadoghq.com which is not trusted in the system keystore?
We are using CentOS Linux release 7.6.1810 (Core), java-1.8.0-openjdk.x86_64 1:1.8.0.212.b04-0.el7_6.

Any idea how to fix that?

@mrmikeace
Copy link

Ditto. Same problem, OS, etc.

@albertvaka
Copy link
Contributor

albertvaka commented Sep 18, 2019

I haven't been able to reproduce it on a fresh CentOS 7 VM with latest puppet.

Let's first verify your machine can reach the endpoint and that the cert is trusted (it should be): Can you run curl -v https://api.datadoghq.com and paste the output here?

Also can you please paste the output of puppetserver --version? Note that Puppet >= 4.6.x is required to use this module, and the original issue mentions using a version older than that.

@mrmikeace
Copy link

mrmikeace commented Sep 18, 2019 via email

@albertvaka
Copy link
Contributor

albertvaka commented Sep 18, 2019

You can make curl use IPv6/IPv4 with something like curl -6 -v https://api.datadoghq.com or curl -4 -v https://api.datadoghq.com to make sure it's an IPv6 connectivity problem.

@mrmikeace
Copy link

Curl -6 times out - looks like IPv6 is the culprit. I'll check my ACLs, firewall, etc.

Curl -4 works fine, and returns:

  • About to connect() to api.datadoghq.com port 443 (#0)
  • Trying 34.197.164.185...
  • Connected to api.datadoghq.com (34.197.164.185) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  • SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  • Server certificate:
  •   subject: CN=*.datadoghq.com,OU=PositiveSSL Wildcard,OU=Domain Control Validated
    
  •   start date: Jul 13 00:00:00 2016 GMT
    
  •   expire date: Oct 12 23:59:59 2019 GMT
    
  •   common name: *.datadoghq.com
    
  •   issuer: CN=COMODO RSA Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB
    

GET / HTTP/1.1
User-Agent: curl/7.29.0
Host: api.datadoghq.com
Accept: /

< HTTP/1.1 307 Temporary Redirect
< Date: Wed, 18 Sep 2019 16:40:45 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 302
< Connection: keep-alive
< Pragma: no-cache
< Cache-Control: no-cache
< Set-Cookie: DD-PSHARD=; Max-Age=0; Path=/; expires=Wed, 31-Dec-97 23:59:59 GMT
< X-DD-VERSION: 35.1707746
< Location: https://api.datadoghq.com/account/login?next=%2F
< X-DD-Debug: FxDUphI+Ut6vkIlm8wxwPjZaDx59Y59Y3TFRApVVWhusfrWrZhPb1oISDRTkDswe
< DD-POOL: dogweb_sameorig
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Strict-Transport-Security: max-age=15724800;
<

<title>307 Temporary Redirect</title>

307 Temporary Redirect

No authenticated user. https://api.datadoghq.com/account/login?next=%2F; you should be redirected automatically. * Connection #0 to host api.datadoghq.com left intact

@mrmikeace
Copy link

Regarding the above - my problem in the AWS datacenter was my SecurityGroup - it was not allowing IPv6 traffic in or out.

@pschyska
Copy link
Author

This is not my problem:

curl -v https://api.datadoghq.com
* About to connect() to proxy proxy port 3128 (#0)
*   Trying 10.0.1.213...
* Connected to proxy (10.0.1.213) port 3128 (#0)
* Establish HTTP proxy tunnel to api.datadoghq.com:443
> CONNECT api.datadoghq.com:443 HTTP/1.1
> Host: api.datadoghq.com:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=*.datadoghq.com,OU=PositiveSSL Wildcard,OU=Domain Control Validated
* 	start date: Jul 13 00:00:00 2016 GMT
* 	expire date: Oct 12 23:59:59 2019 GMT
* 	common name: *.datadoghq.com
* 	issuer: CN=COMODO RSA Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: api.datadoghq.com
> Accept: */*

Maybe the issue is triggered because we use a proxy?

@albertvaka
Copy link
Contributor

albertvaka commented Sep 30, 2019

In case puppet is using the proxy, then you might want to use the --proxy param in curl to reproduce the same scenario. Can you try that?

@pschyska
Copy link
Author

We are setting http_proxy and https_proxy, and curl uses that already Connected to proxy (10.0.1.213) port 3128 (#0). Puppet does too, there would be no internet connectivity otherwise.

@albertvaka
Copy link
Contributor

albertvaka commented Sep 30, 2019

Your puppet-server-release.jar bundles the Ruby 1.9 library, which didn't handle the http_proxy env var, actually. However, we try to detect that and set up the proxy ourselves [1], but it might be broken for some reason (eg: the RUBY_VERSION of the Ruby interpreter running, which is what we check, is newer than the version of the Ruby library, which is what is used). Maybe that's something you can try to debug?

[1] https://github.com/DataDog/dogapi-rb/blob/1.36/lib/dogapi/common.rb#L84

@albertvaka
Copy link
Contributor

Note that Datadog's endpoint requires TLS 1.2 or better, and the puppet version you are using is quite old so it might not support that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants