Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fieri ssl errors #1557

Open
kplimack opened this issue Mar 6, 2017 · 13 comments
Open

fieri ssl errors #1557

kplimack opened this issue Mar 6, 2017 · 13 comments

Comments

@kplimack
Copy link

kplimack commented Mar 6, 2017 •
edited by robbkidd
Loading

supermarket-2.9.7-1.el6.x86_64

2017-03-06_02:28:10.77163 2017-03-06T02:28:10.771Z 30588 TID-zrkxk WARN: {"class":"MetricsRunner","args":[{"cookbook_name":"dmg","cookbook_version":"2.2.0","cookbook_artifact_url":"https://supermarket.myOrg.com/system/cookbook_versions/tarballs/670/original/dmg.tgz?1488766954","controller":"fieri/jobs","action":"create"}],"retry":true,"queue":"default","jid":"9507a140bf0413f9be0c5079","created_at":1488766954.5832617,"enqueued_at":1488767290.766189,"error_message":"SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol","error_class":"OpenSSL::SSL::SSLError","failed_at":1488766954.5861578,"retry_count":4,"retried_at":1488767290.7709894}
2017-03-06_02:28:10.77177 2017-03-06T02:28:10.771Z 30588 TID-zrkxk WARN: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol
2017-03-06_02:28:10.77179 2017-03-06T02:28:10.771Z 30588 TID-zrkxk WARN: /opt/supermarket/embedded/lib/ruby/2.3.0/net/http.rb:933:in `connect_nonblock'
2017-03-06_02:28:10.77180 /opt/supermarket/embedded/lib/ruby/2.3.0/net/http.rb:933:in `connect'
2017-03-06_02:28:10.77180 /opt/supermarket/embedded/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
2017-03-06_02:28:10.77180 /opt/supermarket/embedded/lib/ruby/2.3.0/net/http.rb:852:in `start'
2017-03-06_02:28:10.77181 /opt/supermarket/embedded/lib/ruby/2.3.0/net/http.rb:584:in `start'
2017-03-06_02:28:10.77181 /opt/supermarket/embedded/lib/ruby/2.3.0/net/http.rb:479:in `get_response'
2017-03-06_02:28:10.77181 /opt/supermarket/embedded/lib/ruby/2.3.0/net/http.rb:456:in `get'
2017-03-06_02:28:10.77181 /opt/supermarket/embedded/service/supermarket/vendor/cache/fieri/app/models/supermarket_api_runner.rb:24:in `get_api_response'
2017-03-06_02:28:10.77182 /opt/supermarket/embedded/service/supermarket/vendor/cache/fieri/app/models/supermarket_api_runner.rb:6:in `cookbook_api_response'
2017-03-06_02:28:10.77182 /opt/supermarket/embedded/service/supermarket/vendor/cache/fieri/app/models/metrics_runner.rb:20:in `cookbook_api_response'
2017-03-06_02:28:10.77183 /opt/supermarket/embedded/service/supermarket/vendor/cache/fieri/app/models/metrics_runner.rb:7:in `perform'
2017-03-06_02:28:10.77184 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:152:in `execute_job'
2017-03-06_02:28:10.77184 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:134:in `block (2 levels) in process'
2017-03-06_02:28:10.77184 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
2017-03-06_02:28:10.77184 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidetiq-0.7.2/lib/sidetiq/middleware/history.rb:8:in `call'
2017-03-06_02:28:10.77186 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2017-03-06_02:28:10.77186 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
2017-03-06_02:28:10.77187 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2017-03-06_02:28:10.77187 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
2017-03-06_02:28:10.77187 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/logging.rb:32:in `with_context'
2017-03-06_02:28:10.77187 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/server/logging.rb:7:in `call'
2017-03-06_02:28:10.77188 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2017-03-06_02:28:10.77188 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:133:in `invoke'
2017-03-06_02:28:10.77188 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:129:in `block in process'
2017-03-06_02:28:10.77188 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:168:in `stats'
2017-03-06_02:28:10.77190 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:128:in `process'
2017-03-06_02:28:10.77190 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:80:in `process_one'
2017-03-06_02:28:10.77190 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:68:in `run'
2017-03-06_02:28:10.77191 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/util.rb:17:in `watchdog'
2017-03-06_02:28:10.77191 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-03-06_02:28:10.77256 2017-03-06T02:28:10.772Z 30588 TID-ujnyo MetricsRunner JID-eedf88110b5276f6c179c972 INFO: start
2017-03-06_02:28:10.77547 2017-03-06T02:28:10.775Z 30588 TID-ujnyo MetricsRunner JID-eedf88110b5276f6c179c972 INFO: fail: 0.003 sec
@robbkidd
Copy link
Contributor

robbkidd commented Mar 6, 2017

A couple questions:

  1. Do you want Fieri—the component that runs a growing number of community quality metrics—running in your private Supermarket? If not, omit fieri from the list of features enabled in the features attribute.
  2. With HTTPS enabled, are you using a certificate for https://supermarket.myOrg.com signed by a trusted Certificate Authority?

@kplimack
Copy link
Author

kplimack commented Mar 6, 2017

@robbkidd

  1. I think i want it. foodcritic output isnt super helpful since my cookbooks go through CI to make their way to the supermarket and they must pass foodcritic, etc there. But id like to enable it and see for myself, or if any additional metrics make their way in.

  2. my chef server and supermarket are sharing the same wildcard cert from an external CA, so i "assume" that it should work.

@nellshamrell
Copy link
Contributor

@kplimack does your Supermarket instance run behind a load balancer? If it does, does SSL termination happen at the load balancer, or on the instance that Supermarket is running on?

@kplimack
Copy link
Author

kplimack commented Mar 7, 2017

No. It's accessible without an LB

@nellshamrell
Copy link
Contributor

Could you enter the rails console on the instance you are running Supermarket on? You can do this by SSH'ing to the instance and running $ sudo -u supermarket supermarket-ctl console

Could you let me know what it returns when you run this from within the console?

ENV['FIERI_URL']

@kplimack
Copy link
Author

kplimack commented Mar 7, 2017 •
edited
Loading

@nellshamrell
looks like i also need to track down the source of all this spam about port 7777, too.

ENV['FIERI_URL']E, [2017-03-07T20:44:11.903797 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)

=> "http://localhost:13000/fieri/jobs"
irb(main):002:0> E, [2017-03-07T20:44:12.905355 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)
E, [2017-03-07T20:44:13.906924 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)
E, [2017-03-07T20:44:14.908478 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)
E, [2017-03-07T20:44:15.909958 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)
E, [2017-03-07T20:44:16.911600 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)
E, [2017-03-07T20:44:17.913218 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)
E, [2017-03-07T20:44:18.914706 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)
E, [2017-03-07T20:44:19.916183 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)

[jplimack@supermarket]~% curl -I http://localhost:13000/fieri/status
HTTP/1.1 200 OK

@robbkidd
Copy link
Contributor

robbkidd commented Mar 7, 2017

@kplimack Failed to open TCP connection to localhost:7777 is a byproduct of adding a monitoring agent to Supermarket. We have an issue open on that agent and will make a new release when the agent is updated with a fix.

@nellshamrell
Copy link
Contributor

Alright, that FIERI_URL looks correct, and ty for demonstrating that you can curl it successfully. Taking another look at the stack trace.

@nellshamrell
Copy link
Contributor

What do you have for ENV['FIERI_SUPERMARKET_ENDPOINT']? After taking another look at your stack trace, that seems to be what it is choking on.

@kplimack
Copy link
Author

kplimack commented Mar 8, 2017

@nellshamrell
"https://localhost:13000"

@robbkidd
Copy link
Contributor

robbkidd commented Mar 8, 2017

That's the default set in the omnibus install for FIERI_SUPERMARKET_ENDPOINT and I think it that may be the wrong default to set. The Rails app service running on localhost:13000 does not do SSL; SSL is handled higher up in the stack by nginx.

@kplimack If you were to override the fieri_supermarket_endpoint attribute—e.g. the same way you override fqdn—to set it to http://localhost:13000, I suspect the SSL error would stop and fieri job results would get posted successfully. If you try that, let us know if it succeeds (or doesn't). We'll get a fix in for that.

@kplimack
Copy link
Author

kplimack commented Mar 8, 2017

@robbkidd thanks. I'll try that tomorrow. Would you happen to have any docs on integrating supermarket into a ci/cd pipeline?

@kplimack
Copy link
Author

kplimack commented Mar 8, 2017

@robbkidd That seems to have done the trick!
I would love it if you had any docs on CI/CD workflows using a supermarket. In a chef-zero environment, it makes total sense, but in a client/server setup its role is not as well defined.

@tas50 tas50 removed the Question label Jan 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@robbkidd @nellshamrell @tas50 @kplimack