Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase request timeout #137

Closed
kongslund opened this issue May 26, 2015 · 2 comments
Closed

Increase request timeout #137

kongslund opened this issue May 26, 2015 · 2 comments
Labels
Expeditor: Skip Version Bump Used to skip built_in:bump_version

Comments

@kongslund
Copy link
Contributor

Chef Zero is using WEBrick which by default has a request timeout limit of 30 seconds. I would like to propose that Chef Zero changes this limit to 300 seconds in order to match the default rest_timeout setting of chef-client and so that I can avoid having failed tests in my CI environment due to this timeout.

Why? Because I've encountered this timeout limit on a fairly regular basis when using chef-client -z on a Chef Provisioning recipe that creates 10 nodes and then initiates a converge on each of them. The timeout happens during cookbook synchronization before any recipe is actually executed. Each node fetches 18 recipes - a total of 4 Mb per node - which is not really a lot but for some reason that takes around 5 minutes and sometimes gives the following error:

    [83-back1] [2015-05-21T10:20:52+00:00] INFO: Storing updated cookbooks/yum-mysql-community/attributes/mysql-connectors-community.rb in the cache.
    [84-user1] [2015-05-21T10:20:52+00:00] INFO: Storing updated cookbooks/postgresql/files/default/tests/minitest/server_test.rb in the cache.
    [87-moni] [2015-05-21T10:20:53+00:00] INFO: Storing updated cookbooks/database/templates/default/ebs-db-restore.sh.erb in the cache.
                       - database
    [85-back2] [2015-05-21T10:20:53+00:00] INFO: Storing updated cookbooks/postgresql/recipes/server.rb in the cache.
    [83-back1] [2015-05-21T10:20:53+00:00] INFO: HTTP Request Returned 408 Request Timeout : 
                                                ================================================================================
                        Error Syncing Cookbooks:
                        ================================================================================

                        Unexpected API Request Failure:
                        -------------------------------
                        <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
                        <HTML>
                          <HEAD><TITLE>Request Timeout</TITLE></HEAD>
                          <BODY>
                            <H1>Request Timeout</H1>
                            WEBrick::HTTPStatus::RequestTimeout
                            <HR>
                            <ADDRESS>
                             WEBrick/1.3.1 (Ruby/2.1.4/2014-10-27) OpenSSL/1.0.1m at
                             release-image:80
                            </ADDRESS>
                          </BODY>
                        </HTML>


                        Running handlers:
                      [2015-05-21T10:20:53+00:00] ERROR: Running exception handlers
                        [2015-05-21T10:20:53+00:00] INFO: Storing updated cookbooks/yum-mysql-community/attributes/mysql56-community.rb in the cache.
                      Running handlers complete
                      [2015-05-21T10:20:53+00:00] ERROR: Exception handlers complete
                      [2015-05-21T10:20:53+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
                        Chef Client failed. 0 resources updated in 218.688218568 seconds
                      [2015-05-21T10:20:53+00:00] ERROR: 408 "Request Timeout "
                      [2015-05-21T10:20:53+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

When increasing the timeout limit (by temporarily monkey patching chef-client) the issue disappears.

Setting the request timeout can be done like this:

@chef_zero_server.server.config[:RequestTimeout] = 300

It still takes around 5 minutes for cookbook synchronization to complete which is strangely slow but at least it completes without timeout errors.
@kongslund kongslund mentioned this issue May 26, 2015
Merged
@ghempton
Copy link

👍 we are hitting this issue as well.

ghempton pushed a commit to getoutreach/chef-zero that referenced this issue Aug 18, 2015
@kongslund @ghempton
Increase WEBrick request timeout to 300 seconds. Fixes chef#137
ff9c5e5
@kongslund
Copy link
Contributor Author

@ghempton

We perform the monkey patching in our client.rb:

require_relative 'monkey_patch_request_timeout'

The monkey patch file, monkey_patch_request_timeout.rb, looks like this:

# Monkey patch /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.2.1/lib/chef/local_mode.rb
# Default timeout is 30.
class Chef
  module LocalMode
    def self.with_server_connectivity
      setup_server_connectivity
      @chef_zero_server.server.config[:RequestTimeout] = 300
      begin
        yield
      ensure
        destroy_server_connectivity
      end
    end
  end
end

I hope this helps.

thommay added a commit that referenced this issue Jul 19, 2016
@thommay
Merge pull request #138 from kongslund/request-timeout
1b1bc10 
Increase WEBrick request timeout to 300 seconds. Fixes #137
@poliva83 poliva83 mentioned this issue Aug 27, 2016
Closed
@thommay thommay added Expeditor: Skip Version Bump Used to skip built_in:bump_version and removed Meta: Exclude From Changelog labels Sep 22, 2017
@chef chef locked and limited conversation to collaborators Nov 16, 2017
@chef chef deleted a comment from ghempton Jan 23, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Expeditor: Skip Version Bump Used to skip built_in:bump_version
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@thommay @ghempton @kongslund