NoMethodError: undefined method `sigv4_name'

[lardcanoe]

I am getting this error in production when run at scale. I can't reproduce locally. When it prod, creating a new Aws::Ec2 class and calling the describe_instances call may work 50 times in a row for a specific aws account, but then throw this exception, and then the next call work again.

I'm looking for some help in debugging. Anything in particular to try or look at.

Thanks!

::Aws::EC2.new({
          region: region['regionName'],
          credentials: creds,
          ssl_verify_peer: false
        }).describe_instances.each do |page|

NoMethodError: undefined method `sigv4_name' for #<#<Class:0x524347e0>:0x49fa5679>
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/lib/aws/signers/v4.rb" line 10 in sign
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/lib/aws/signers/handler.rb" line 12 in call
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/lib/aws/error_handler.rb" line 12 in call
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/lib/aws/plugins/retry_errors.rb" line 78 in call
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/lib/aws/response_handler.rb" line 12 in call
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/lib/aws/request_handler.rb" line 14 in call
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/lib/aws/plugins/response_paging.rb" line 9 in call
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/lib/aws/plugins/user_agent.rb" line 12 in call
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/vendor/seahorse/lib/seahorse/client/plugins/restful_bindings.rb" line 13 in call
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/vendor/seahorse/lib/seahorse/client/plugins/endpoint.rb" line 27 in call
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/vendor/seahorse/lib/seahorse/client/plugins/param_validation.rb" line 22 in call
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/vendor/seahorse/lib/seahorse/client/plugins/raise_response_errors.rb" line 14 in call
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/vendor/seahorse/lib/seahorse/client/plugins/param_conversion.rb" line 22 in call
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/vendor/seahorse/lib/seahorse/client/request.rb" line 70 in send_request
File "/home/releases/127be001dfb1d900f71717dd2cf4d21eac4a2cc1/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.0.0.rc8/vendor/seahorse/lib/seahorse/client/plugins/operation_methods.rb" line 42 in describe_instances

[trevorrowe]

Can you tell me a bit more about your environment? I see you are using JRuby. Are you using threads? Anything else not obvious from the code snippet above?

[lardcanoe]

JRuby 1.7.12
Using threads. 45 per server.

Also, when the process is starting up, before any worker threads are created, we run:

ec2 = ::Aws::EC2.new({
  region: 'us-east-1',
  credentials: ::Aws::Credentials.new('aws_access_key_id', 'secret_access_key')
})

elastic_cache = ::Aws::ElastiCache.new({
  region: 'us-east-1',
  credentials: ::Aws::Credentials.new('aws_access_key_id', 'secret_access_key')
})

to force loading of code. Without this, we usually get errors related to missing methods for describe_cache_cluster and describe_instances.

[trevorrowe]

There appears to be two issues going on here.

The first issue is the undefined operation methods. That one, I believe I have found the core issue in the Seahorse::Client::Plugin::OperationMethods plugin. This plugin adds operation methods to the client class.

The second issue, the primary issue you reported, is a bit more complex. The key information is the stack trace indicates that the ec2 client is running the v4 signer handler. It should be executing the v2 signer handler instead. I'm going to keep digging into a possible cause for this. Hopefully I can get a patch posted soon.

[trevorrowe]

At the very bottom of the commit 9a9e4d4, is a seemingly unrelated change. While writing the plugin to support pre-signed urls for EC2#copy_snapshot, I ran into an issue where a plugin could reuse the same instance of a handler. That commit resolved this issue, but a new RC has not been published since the fix.

I have verified this commit resolves the issue you are experiencing by first reproducing the bug and then using the tip of master where the bug no-longer exists. I'll try cutting a release shortly. This should resolve the thread safety issues you have encountered.

[trevorrowe]

The fix is live. Please re-open this issue if the issue persists after updating to the latest release.