Traceback on salt-run cloud.destroy $host #10947

clearclaw · 2014-03-04T22:01:06Z

[ERROR   ] An un-handled exception was caught by salt's global exception handler:
KeyError: 'instanceId'
Traceback (most recent call last):
  File "/usr/local/bin/salt-run", line 10, in <module>
    salt_run()
  File "/usr/lib/pymodules/python2.7/salt/scripts.py", line 95, in salt_run
    client.run()
  File "/usr/lib/pymodules/python2.7/salt/cli/__init__.py", line 357, in run
    runner.run()
  File "/usr/lib/pymodules/python2.7/salt/runner.py", line 219, in run
    self.opts['fun'], self.opts['arg'], self.opts)
  File "/usr/lib/pymodules/python2.7/salt/runner.py", line 126, in cmd
    return self.functions[fun](*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/salt/runners/cloud.py", line 96, in destroy
    info = client.destroy(names)
  File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 151, in destroy
    mapper.destroy(names))
  File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 645, in destroy
    ret = self.clouds[fun](name)
  File "/usr/lib/pymodules/python2.7/salt/cloud/clouds/ec2.py", line 1769, in destroy
    instance_id = node_metadata['instanceId']
KeyError: 'instanceId'
Traceback (most recent call last):
  File "/usr/local/bin/salt-run", line 10, in <module>
    salt_run()
  File "/usr/lib/pymodules/python2.7/salt/scripts.py", line 95, in salt_run
    client.run()
  File "/usr/lib/pymodules/python2.7/salt/cli/__init__.py", line 357, in run
    runner.run()
  File "/usr/lib/pymodules/python2.7/salt/runner.py", line 219, in run
    self.opts['fun'], self.opts['arg'], self.opts)
  File "/usr/lib/pymodules/python2.7/salt/runner.py", line 126, in cmd
    return self.functions[fun](*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/salt/runners/cloud.py", line 96, in destroy
    info = client.destroy(names)
  File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 151, in destroy
    mapper.destroy(names))
  File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 645, in destroy
    ret = self.clouds[fun](name)
  File "/usr/lib/pymodules/python2.7/salt/cloud/clouds/ec2.py", line 1769, in destroy
    instance_id = node_metadata['instanceId']
KeyError: 'instanceId'

The text was updated successfully, but these errors were encountered:

clearclaw · 2014-03-04T22:01:30Z

This is with the latest 2014.1 packages under Ubuntu.

clearclaw · 2014-03-04T22:10:02Z

This also results in renamed_keys (from destroy) not being removed even though

delete_sshkeys: True

is set in /etc/salt/cloud .

basepi · 2014-03-05T18:29:26Z

Thanks for the report!

@techhat ping

tateeskew · 2014-03-12T20:24:50Z

Any update on this? This is happening here as well. Exact same problem.
Here's the salt-cloud versions-report

root@saltmaster:~# salt-cloud --versions-report
            Salt: 2014.1.0
          Python: 2.7.3 (default, Feb 27 2014, 19:58:35)
          Jinja2: 2.7.2
        M2Crypto: 0.21.1
  msgpack-python: 0.4.1
    msgpack-pure: Not Installed
        pycrypto: 2.6.1
          PyYAML: 3.10
           PyZMQ: 14.1.0
             ZMQ: 4.0.4
 Apache Libcloud: 0.14.1

clearclaw · 2014-03-13T00:41:28Z

And for completeness:

$ salt-run --versions-report
           Salt: 2014.1.0
         Python: 2.7.3 (default, Feb 27 2014, 19:58:35)
         Jinja2: 2.6
       M2Crypto: 0.21.1
 msgpack-python: 0.1.10
   msgpack-pure: Not Installed
       pycrypto: 2.4.1
         PyYAML: 3.10
          PyZMQ: 13.0.0
            ZMQ: 3.2.2

clearclaw · 2014-03-13T00:42:21Z

BTW do you also get the traceback-is-shown-twice behaviour?

AnneTheAgile · 2014-03-15T18:55:44Z

@clearclaw , I think you meant to ask @tateeskew if he gets the traceback-is-shown-twice behaviour?

Also, do you know for sure which version the error began? Or maybe you don't try to cloud.destroy very often?

//here to help via CodeTriage

AnneTheAgile · 2014-03-15T18:59:03Z

Hmm, now I see that destroy was only added to the runner in 2014.01.
9dc9f7b
It doesn't do too much right there;

def destroy(names):
    '''
    Destroy the named vm(s)
    '''
    client = _get_client()
    info = client.destroy(names)
    return info

AnneTheAgile · 2014-03-15T19:20:59Z

Things I notice;
TLDR = #1 only ;)
1.There was a very new commit to fix; salt.cloud: race if item was removed in another loop
94c3c68
It seems possibly related to this problem of not finding the 'cloud'?
94c3c68 » kiorky 2014-03-01 salt.cloud: race if item was removed in another loop
796 if name in names: 797 names.remove(name)

2.From the stacktrace, the calling method fails on line 1769 which is the last DocString line abve in my prior message,
File "/usr/lib/pymodules/python2.7/salt/cloud/clouds/ec2.py", line 1769, in destroy
3.The next stack item seems to have a typo in the message, there should be a space before 'fun' which was the argument;
File "/usr/lib/pymodules/python2.7/salt/cloud/init.py", line 645, in destroy
ret = self.cloudsfun
4.Item [3] is modified very frequently.
https://github.com/saltstack/salt/commits/93d7ff22ecc7af0642537dfacbbfb86456bf8396/salt/cloud/__init__.py
5.Inside [3] there are 3 destroy* methods, but the plain one was added in Nov., 411e312 thatch45
2013-11-15 Add destroy to base cloud client api 267 def destroy(self, names):
6.It's now on line 267 and I didn't check to ensure line 645 used to be correct.

kiorky · 2014-03-15T19:56:53Z

94c3c68 cannot hurt you ... it can only give you less tracebacks as you would have keyerrors more over all the errors you have.
Problem is that i dont suceed in decrypting your comments which are not very clear :).

AnneTheAgile · 2014-03-15T21:49:43Z

thank you for your quick reply, @kiorky ! I agree, that commit seems very good, but the users in this thread do not yet have it, right? The errors they see are Key Errors, which in turn appear to be due to the cloud not found, per my research.

I'm sorry if my comments were not clear :). I was making notes on the stacktrace, looking at each line and checking for possible failure causes. The item#1 is the summary.

tateeskew · 2014-04-08T16:12:50Z

This is still happening here:

    root@saltmaster-vpc:~# salt-cloud -d p-aws-emmaplatformsingle06
    [INFO    ] salt-cloud starting
    The following virtual machines are set to be destroyed:
      dev-emma-aws-us-east-1a:
        ec2:
          p-aws-emmaplatformsingle06
      dev-emma-aws-us-east-1b:
        ec2:
          p-aws-emmaplatformsingle06
      dev-emma-aws-us-east-1c:
        ec2:
          p-aws-emmaplatformsingle06
      dev-emma-aws-us-east-1d:
        ec2:
          p-aws-emmaplatformsingle06
      dev-emma-aws-us-east-1e:
        ec2:
          p-aws-emmaplatformsingle06
      emma-aws-production-us-east-1e:
        ec2:
          p-aws-emmaplatformsingle06
      emma-aws-production-us-east-1d:
        ec2:
          p-aws-emmaplatformsingle06
      emma-aws-production-us-east-1c:
        ec2:
          p-aws-emmaplatformsingle06
      emma-aws-production-us-east-1b:
        ec2:
          p-aws-emmaplatformsingle06
      emma-aws-production-us-east-1a:
        ec2:
          p-aws-emmaplatformsingle06

    Proceed? [N/y] y
    ... proceeding
    [INFO    ] Destroying in non-parallel mode.
    [INFO    ] Renaming p-aws-emmaplatformsingle06 to p-aws-emmaplatformsingle06-    DEL696fc02ca1b5495cb8801494ac138da6
    [INFO    ] Machine will be identified as p-aws-emmaplatformsingle06-    DEL696fc02ca1b5495cb8801494ac138da6 until it has been cleaned up.
    [INFO    ] [{'instanceId': 'i-c41dcfe7', 'currentState': {'code': '32', 'name': 'shutting-down'},     'previousState': {'code': '16', 'name': 'running'}}]
    [ERROR   ] There was an error destroying machines: 'instanceId'
    Traceback (most recent call last):
      File "/usr/lib/pymodules/python2.7/salt/cloud/cli.py", line 267, in run
        ret = mapper.destroy(names, cached=True)
      File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 645, in destroy
        ret = self.clouds[fun](name)
      File "/usr/lib/pymodules/python2.7/salt/cloud/clouds/ec2.py", line 1805, in destroy
        instance_id = node_metadata['instanceId']
    KeyError: 'instanceId'



    root@saltmaster-vpc:~# salt-key
    Accepted Keys:
    d-aws-emmaplatformsingle
    p-aws-emmaplatformsingle01
    p-aws-emmaplatformsingle02
    p-aws-emmaplatformsingle03
    p-aws-emmaplatformsingle04
    p-aws-emmaplatformsingle05
    p-aws-emmaplatformsingle06-DEL696fc02ca1b5495cb8801494ac138da6
    s-aws-emmaplatformsingle01
    s-aws-emmaplatformsingle02
    s-aws-emmaplatformsingle03
    Unaccepted Keys:
    Rejected Keys:

root@saltmaster-vpc:~# salt-cloud --versions-report
            Salt: 2014.1.1
          Python: 2.7.3 (default, Feb 27 2014, 19:58:35)
          Jinja2: 2.6
        M2Crypto: 0.21.1
  msgpack-python: 0.4.1
    msgpack-pure: Not Installed
        pycrypto: 2.4.1
          PyYAML: 3.10
           PyZMQ: 14.0.1
             ZMQ: 4.0.3
 Apache Libcloud: 0.14.1

root@saltmaster-vpc:~# lsb_release -a
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.4 LTS
Release:        12.04
Codename:       precise

AnneTheAgile · 2014-04-08T17:54:23Z

Hmm, rats, so I presume the commit we were discussing, being authored a month ago, 94c3c68 1 parent 0d8661b
was included in 2014.1.1, which was tagged 21 days ago.

That indicates we definitely need a new fix. Ty @tateeskew for all your details!

basepi · 2014-04-08T22:08:42Z

@AnneTheAgile That's actually not true. It appears that 94c3c68 was not cherry-picked into the 2014.1 branch (and so is not in 2014.1.1). Our dot releases are not cut from develop, but rather from a release branch where we cherry-pick fixes from develop.

I have now cherry-picked that commit, and it will be in 2014.1.2.

AnneTheAgile · 2014-04-17T01:51:18Z

@basepi thank you that's great news!

basepi · 2014-06-11T21:33:47Z

Can someone test this again? I think since I cherry-picked the commit mentioned above that this is resolved.

clearclaw · 2014-07-11T22:02:55Z

This is still a problem with 2014.1.7 :

Unable to capture event: Missing required attribute in authentication header: sentry_secret
Unable to capture event: Missing required attribute in authentication header: sentry_secret
[ERROR   ] Unable to capture event: Missing required attribute in authentication header: sentry_secret
Failed to submit message: u'An un-handled exception was caught by salt\'s global exception handler:\nKeyError: \'instanceId\'\nTraceback (most recent call last):\n  File "/usr/local/bin/salt-run", line 10, in <module>\n    salt_run()\n  File "/usr/lib/pymodules/python2.7/salt/scripts.py", line 95, in salt_run\n    client.run()\n  File "/usr/lib/pymodules/python2.7/salt/cli/__init__.py", line 362, in run\n    runner.run()\n  File "/usr/lib/pymodules/python2.7/salt/runner.py", line 218, in run\n    self.opts[\'fun\'], self.opts[\'arg\'], self.opts)\n  File "/usr/lib/pymodules/python2.7/salt/runner.py", line 125, in cmd\n    return self.functions[fun](*args, **kwargs)\n  File "/usr/lib/pymodules/python2.7/salt/runners/cloud.py", line 96, in destroy\n    info = client.destroy(names)\n  File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 151, in destroy\n    mapper.destroy(names))\n  File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 646, in destroy\n    ret = self.clouds[fun](name)\n  File "/usr/lib/pymodules/python2.7/salt/cloud/clouds/ec2.py", line 1800, in destroy\n    instance_id = node_metadata[\'instanceId\']\nKeyError: \'instanceId\''
Failed to submit message: u'An un-handled exception was caught by salt\'s global exception handler:\nKeyError: \'instanceId\'\nTraceback (most recent call last):\n  File "/usr/local/bin/salt-run", line 10, in <module>\n    salt_run()\n  File "/usr/lib/pymodules/python2.7/salt/scripts.py", line 95, in salt_run\n    client.run()\n  File "/usr/lib/pymodules/python2.7/salt/cli/__init__.py", line 362, in run\n    runner.run()\n  File "/usr/lib/pymodules/python2.7/salt/runner.py", line 218, in run\n    self.opts[\'fun\'], self.opts[\'arg\'], self.opts)\n  File "/usr/lib/pymodules/python2.7/salt/runner.py", line 125, in cmd\n    return self.functions[fun](*args, **kwargs)\n  File "/usr/lib/pymodules/python2.7/salt/runners/cloud.py", line 96, in destroy\n    info = client.destroy(names)\n  File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 151, in destroy\n    mapper.destroy(names))\n  File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 646, in destroy\n    ret = self.clouds[fun](name)\n  File "/usr/lib/pymodules/python2.7/salt/cloud/clouds/ec2.py", line 1800, in destroy\n    instance_id = node_metadata[\'instanceId\']\nKeyError: \'instanceId\''
[ERROR   ] Failed to submit message: u'An un-handled exception was caught by salt\'s global exception handler:\nKeyError: \'instanceId\'\nTraceback (most recent call last):\n  File "/usr/local/bin/salt-run", line 10, in <module>\n    salt_run()\n  File "/usr/lib/pymodules/python2.7/salt/scripts.py", line 95, in salt_run\n    client.run()\n  File "/usr/lib/pymodules/python2.7/salt/cli/__init__.py", line 362, in run\n    runner.run()\n  File "/usr/lib/pymodules/python2.7/salt/runner.py", line 218, in run\n    self.opts[\'fun\'], self.opts[\'arg\'], self.opts)\n  File "/usr/lib/pymodules/python2.7/salt/runner.py", line 125, in cmd\n    return self.functions[fun](*args, **kwargs)\n  File "/usr/lib/pymodules/python2.7/salt/runners/cloud.py", line 96, in destroy\n    info = client.destroy(names)\n  File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 151, in destroy\n    mapper.destroy(names))\n  File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 646, in destroy\n    ret = self.clouds[fun](name)\n  File "/usr/lib/pymodules/python2.7/salt/cloud/clouds/ec2.py", line 1800, in destroy\n    instance_id = node_metadata[\'instanceId\']\nKeyError: \'instanceId\''
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
KeyError: 'instanceId'
Traceback (most recent call last):
  File "/usr/local/bin/salt-run", line 10, in <module>
    salt_run()
  File "/usr/lib/pymodules/python2.7/salt/scripts.py", line 95, in salt_run
    client.run()
  File "/usr/lib/pymodules/python2.7/salt/cli/__init__.py", line 362, in run
    runner.run()
  File "/usr/lib/pymodules/python2.7/salt/runner.py", line 218, in run
    self.opts['fun'], self.opts['arg'], self.opts)
  File "/usr/lib/pymodules/python2.7/salt/runner.py", line 125, in cmd
    return self.functions[fun](*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/salt/runners/cloud.py", line 96, in destroy
    info = client.destroy(names)
  File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 151, in destroy
    mapper.destroy(names))
  File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 646, in destroy
    ret = self.clouds[fun](name)
  File "/usr/lib/pymodules/python2.7/salt/cloud/clouds/ec2.py", line 1800, in destroy
    instance_id = node_metadata['instanceId']
KeyError: 'instanceId'
Traceback (most recent call last):
  File "/usr/local/bin/salt-run", line 10, in <module>
    salt_run()
  File "/usr/lib/pymodules/python2.7/salt/scripts.py", line 95, in salt_run
    client.run()
  File "/usr/lib/pymodules/python2.7/salt/cli/__init__.py", line 362, in run
    runner.run()
  File "/usr/lib/pymodules/python2.7/salt/runner.py", line 218, in run
    self.opts['fun'], self.opts['arg'], self.opts)
  File "/usr/lib/pymodules/python2.7/salt/runner.py", line 125, in cmd
    return self.functions[fun](*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/salt/runners/cloud.py", line 96, in destroy
    info = client.destroy(names)
  File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 151, in destroy
    mapper.destroy(names))
  File "/usr/lib/pymodules/python2.7/salt/cloud/__init__.py", line 646, in destroy
    ret = self.clouds[fun](name)
  File "/usr/lib/pymodules/python2.7/salt/cloud/clouds/ec2.py", line 1800, in destroy
    instance_id = node_metadata['instanceId']
KeyError: 'instanceId'

rallytime · 2015-06-02T14:39:40Z

I am going around to some older issues. Is this still an issue on some newer releases? Has anyone run into this one on 2014.7.x or, even better to test, 2015.5.x releases? The 2014.1 branch has been retired, and we will no longer be releasing from 2014.7, either. Is this still a problem or can this be closed?

rallytime · 2015-06-16T23:19:09Z

Since I didn't hear back here, and I think this is fixed, I am going to close this. If this remains an issue for anyone, please comment and we will be glad to re-open and address any concerns.

basepi added Bug labels Mar 5, 2014

basepi added this to the Outstanding Bugs milestone Mar 5, 2014

basepi modified the milestones: Approved, Outstanding Bugs Apr 21, 2014

rallytime added severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around fixed-pls-verify fix is linked, bug author to confirm fix P4 Priority 4 and removed severity-low 4th level, cosemtic problems, work around exists labels Jun 2, 2015

jfindlay added the RIoT Relates to integration with cloud providers, hypervisors, API-based services, etc. label Jun 9, 2015

rallytime closed this as completed Jun 16, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Traceback on salt-run cloud.destroy $host #10947

Traceback on salt-run cloud.destroy $host #10947

clearclaw commented Mar 4, 2014

clearclaw commented Mar 4, 2014

clearclaw commented Mar 4, 2014

basepi commented Mar 5, 2014

tateeskew commented Mar 12, 2014

clearclaw commented Mar 13, 2014

clearclaw commented Mar 13, 2014

AnneTheAgile commented Mar 15, 2014

AnneTheAgile commented Mar 15, 2014

AnneTheAgile commented Mar 15, 2014

kiorky commented Mar 15, 2014

AnneTheAgile commented Mar 15, 2014

tateeskew commented Apr 8, 2014

AnneTheAgile commented Apr 8, 2014

basepi commented Apr 8, 2014

AnneTheAgile commented Apr 17, 2014

basepi commented Jun 11, 2014

clearclaw commented Jul 11, 2014

rallytime commented Jun 2, 2015

rallytime commented Jun 16, 2015

Traceback on salt-run cloud.destroy $host #10947

Traceback on salt-run cloud.destroy $host #10947

Comments

clearclaw commented Mar 4, 2014

clearclaw commented Mar 4, 2014

clearclaw commented Mar 4, 2014

basepi commented Mar 5, 2014

tateeskew commented Mar 12, 2014

clearclaw commented Mar 13, 2014

clearclaw commented Mar 13, 2014

AnneTheAgile commented Mar 15, 2014

AnneTheAgile commented Mar 15, 2014

AnneTheAgile commented Mar 15, 2014

kiorky commented Mar 15, 2014

AnneTheAgile commented Mar 15, 2014

tateeskew commented Apr 8, 2014

AnneTheAgile commented Apr 8, 2014

basepi commented Apr 8, 2014

AnneTheAgile commented Apr 17, 2014

basepi commented Jun 11, 2014

clearclaw commented Jul 11, 2014

rallytime commented Jun 2, 2015

rallytime commented Jun 16, 2015