Add support for spot instance

[EmilVarona]

Just came across this useful tool. Can you please tell me if you plan to add support for reporting on spot instances when running awslimitchecker -u? Most of our instances are spot and this would help very much. Thanks for considering this.

[jantman]

@EmilVarona I hadn't planned on it, but I'd have nothing against it. The main problem is that I don't have access to an AWS account with any spot instances - we don't use them at all. So I'd either need someone to cut a pull request for this, or else I'd need someone to:

1. confirm how the spot instance limits work, as the docs say they're both per-region and per-instance-type, but the Spot Instance Limits docs only list one limit, "The total number of Spot Instance requests" with a limit of 20 per region; and
2. help me with testing the code, as I don't have any spot instances

-Jason

[EmilVarona]

Thanks for considering this Jason. I can give it a shot but my time it limited and my coding skilz, well, a bit less than adequate. I'll reach out to you directly for questions.

Cheers,
--Emil

[jantman]

Emil,

Ok. If you can handle helping me figure out my question (#1 above) about the limits we'd need to check and how to calculate them, and you're willing to help test the code by installing off of a branch, I can probably give it a shot.

[jleader]

It looks like they've added more spot instance limits; the docs now say that there are spot request limits per instance-type and per region. There's a spot bid price limit, which doesn't say whether it's per instance type, region, or global (the fact that it's just a multiplier times the on-demand price suggests that the multiplier might be global). They also list 5 new spot fleet limits, some of which sound like they're per region, and some sound like they're per-fleet.

My employer uses a lot of spot instances, I could probably run test code for you if you're interested.

[jantman]

@jleader Thanks for updating this.

Can you confirm if http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-limits.html lists all of the spot limits?

This is going to be a bit difficult for me to develop, since we don't use any spot limits. But I can probably take a crack at a branch of awslimitchecker that tries to get spot usage, which you could install and tell me if it works right. If we can figure out - or find any public documentation confirming - exactly how the limits are calculated, that would be great. If not, I can try to reach out to a few personal contacts at AWS, but I can't use my employer's support for my F/OSS work.

If anyone else ( @EmilVarona ? ) would be willing to test the branch and see if the usage and limits seem right, it would be greatly appreciated.

I'm really busy this week, but I can try to get something started in the next few days.

[jantman]

I just wanted to comment that on the "Limits" page of the EC2 Console for my employer's two accounts, as well as my two personal accounts, the only spot-related limit listed is the one "Spot instance requests". It also appears that Trusted Advisor doesn't yet include anything spot-related.

[jantman]

@jleader @EmilVarona I've pushed up some initial code for spot instances and fleets in the issue51 branch. You can install this in a new virtualenv with:

pip install git+https://github.com/jantman/awslimitchecker.git@issue51#egg=awslimitchecker

And run (for just EC2) with:

awslimitchecker -vv -S EC2 -u | grep spot

Note that this requires new IAM permissions, ec2:DescribeSpotInstanceRequests and ec2:DescribeSpotFleetRequests.

You should get a bunch of debug output, which includes information about each spot instance and fleet. I'll remove that before this is merged, but thought it might help with testing.

I've implemented (once again, just going by the API) the following limits:

• Max spot instance requests per region
• Max active spot fleets per region
• Max launch specifications per spot fleet
• Max target capacity per spot fleet
• Max target capacity for all spot fleets in region

I've assumed that Max spot instance requests per region only counts requests in open or active state, and that the fleet-related limits (according to my reading of the doc) only count fleets in active state (i.e. I'm totally ignoring any fleets in submitted or modifying state).

[jleader]

It wrote the following to stderr:

• 48 times [WARNING ec2.py:223 - awslimitchecker.services.ec2._instance_usage() ] Spot instance found (i-X); awslimitchecker does not yet support spot instances.
• 112 times [DEBUG ec2.py:127 - awslimitchecker.services.ec2._find_usage_spot_instances() ] NOT counting spot instance request sir-X state=cancelled
• 4 times DEBUG ec2.py:124 - awslimitchecker.services.ec2._find_usage_spot_instances() ] Counting spot instance request sir-X state=active

Here's the stdout after grep'ing for "spot":

EC2/Max active spot fleets per region 0
EC2/Max launch specifications per spot fleet
EC2/Max spot instance requests per region 0
EC2/Max target capacity for all spot fleets in region 0
EC2/Max target capacity per spot fleet

[jantman]

@jleader

Hmm... so the first two ("does not yet support spot instances" and "NOT counting spot instance [...] state=cancelled") are expected.

If you don't have any spot fleet requests, then all of the fleet limits being 0 or (no data) makes sense.

However, "Counting spot instance request sir-X state=active" doesn't make sense with the output of "EC2/Max spot instance requests per region 0".

I ran the code myself, replacing the describe_spot_instance_requests API call with a test fixture API response, and I'm seeing the results I expect... so I'm not terribly sure what's going on here.

I pushed up two commits , c36b2db and 20b9b7f, that add a bit more explicit debugging.

I checked all 20 of my employer's AWS accounts, and confirmed we don't have any spot instance requests.

Sometime in the next few days I'm going to try opening some spot instance requests on my personal account with unrealistically-low bid prices (which should keep them from running in the short term), test this myself, and then cancel them.

[jantman]

@jleader I hacked together a script (admittedly including some things specific to my test account) that creates instance and fleet requests at half of the current market price and a ValidFrom of 2 weeks in the future, so they should never run. As far as I can tell, everything looks to be working correctly.

Could you pull down the latest commits on that branch and run again with the added debug code? I can't seem to find anything that would cause the discrepancy you're seeing.

[jantman]

@jleader @EmilVarona Thanks to some anonymous assistance, I've been able to get access to an account that makes use of spot instances and fleets and test this, and will be able to release experimental support soon. Still TBD (need advice on #1):

1. Whether or not to count requests in the "pending-evaluation" and "pending-fulfillment" status or fleet requests in the "submitted" state (my gut reaction is no, since AWS says they only count "active" and my goal is to match how they calculate limit usage as close as possible).
2. Since this will be an experimental feature, I'm going to add to the docs some details on it, and add a printed warning message with a link to the docs.

[jantman]

Ok, I've merged the experimental PR for this. It will be present in the next release.

[jantman]

I'm reopening this as (1) it's not yet merged, and (2) I've had some private discussion with someone via email trying to figure out exactly how the limits work (his experience with spot instances indicates that the limit is on instances, not requests).

I've opened a thread on the AWS forum asking for clarification about spot limits.

Some research:

• Expired spot requests reduce spot limit - from AWS, "You're correct that the only Spot requests that should count against your Spot limit are open and active requests and user-canceled requests where the instance is left running." They also confirm that there's a lag between limit calculation/application and reality.
• one of many threads referring to per-instance-type limits, that aren't documented anywhere

[jantman]

@EmilVarona @jleader sorry for the delay; this is now released in 0.5.1 and live on PyPI