EC2 RequestSpotFleet ValidFrom / ValidUntil timestamp issue

[jantman]

Passing a DateTime as the ValidFrom or ValidUntil argument to ec2 client request_spot_instances() works fine. However, passing a similar DateTime object to request_spot_fleet() results in an invalid parameter value from EC2: botocore.exceptions.ClientError: An error occurred (InvalidSpotFleetRequestConfig) when calling the RequestSpotFleet operation: Parameter: SpotFleetRequestConfig.ValidFrom is invalid.

The problem appears to be that request_spot_fleet() sends the parameter as a string in YYYY-MM-DDTHH:MM:SS.xxxxxxZ format (debug-level logging on a call to request_spot_fleet() where ValidFrom and ValidUntil are DateTimes shows the request as including u'SpotFleetRequestConfig.ValidFrom': '2016-07-29T19:32:29.648833Z', u'SpotFleetRequestConfig.ValidUntil': '2016-07-29T19:32:30.648833Z').

If I change the call to have arguments of ValidFrom=valid_from.strftime('%Y-%m-%dT%H:%M:%SZ'), ValidUntil=valid_to.strftime('%Y-%m-%dT%H:%M:%SZ') it then works correctly.

I don't know if this is an issue with boto3 or on the AWS side, but it appears that there needs to be some additional manipulation of the timestamp string sent to AWS.

[kyleknap]

Interesting, based on our model, the ValidUntil and ValidFrom parameters are modeled as datetimes (i.e. the DateTime shape translates to timestamps whichs translates to python datetimes). Can you share a snippet of code using a client or resource that works and does not work?

[jantman]

@kyleknap Sure...

proof of concept code (note this is running in us-west-2):

#!/usr/bin/env python

import boto3
from botocore.exceptions import ClientError
from pprint import pformat
from datetime import datetime, timedelta
from tzlocal import get_localzone
from pytz import utc
import uuid

target_capacity = 4
role = 'arn:aws:iam::860309399526:role/spotfleet'
i_type = 'm3.medium'
az = 'us-west-2b'
bid_price = 0.0173

conn = boto3.client('ec2')

now_utc = datetime.now(get_localzone()).astimezone(utc)
valid_from = now_utc + timedelta(days=14)
valid_to = valid_from + timedelta(seconds=1)

# kwargs
args = {
    'SpotFleetRequestConfig': {
        'SpotPrice': '%s' % bid_price,
        'ClientToken': str(uuid.uuid4()),
        'TargetCapacity': target_capacity,
        'ValidFrom': valid_from,
        'ValidUntil': valid_to,
        'TerminateInstancesWithExpiration': True,
        'ExcessCapacityTerminationPolicy': 'default',
        'AllocationStrategy': 'lowestPrice',
        'IamFleetRole': role,
        'LaunchSpecifications': [
            {
                'ImageId': 'ami-d2c924b2',  # Centos 7 HVM, us-west-2
                'InstanceType': i_type,
                'Placement': {
                    'AvailabilityZone': az
                }
            }
        ]
    }
}


print("#### RequestSpotFleet with DateTime\n")
try:
    print("Calling request_spot_fleet() with kwargs: %s" % pformat(args))
    res = conn.request_spot_fleet(**args)
    print('Result: %s', pformat(res))
except ClientError as ex:
    print(ex)


print("\n### RequestSpotFleet with strftime\n")
args['SpotFleetRequestConfig']['ValidFrom'] = args[
    'SpotFleetRequestConfig']['ValidFrom'].strftime('%Y-%m-%dT%H:%M:%SZ')
args['SpotFleetRequestConfig']['ValidUntil'] = args[
    'SpotFleetRequestConfig']['ValidUntil'].strftime('%Y-%m-%dT%H:%M:%SZ')

try:
    print("Calling request_spot_fleet() with kwargs: %s" % pformat(args))
    res = conn.request_spot_fleet(**args)
    print('Result: %s' % pformat(res))
except ClientError as ex:
    print(ex)

Output:

$ python ~/tmp/spot_test.py 
#### RequestSpotFleet with DateTime

Calling request_spot_fleet() with kwargs: {'SpotFleetRequestConfig': {'AllocationStrategy': 'lowestPrice',
                            'ClientToken': '164d3980-3953-4af5-82a2-feab6455e506',
                            'ExcessCapacityTerminationPolicy': 'default',
                            'IamFleetRole': 'arn:aws:iam::860309399526:role/spotfleet',
                            'LaunchSpecifications': [{'ImageId': 'ami-d2c924b2',
                                                      'InstanceType': 'm3.medium',
                                                      'Placement': {'AvailabilityZone': 'us-west-2b'}}],
                            'SpotPrice': '0.0173',
                            'TargetCapacity': 4,
                            'TerminateInstancesWithExpiration': True,
                            'ValidFrom': datetime.datetime(2016, 7, 30, 1, 48, 16, 484001, tzinfo=<UTC>),
                            'ValidUntil': datetime.datetime(2016, 7, 30, 1, 48, 17, 484001, tzinfo=<UTC>)}}
An error occurred (InvalidSpotFleetRequestConfig) when calling the RequestSpotFleet operation: Parameter: SpotFleetRequestConfig.ValidFrom is invalid.

### RequestSpotFleet with strftime

Calling request_spot_fleet() with kwargs: {'SpotFleetRequestConfig': {'AllocationStrategy': 'lowestPrice',
                            'ClientToken': '164d3980-3953-4af5-82a2-feab6455e506',
                            'ExcessCapacityTerminationPolicy': 'default',
                            'IamFleetRole': 'arn:aws:iam::860309399526:role/spotfleet',
                            'LaunchSpecifications': [{'ImageId': 'ami-d2c924b2',
                                                      'InstanceType': 'm3.medium',
                                                      'Placement': {'AvailabilityZone': 'us-west-2b'}}],
                            'SpotPrice': '0.0173',
                            'TargetCapacity': 4,
                            'TerminateInstancesWithExpiration': True,
                            'ValidFrom': '2016-07-30T01:48:16Z',
                            'ValidUntil': '2016-07-30T01:48:17Z'}}
Result: {'ResponseMetadata': {'HTTPHeaders': {'content-type': 'text/xml;charset=UTF-8',
                                      'date': 'Sat, 16 Jul 2016 01:48:16 GMT',
                                      'server': 'AmazonEC2',
                                      'transfer-encoding': 'chunked',
                                      'vary': 'Accept-Encoding'},
                      'HTTPStatusCode': 200,
                      'RequestId': 'a4845e0a-7ff9-4443-994e-70b392d7071f'},
 u'SpotFleetRequestId': 'sfr-3fe219cd-ddab-4202-883d-3e0441f2e738'}

[patrickpierson]

Ran into the same issue. you need to strip off the microseconds from your valid_until time.

now_utc = datetime.datetime.utcnow()
now_utc = now_utc.replace(microsecond=0)
valid_until = now_utc + datetime.timedelta(hours=1)

[JordonPhillips]

It looks like @patrickpierson may have solved the issue. If you still need help I would recommend making use of our community resources for help with api usage.