AWS region locked account causes credentials test to fail

[rpetit3]

Hello!

I have an IAM User that can only be accessed from us-east and us-west regions. Due to this when I click the "Test and Use these Credentials" button I get the following error:

Error: The credentials you have entered could not be validated.

ERROR Details: Authentication with cloud provider failed: CloudBridgeBaseException: An error occurred (UnauthorizedOperation) when calling the DescribeKeyPairs operation: You are not authorized to perform this operation. from exception type: <class 'botocore.exceptions.ClientError'>

After following up with IT this is due to the "Test Credentials" being called against the ap-northeast region instead of one of the us regions.

Is there any way to change where the test is called from or a way to override the test?

Thank you very much!
Robert

[nuwang]

@rpetit3 Hi, this is most likely due to a permissions issue. Please take a look at this comment and let us know whether that solves the problem for you: #198 (comment)

[rpetit3]

It's definitely a permissions issue. Our accounts are region locked to only US zones at the organization level for our university. Any permissions for non-US zones are blocked.

Is there a way to override requests coming from ap-northeast by CloudLaunch and instead make them originate from a US zone? I should mentioned that this is for https://launch.usegalaxy.org/catalog

Thank you very much for your time!

[nuwang]

@rpetit3 I don't think there's any part of the code that's hard-coded to ap-northeast. This is the line in question that does the credential verification: https://github.com/CloudVE/cloudbridge/blob/5820f681c33d74c93ede22e56d8c7a5f7b9f54e7/cloudbridge/base/provider.py#L143
It utilizes the cloud region selected by the user, along with the user's credentials to perform the authentication. The code was always built to not be tied to any specific cloud or region - the default tests for AWS in fact run on us-east-1. So do you know what led IT to believe that this is tied to ap-northeast?

Also, can you separately check whether you have the requisite DescribeKeyPairs permissions on ec2, for the specific cloud region that you selected? Maybe writing a boto3 or cloudbridge script that uses your credentials would be a good way to check. In general, the easiest way to eliminate credentials as a stumbling block is to give yourself Admin permissions on the ec2 account. Hope this helps and let us know if it fixes the issue.

[rpetit3]

I noticed that as well (about us-east-1).

Little more info on my IAM User. It has EC2FullAccess and S3FullAccess on all requests coming from the IP address 149.165.157.211, which is presumed to be that for CloudLaunch. Unfortunately, providing full AdminAccess will not be possible.

IT is using Splunk Cloud on their end to audit University AWS accounts. They reviewed requests for my IAM user, all requests (credential testing) from CloudLaunch (149.165.157.211) are always hitting the ap-northeast-1 for some reason. I have tried changing target zones during setup and get the same error.

I think for now its safe to close this issue, since it doesn't seem like a CloudLaunch thing and more so an AWS thing. Feel free to reopen if you can think of anything else!

[almahmoud]

I looked into this a bit today and I do think the issue is on our side, not AWS, so I'll re-open the issue.

@rpetit3 As a temporary workaround I have created a separate AWS cloud that only contains US regions, so you should be able to launch under your account pointing to the new targets. I added the 4 regions us-[east|west]-[1|2] with default zones for now. You should pick one of the ones called "AWS US":

[almahmoud]

Beyond that, if we want to address it more generally, I'm also detailing the issue here so we can think of a long-term solution:

In our current design, we've tied Credentials to Clouds irrespective of regions. I guess the assumption was that if a set of credentials work on AWS, they should work across AWS regions, at least in terms of being able to authenticate (eg: even if permissions would restrict manipulating resources in that region). There is the notable exception of "AWS China" which is treated essentially as a separate cloud, not by us but more generally (eg: https://medium.com/@david_packman/aws-china-tips-and-differences-9a1aed537a6b).
In this situation, the particular issue is that when you try to save the credentials, CloudLaunch hits the authenticate endpoint to try and save the credentials for that cloud in general (which can also be done from "My Credentials" page as part of one's user page). Since the credentials are only tied to the cloud and not to a particular application (and hence no deployment target), it seems to ignore the region specified in the application form and uses the first region in the list of regions for that cloud to test the credentials: https://github.com/CloudVE/djcloudbridge/blob/62f5d51348e3c7f198a1bc33cf3f5d3442a0db64/djcloudbridge/serializers.py#L875 .

One thing that we need to decide is if we want to put in the effort to change this, or if it's ok workaround to just create a specific separate cloud as needed (which is fairly easy to do, but does pollute the drop-down if it becomes a recurring thing). If we were to try to fix it, one possible solution could be to make the authenticate endpoint accept an optional region, so when credentials are saved from the profile, it defaults to the first for that cloud, but when saving from a launch page, we can inherit the region/target from the form? Then down the line potentially allowing an optional "Advanced" setting in the profile as well to specify the region against which to authenticate when testing?

[rpetit3]

Thank you very much @almahmoud! This works great, I got past the credentials check. I'll test more in the coming days, but I think its safe to assume if the credentials validated I should be able to launch an instance.

I think your proposed solutions would work just fine. I like the idea of validating against what ever the selected target region is. But I think that was more so directed to Nuwan.

[nuwang]

Thanks for tracking that down Alex. I should have looked one level higher, where there's a clear assumption that authentication would be region independent, which in Robert's case, is clearly an incorrect one. Your plan of adding an extra param sounds good.