Support launching clusters into private VPCs

[nchammas]

Some users work in environments where they want to or have to launch clusters into VPCs with no public subnets.

This seems to be a fairly common use-case, so I think we should support it if it does not add too much complexity.

A design goal for this feature should be to automatically do the appropriate setup whether the subnet we are launching into is public or private. That means, if possible, the user shouldn't have to specify anything.

I think this is possible because we know what VPC we're launching into (either the user's default VPC, or an explicitly specified one), and we know what subnet we're launching into (either the VPC's default subnet, or an explicitly specified one). Flintrock should be able to query AWS for information about the subnet and figure out whether to use public or private addresses automatically.

[MuksSharma]

Thanks @nchammas. This use case is becoming more common in enterprise setting. Any thoughts on when this feature might become available?

[nchammas]

As with the typical open source project, there are no promises on when a feature might be delivered. The things that usually get worked on are critical bug fixes, fun stuff, and everything else, in that order. 😄

My current intent is to work next on adding support for resizing existing clusters and improving launch progress reporting. If I get some time at work to devote to Flintrock (e.g. for this issue), I may work on it after I get those 2 features out.

[rmessner]

It is exactly our case ( and we don't even have hostname resolution, and that would need some hack for spark and hadoop ), I will work out something and submit it to you @nchammas.

My use case will be :

• Downloading spark from a custom location ( thanks to Add a new option for an alternate mirror for spark binaries #104 )
• Use private DNS names instead of public ones
• Define several origins ( multiple CIDR addresses to access the cluster for the security groups )

[rmessner]

In case of the private VPC, I'm struggling with the ssh access when installing and configuring, because i'm outside the security group and it take my public IP to access an private IP so doesn't work.

I'm thinking about 2 solutions :

• Opening a security breach for the install and configuration step
• Bind the access_origins to the VPC ( means my private IP is within the range of the origins )

What do you think about it @nchammas ?

[nchammas]

Hey @rmessner, I updated the description of the issue and commented on your PRs as well.

[eshioji]

For people who want to have a quick hacky solution to this. One can create a machine in the VPC and launch a cluster from that machine using private DNS/IP.

However, one needs to do the following:

• Hack flintrock (https://github.com/eshioji/flintrock/pull/1/files)
• Assign additional security group that allows access to necessary ports (the one flintrock assigns didn't work for my VPC setting)
• Make it so that the machines can resolve their own hostname (the default AMI gets hostnames like ip-xx-xx-xx-xx and Spark fails to launch because it can't resolve it).

For the last problem, I did the following:
Create a script with the following content and let it run at startup using rc.local. Not the nicest way but it works.

OWN_IP=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`
HOSTNME=`hostname`
echo "$OWN_IP $HOSTNME" | sudo tee --append /etc/hosts

Finally thank you @nchammas , I wanted to use a custom AMI and this project was a life-saver!

[nchammas]

Related PRs from others who have implemented their own fixes for this:

• Implementation to use private subnet chaordic/flintrock#4

[pferrel]

is something like the chaordic fork implemented here? If so are their instructions? I'm trying to run Spark behind the VPC with not outside traffic.