Customers often want the core services of their OpenShift cluster to be
available only on the internal network and not on the Internet. The API server
is one such service that customers do not want to be accessible over the
Internet. The OpenShift Installer allows creating clusters that have their
services published only on the internal network by setting publish: Internal
in the install-config.yaml.
Since Hive is usually running outside the network where the clusters exists, it requires access to the cluster's API server which mostly translates to having the API reachable over the Internet. There can be some restrictions setup to allow only Hive to access the API but these are usually not acceptable by security focused customers.
AWS provides a feature called AWS Private Link (see doc) that allows accessing private services in customer VPCs from another account using AWS's internal networking and not the Internet. AWS Private Link involves creating a VPC Endpoint Service in customer's account that is backed by one or more internal NLBS and a VPC Endpoint in service provider's account. This allows clients in service provider's VPC to access the NLB backed service in customer's account using the endpoint-endpoint service Private Link. So the internal service is now accessible to the service provider without exposing the service to the Internet.
Using this same architecture, we create a VPC Endpoint Service, VPC Endpoint pair to create an Private Link to cluster's internal NLB for k8s API server, allowing Hive to access the API without forcing the cluster to publish it on the Internet.
The hiveutil awsprivatelink command provides multiple subcommands for managing AWS PrivateLink configuration.
It automates tasks such as managing HiveConfig.spec.awsPrivateLink and setting up
or tearing down networking between associated VPCs and endpoint VPCs.
For detailed information about these subcommands and their usage,
please refer to the hiveutil documentation.
Disclaimer: hiveutil is an internal utility for use by hive developers and
hive itself. It is not supported for general use.
To configure Hive to support Private Link in a specific region,
-
Create VPCs in that region that can be used to create VPC Endpoints.
NOTE: There is a hard limit of 255 VPC Endpoints in a region, therefore you will need multiple VPCs to support more cluster in that region.
-
For each VPC, create subnets in all the supported availability zones of the region.
NOTE: each subnet must have at least 255 usuable IPs because the controller.
For example let's create VPCs in us-east-1 region, that has 6 AZs.
vpc-1 (us-east-1) : 10.0.0.0/20 subnet-11 (us-east-1a): 10.0.0.0/23 subnet-12 (us-east-1b): 10.0.2.0/23 subnet-13 (us-east-1c): 10.0.4.0/23 subnet-12 (us-east-1d): 10.0.8.0/23 subnet-12 (us-east-1e): 10.0.10.0/23 subnet-12 (us-east-1f): 10.0.12.0/23
vpc-2 (us-east-1) : 10.0.16.0/20 subnet-21 (us-east-1a): 10.0.16.0/23 subnet-22 (us-east-1b): 10.0.18.0/23 subnet-23 (us-east-1c): 10.0.20.0/23 subnet-24 (us-east-1d): 10.0.22.0/23 subnet-25 (us-east-1e): 10.0.24.0/23 subnet-26 (us-east-1f): 10.0.28.0/23
-
Make sure all the Hive environments (Hive VPCs) have network reachability to these VPCs created above for VPC Endpoints using peering, transit gateways, etc.
-
Gather a list of VPCs that will need to resolve the DNS setup for Private Link. This should at least include the VPC of the Hive being configured, and can include list of all VPCs where various Hive controllers exists.
-
Update the HiveConfig to enable Private Link for clusters in that region.
## hiveconfig spec: awsPrivateLink: ## this this is list if inventory of VPCs that can be used to create VPC ## endpoints by the controller endpointVPCInventory: - region: us-east-1 vpcID: vpc-1 subnets: - availabilityZone: us-east-1a subnetID: subnet-11 - availabilityZone: us-east-1b subnetID: subnet-12 - availabilityZone: us-east-1c subnetID: subnet-13 - availabilityZone: us-east-1d subnetID: subnet-14 - availabilityZone: us-east-1e subnetID: subnet-15 - availabilityZone: us-east-1f subnetID: subnet-16 - region: us-east-1 vpcID: vpc-2 subnets: - availabilityZone: us-east-1a subnetID: subnet-21 - availabilityZone: us-east-1b subnetID: subnet-22 - availabilityZone: us-east-1c subnetID: subnet-23 - availabilityZone: us-east-1d subnetID: subnet-24 - availabilityZone: us-east-1e subnetID: subnet-25 - availabilityZone: us-east-1f subnetID: subnet-26 ## credentialsSecretRef points to a secret with permissions to create ## resources in account where the inventory of VPCs exist. credentialsSecretRef: name: < hub-account-credentials-secret-name > ## this is a list of VPC where various Hive clusters exists. associatedVPCs: - region: region-hive1 vpcID: vpc-hive1 credentialsSecretRef: name: < credentials that have access to account where Hive1 VPC exists > - region: region-hive2 vpcID: vpc-hive2 credentialsSecretRef: name: < credentials that have access to account where Hive2 VPC exists>
You can include VPC from all the regions where private link is supported in the endpointVPCInventory list. The controller will pick a VPC appropriate for the ClusterDeployment.
Each VPC Endpoint in AWS has a Security Group attached to control access to the endpoint. See the docs for details.
When Hive creates VPC Endpoint, it does not specify any Security Group and therefore the default Security Group of the VPC is attached to the VPC Endpoint. Therefore, the default security group of the VPC where VPC Endpoints are created must have rules to allow traffic from the Hive installer pods.
For example, if Hive is running in hive-vpc(10.1.0.0/16), there must be a rule in default Security Group of VPC where VPC Endpoint is created that allows ingess from 10.1.0.0/16.
Once Hive is configured to support Private Link for AWS clusters, customers can
create ClusterDeployment objects with Private Link by setting the
privateLink.enabled to true in aws platform. This is only supported in
regions where Hive is configured to support Private Link, the validating
webhooks will reject ClusterDeployments that request private link in
unsupported regions.
spec:
platform:
aws:
privateLink:
enabled: trueThe controller provides progress and failure updates using AWSPrivateLinkReady and
AWSPrivateLinkFailed conditions on the ClusterDeployment.
There multiple credentials involved in the configuring AWS Private Link and there are different expectations of required permissions for these credentials.
-
The credentials on ClusterDeployment
The following permissions are required:
ec2:CreateVpcEndpointServiceConfiguration ec2:DescribeVpcEndpointServiceConfigurations ec2:ModifyVpcEndpointServiceConfiguration ec2:DescribeVpcEndpointServicePermissions ec2:ModifyVpcEndpointServicePermissions ec2:DeleteVpcEndpointServiceConfigurations
-
The credentials specified in HiveConfig for endpoint VPCs account
.spec.awsPrivateLink.credentialsSecretRefThe following permissions are required:
ec2:DescribeVpcEndpointServices ec2:DescribeVpcEndpoints ec2:CreateVpcEndpoint ec2:CreateTags ec2:DescribeNetworkInterfaces ec2:DescribeVPCs ec2:DeleteVpcEndpoints route53:CreateHostedZone route53:GetHostedZone route53:ListHostedZonesByVPC route53:AssociateVPCWithHostedZone route53:DisassociateVPCFromHostedZone route53:CreateVPCAssociationAuthorization route53:DeleteVPCAssociationAuthorization route53:ListResourceRecordSets route53:ChangeResourceRecordSets route53:DeleteHostedZone
-
The credentials specified in HiveConfig for associating VPCs to the Private Hosted Zone.
.spec.awsPrivateLink.associatedVPCs[$idx].credentialsSecretRefThe following permissions are required in the account where the VPC exists:
route53:AssociateVPCWithHostedZone route53:DisassociateVPCFromHostedZone ec2:DescribeVPCs
Currently the private link controller creates an ALIAS record in the PHZ created for customer's cluster. This record points the dns resolution of customer cluster's k8s API to the VPC endpoint, therefore allowing hive to transparently access the cluster over the private link.
In some cases the ALIAS record cannot be used. For example, in GovCloud environments the Route53 private hosted zones do not support the ALIAS record type and therefore we have to use A records.
CNAME records would have been a more suitable replacement for ALIAS records, but
CNAME records cannot be created at the apex of a DNS zone like ALIAS records.
The private link architecture uses a DNS zone like api.<clustername>.<clusterdomain>
so that there are no conflicts with authority on DNS resolution. Since CNAME
records are not supported on the apex, we use A records pointing the cluster's
API DNS name to the IP addresses of the VPC endpoint directly. Since these IP
addresses do not change as they are backed by Elastic Network Interfaces in the
corresponding subnets of the VPC endpoint, this DNS record should remain stable.
For simplicity and current desired use-case, the DNS record type can be configured globally for all clusters managed by private link controller.
spec:
awsPrivateLink:
dnsRecordType: (Alias | ARecord)Private link requires some networking setup before Hive controller can create clusters with Private link routing.
Create 2 sets of VPCs in the same region in an AWS account. One VPC will be used for creating an OCP cluster for Hive, and the other will be used as inventory for VPC endpoints.
- Create a Hive VPC using cloudformation template.
aws --region us-east-1 cloudformation create-stack \
--stack-name hive-vpc \
--template-body "file://hack/awsprivatelink/vpc.cf.yaml" \
--parameters ParameterKey=AvailabilityZoneCount,ParameterValue=3 ParameterKey=VpcCidr,ParameterValue=10.0.0.0/16
- Once the cloudformation completes, extract the VPC and subnets from the output.
aws --region us-east-1 cloudformation describe-stacks \
--stack-name hive-vpc --query 'Stacks[].Outputs'
[
[
{
"OutputKey": "PrivateSubnetIds",
"OutputValue": "subnet-0bf2b8faf1c7d200b,subnet-09d53f09b54c38029,subnet-015d46e5f7fb7055d",
"Description": "Subnet IDs of the private subnets."
},
{
"OutputKey": "PublicSubnetIds",
"OutputValue": "subnet-00d5d89ce7b049296,subnet-0253d95c4bcfe7e17,subnet-03a4cfddbf7d9f73a",
"Description": "Subnet IDs of the public subnets."
},
{
"OutputKey": "VpcId",
"OutputValue": "vpc-0e0b3939a23a5e33e",
"Description": "ID of the new VPC."
}
]
]
- Create a Private link VPC using cloudformation template.
aws --region us-east-1 cloudformation create-stack \
--stack-name private-link-vpc \
--template-body "file://hack/awsprivatelink/vpc.cf.yaml" \
--parameters ParameterKey=AvailabilityZoneCount,ParameterValue=3 ParameterKey=VpcCidr,ParameterValue=10.1.0.0/16
- Once the cloudformation completes, extract the VPC and subnets from the output.
aws --region us-east-1 cloudformation describe-stacks \
--stack-name private-link-vpc --query 'Stacks[].Outputs'\
[
[
{
"OutputKey": "PrivateSubnetIds",
"OutputValue": "subnet-0bf2b8faf1c7d200b,subnet-09d53f09b54c38029,subnet-015d46e5f7fb7055d",
"Description": "Subnet IDs of the private subnets."
},
{
"OutputKey": "PublicSubnetIds",
"OutputValue": "subnet-00d5d89ce7b049296,subnet-0253d95c4bcfe7e17,subnet-03a4cfddbf7d9f73a",
"Description": "Subnet IDs of the public subnets."
},
{
"OutputKey": "VpcId",
"OutputValue": "vpc-0e0b3939a23a5e33e",
"Description": "ID of the new VPC."
}
]
]
Follow the steps outlines in OpenShift documentation.
# install-config.yaml
...
platform:
aws:
region: us-east-1
subnets:
- # include all the subnets public and private created by cloudformation stack
defaultMachinePlatform:
zones:
- us-east-1a
- us-east-1b
- us-east-1c
...Create and accept the VPC peering connection using AWS documentation
aws --region us-east-1 ec2 create-vpc-peering-connection \
--vpc-id $HIVE_VPC_ID \
--peer-vpc-id $PRIVATE_LINK_VPC_ID \
--tag-specification "ResourceType=vpc-peering-connection,Tags=[{Key=hive-private-link,Value=owned}]"
aws --region us-east-1 ec2 accept-vpc-peering-connection --vpc-peering-connection-id $VPC_PEERING_CONNECTION_ID
Now update the Route tables using AWS documentation
The cloudformation stack creates one route table for all the public subnets, and one route table for each private subnet.
First update the private route tables for Hive VPC. Update the routes so that 10.1.0.0/16
prefix routes to the VPC peering connection.
Second update the private route tables for Private link VPC. Update the routes so that 10.0.0.0/16
prefix routes to the VPC peering connection.
So we want the workers in the Hive cluster to be able to communicate with the VPC endpoints that will be created in the Private link VPC. Secondly, we also want the workers to receive traffic from the VPC endpoints.
You can use security groups from peered VPCs to allow flow of traffic using AWS documentation
- Find the security group for workers in Hive Cluster
Use the Hive VPC id to list the security groups.
aws --region us-east-1 ec2 describe-security-groups \
--filter Name=vpc-id,Values=$HIVE_VPC_ID Name=tag:Name,Values=$HIVE_CLUSTER_INFRA_ID-worker-sg \
--query "SecurityGroups[*].{GroupName:GroupName,GroupId:GroupId}"
- Find the default security group of the Private link VPC
aws --region us-east-1 ec2 describe-security-groups \
--filters Name=vpc-id,Values=$PRIVATE_LINK_VPC_ID,Name=group-name,Values=default \
--query "SecurityGroups[*].{GroupName:GroupName,GroupId:GroupId}"
- Allow all incoming traffic from Hive cluster's worker security group to VPC endpoints
Create an incoming security group rule in Private link VPC's default security group that allows ALL traffic from the Hive cluster's worker security group.
aws --region us-east-1 ec2 authorize-security-group-ingress \
--group-id $PRIVATE_LINK_VPC_DEFAULT_SG --protocol all --port -1 \
--source-group $HIVE_CLUSTER_WORKER_SG
- Allow all incoming traffic from VPC endpoints to Hive cluster's worker security group
Create an incoming security group rule in Hive cluster's worker security group that allows ALL traffic from the Private link VPC's default security group.
aws --region us-east-1 ec2 authorize-security-group-ingress \
--group-id $HIVE_CLUSTER_WORKER_SG --protocol all --port -1 \
--source-group $PRIVATE_LINK_VPC_DEFAULT_SG
Use the private subnets from Private link VPC as inventory for VPC endpoints by following the steps mentioned above