ci-operator/step-registry/ipi/conf/aws/blackholenetwork/blackhole_vpc…

…_yaml: Add EC2 endpoint

The machine-API currently ignores the proxy configuration, although
future machine-API might grow support for it [1].  That means CI jobs
in the blackhole VPC die on i/o timeouts trying to reach
https://ec2.${region}.amazonaws.com/ while provisioning compute
machines, and the install subsequently dies because we fail to
schedule monitoring, ingress, and other compute-hosted workloads [2].
This commit adds a VPC endpoint to allow EC2 access from inside the
cluster [3].  It's similar to the existing S3 VPC endpoint, but:

* It's an interface type, while S3 needs the older gateway type.  This
  avoids:

    Endpoint type (Gateway) does not match available service types
    ([Interface]). (Service: AmazonEC2; Status Code: 400; Error Code:
    InvalidParameter; Request ID: ...; Proxy: null)

  while creating the stack.

* There are no RouteTableIds, because the interface type does not
  support them.  This avoids:

    Route table IDs are only supported for Gateway type VPC
    Endpoint. (Service: AmazonEC2; Status Code: 400; Error Code:
    InvalidParameter; Request ID: ...; Proxy: null)

  while creating the stack.

* I've created a new security group allowing HTTPS connections to the
  endpoint, because SecurityGroupIds is required for interface
  endpoints [3].  I've also placed the network interfaces in the
  public subnets, because SubnetIds is requried for interface
  endpoints [3].

* I've set PrivateDnsEnabled [3] so the machine-API operator doesn't
  have to do anything special to get DNS routing it towards the
  endpoint interfaces.

Rolled out to the CI account following 9b39dd2 (Creating private
subnets without direct external internet access and updating proxy e2e
to use this instead, 2020-07-20, #10355):

  for REGION in us-east-1 us-east-2 us-west-1 us-west-2
  do
    COUNT=3
    if test us-west-1 = "${REGION}"
    then
      COUNT=2
    fi
    for INDEX in 1
    do
      NAME="do-not-delete-shared-vpc-blackhole-${INDEX}"
      aws --region "${REGION}" cloudformation update-stack --stack-name "${NAME}" --template-body "$(cat ci-operator/step-registry/ipi/conf/aws/blackholenetwork/blackhole_vpc_yaml.md)" --parameters "ParameterKey=AvailabilityZoneCount,ParameterValue=${COUNT}" >/dev/null
      aws --region "${REGION}" cloudformation wait stack-update-complete --stack-name "${NAME}"
      SUBNETS="$(aws --region "${REGION}" cloudformation describe-stacks --stack-name "${NAME}" | jq -c '[.Stacks[].Outputs[] | select(.OutputKey | endswith("SubnetIds")).OutputValue | split(",")[]]' | sed "s/\"/'/g")"
      echo "${REGION}_$((INDEX - 1))) subnets=\"${SUBNETS}\";;"
    done
  done

We could also have deleted the previous stacks, used 'create-stack'
instead of 'update-stack', and used 'stack-create-complete' instead of
'stack-update-complete'.

Unsurprisingly, since we were not updating the subnets themselves, the
output has not changed:

  us-east-1_0) subnets="['subnet-0a7491aa76f9b88d7','subnet-0f0b2dcccdcbc7c1d','subnet-0680badf68cbf198c','subnet-02b25dd65f806e41b','subnet-010235a3bff34cf6f','subnet-085c78d8c562b5a51']";;
  us-east-2_0) subnets="['subnet-0ea117d9499ef624f','subnet-00adc83d4719d4176','subnet-0b9399990fa424d7f','subnet-060d997b25f5bb922','subnet-015f4e65b0ef1b0e1','subnet-02296b47817923bfb']";;
  us-west-1_0) subnets="['subnet-0d003f08a541855a2','subnet-04007c47f50891b1d','subnet-02cdb70a3a4beb754','subnet-0d813eca318034290']";;
  us-west-2_0) subnets="['subnet-05d8f8ae35e720611','subnet-0f3f254b13d40e352','subnet-0e23da17ea081d614','subnet-0f380906f83c55df7','subnet-0a2c5167d94c1a5f8','subnet-01375df3b11699b77']";;

so no need to update ipi-conf-aws-blackholenetwork-commands.sh.

I generated the reaper keep-list following 1b21187 (ci-operator:
Fresh AWS shared subnets for us-east-2, etc., 2020-01-30, #6949):

  for REGION in us-east-1 us-east-2 us-west-1 us-west-2
  do
    for INDEX in 1
    do
      NAME="do-not-delete-shared-vpc-blackhole-${INDEX}"
      aws --region "${REGION}" resourcegroupstaggingapi get-resources --tag-filters "Key=aws:cloudformation:stack-name,Values=${NAME}" --query 'ResourceTagMappingList[].ResourceARN[]' | jq -r ".[] | . + \"  # CI exclusion per DPP-5789, ${REGION} ${NAME}\""
    done
  done | sort

and passed that along to the Developer Productivity Platform (DPP)
folks so they can update their reaper config.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1769223
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=1875773
[3]: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpcendpoint.html

ci-operator/step-registry/ipi/conf/aws/blackholenetwork/blackhole_vpc_yaml.md

@@ -173,6 +173,42 @@ Resources:
  Properties:
  SubnetId: !Ref PrivateSubnet3
  RouteTableId: !Ref PrivateRouteTable3
+ EC2EndpointSecurityGroup:
+ Type: AWS::EC2::SecurityGroup
+ Properties:
+ GroupDescription: EC2 Endpoint Security Group
+ SecurityGroupIngress:
+ - IpProtocol: tcp
+ FromPort: 443
+ ToPort: 443
+ CidrIp: !Ref VpcCidr
+ VpcId: !Ref VPC
+ EC2Endpoint:
+ Type: AWS::EC2::VPCEndpoint
+ Properties:
+ PolicyDocument:
+ Version: 2012-10-17
+ Statement:
+ - Effect: Allow
+ Principal: '*'
+ Action:
+ - '*'
+ Resource:
+ - '*'
+ PrivateDnsEnabled: "true"
+ SecurityGroupIds:
+ - !Ref EC2EndpointSecurityGroup
+ ServiceName: !Join
+ - ''
+ - - com.amazonaws.
+ - !Ref 'AWS::Region'
+ - .ec2
+ SubnetIds:
+ - !Ref PublicSubnet
+ - !If [DoAz2, !Ref PublicSubnet2, !Ref "AWS::NoValue"]
+ - !If [DoAz3, !Ref PublicSubnet3, !Ref "AWS::NoValue"]
+ VpcEndpointType: Interface
+ VpcId: !Ref VPC
  S3Endpoint:
  Type: AWS::EC2::VPCEndpoint
  Properties:
@@ -214,4 +250,4 @@ Outputs:
  !Join [
  ",",
  [!Ref PrivateSubnet, !If [DoAz2, !Ref PrivateSubnet2, !Ref "AWS::NoValue"], !If [DoAz3, !Ref PrivateSubnet3, !Ref "AWS::NoValue"]]
- ]
+ ]