Initial Implementation #1

joshmyers · 2019-01-16T20:06:45Z

what

This PR is a standalone Terraform module to create CodeFresh Enterprise backing services:

RDS Aurora Postgres Cluster
Elasticache Redis
EFS
AmazonMQ
S3 buckets with an associated IAM user

Note that this is WIP until cloudposse/terraform-aws-s3-bucket#1 and https://github.com/cloudposse/terraform-aws-mq-broker have been addressed.

why

So we can run CodeFresh Enterprise in Kops clusters.

testing

Given the below variable tfvars:

vpc_id = "vpc-0b7622f0c6f33fc47"
subnet_ids = ["subnet-028e95f855e889902", "subnet-03a7ffd7204e0dca6", "subnet-06ac8a3ce7e6e2d3d"]
node_security_groups = ["sg-0fec58dab9e181609"]
mq_subnet_ids = ["subnet-028e95f855e889902", "subnet-03a7ffd7204e0dca6"]

Note that when ActiveMQ is running in ACTIVE_STANDBY_MULTI_AZ mode, it only takes 2 subnets and will error if more are given. We could just use subnet_ids and pick 2 of the list.

Running the thing:

aurora_postgres_cluster_name = cpco-testing-cpco-testing-codefresh
aurora_postgres_database_name = touched_anemone
aurora_postgres_master_hostname = master.cpco-testing-codefresh.testing.cloudposse.co
aurora_postgres_master_username = qNMVVNru
aurora_postgres_replicas_hostname = replicas.cpco-testing-codefresh.testing.cloudposse.co
efs_arn = arn:aws:elasticfilesystem:us-west-2:126450723953:file-system/fs-bbeb8313
efs_dns_name = fs-bbeb8313.efs.us-west-2.amazonaws.com
efs_host = cpco-testing-codefresh.testing.cloudposse.co
efs_id = fs-bbeb8313
efs_mount_target_dns_names = [
    fs-bbeb8313.efs.us-west-2.amazonaws.com,
    fs-bbeb8313.efs.us-west-2.amazonaws.com,
    fs-bbeb8313.efs.us-west-2.amazonaws.com
]
efs_mount_target_ids = [
    fsmt-8fbabf26,
    fsmt-8ebabf27,
    fsmt-8cbabf25
]
efs_mount_target_ips = [
    10.0.68.239,
    10.0.55.9,
    10.0.30.30
]
efs_network_interface_ids = [
    eni-083f8cdc53854ed34,
    eni-0e152056ad84d4fd3,
    eni-0f57a26ccbbbae6bc
]
elasticache_redis_host = redis.testing.cloudposse.co
elasticache_redis_id = cpco-testing-redis
elasticache_redis_security_group_id = sg-09a561d7a705f8d20
mq_admin_username = PrjuBTXW
mq_application_username = tcSwuwwZ
mq_broker_arn = arn:aws:mq:us-west-2:126450723953:broker:cpco-testing-codefresh-mq:b-ffc4e4fd-8d50-4030-950f-011e2269fa46
mq_broker_id = b-ffc4e4fd-8d50-4030-950f-011e2269fa46
mq_primary_ampq_ssl_endpoint = amqp+ssl://b-ffc4e4fd-8d50-4030-950f-011e2269fa46-1.mq.us-west-2.amazonaws.com:5671
mq_primary_console_url = https://b-ffc4e4fd-8d50-4030-950f-011e2269fa46-1.mq.us-west-2.amazonaws.com:8162
mq_primary_ip_address = 10.0.43.209
mq_secondary_ampq_ssl_endpoint = amqp+ssl://b-ffc4e4fd-8d50-4030-950f-011e2269fa46-2.mq.us-west-2.amazonaws.com:5671
mq_secondary_console_url = https://b-ffc4e4fd-8d50-4030-950f-011e2269fa46-2.mq.us-west-2.amazonaws.com:8162
mq_secondary_ip_address = 10.0.66.174

Note that values are written into SSM as so:

/terraform-aws-codefresh-backing-services/aurora_postgres_database_name
/terraform-aws-codefresh-backing-services/aurora_postgres_master_hostname
/terraform-aws-codefresh-backing-services/aurora_postgres_master_username
/terraform-aws-codefresh-backing-services/aurora_postgres_replicas_hostname
/terraform-aws-codefresh-backing-services/mq_admin_username
/terraform-aws-codefresh-backing-services/mq_application_password
/terraform-aws-codefresh-backing-services/redis_auth_token
/terraform-aws-codefresh-backing-services/aurora_postgres_cluster_name
/terraform-aws-codefresh-backing-services/aurora_postgres_master_password
/terraform-aws-codefresh-backing-services/mq_admin_password
/terraform-aws-codefresh-backing-services/mq_application_username

This will hold our global config to be used in other resources

This commit has largely been lifted from our backing services module for Postgres Aurora. It creates a Postgres Aurora cluster and writes the outputs of that to SSM. If no master username / password is given, these are generated as a random strings and saved to SSM using SecureString where necessary. If not postgres_name is given, it defaults to using using the terraform-null-label outputted id.

This is part of terraform-root-modules backing service but I think we will likely want to create the VPC outside of this module and pass it in. I’ve left in for now but we can take out later.

This is used to pull information from our kops cluser by doing a load of data lookups based on some tags. We can then pass things like Kops node security groups around to be used by other modules.

This commit adds ActiveMQ broker and config for CodeFresh Enterprise. An admin user is created and credentials stored encrypted in SSM. A DNS hostname is created for the ActiveMQ endpoints. Note that unlike the other backing modules in here, AmazonMQ resources are not currently their own module. There are only a handful of resources for this AmazonMQ stuff but we can pull it out into a module if we so wish. The default ActiveMQ config [1] has been added. This is optional in the `aws_mq_broker` but due to Terraform not supporting conditional blocks beyond a basic count, it is a pain to conditionally add this. The schema can be found [2] [1] http://svn.apache.org/repos/asf/activemq/trunk/assembly/src/release/conf/activemq.xml [2] https://s3-us-west-2.amazonaws.com/amazon-mq-docs/XML/amazon-mq-active-mq-5.15.0.xsd

We will be injecting these resources into an existing VPC

3.2.10 doesn’t support encryption, see below: ``` Error creating Elasticache Replication Group: InvalidParameterCombination: Encryption features are not supported for engine version 3.2.10. Please use engine version 3.2.6 ```

aurora-postgres.tf

aknysh · 2019-01-17T20:51:41Z

aurora-postgres.tf

@@ -0,0 +1,188 @@
+variable "postgres_name" {


use var.name

@osterman should we move all variables from all files into variables.tf for consistency?
since this is a low-level module and not part of terraform-root-modules (where we include variables into the main files)

and also all outputs to outputs.tf

mq.tf

aknysh · 2019-01-18T18:49:14Z

aurora-postgres.tf

+variable "postgres_cluster_family" {
+  type        = "string"
+  default     = "aurora-postgresql9.6"
+  description = "Postgres cluster DB family. Currently supported values are aurora-postgresql9.6 / aurora-postgresql10"


Suggested change

description = "Postgres cluster DB family. Currently supported values are aurora-postgresql9.6 / aurora-postgresql10"

description = "Postgres cluster DB family. Currently supported values are `aurora-postgresql9.6` and `aurora-postgresql10`"

aurora-postgres.tf

efs.tf

elasticache-redis.tf

mq.t2.micro is likely not what you want in production.

It is in use in all the submodule we call and we delegate down to them for naming.

To ensure we can use enabled flags on it

Since we have the initial PR merged and have cut a release

Bump aws-mq-broker to 0.3.0, which does not feature enabled flags, so remove them. In its current incarnation the terraform-aws-mq-broker does not support the enabled variable allowing for boolean creation of resources in the module, see [1] for more context. [1] cloudposse/terraform-aws-mq-broker#4

This commit adds variables to enable overriding what VPC and subnet that EFS runs in. If you don’t provide them, it default back to the `var.vpc_id` and `var.subnet_ids` values. During testing of this module, we found something interesting. We were deploying EFS to the backing services VPC, which is a different VPC to the k8s cluster. Our pods were unable to resolve the DNS endpoint of the EFS cluster, despite there being VPC peering between the two, with DNS lookups between them enabled. AWS documentation [1] states that in this scenario, you have a few options: 1) Use the EFS IP address directly (no thanks) 2) Create a DNS entry in Route53 CNAME’ing to the EFS DNS endpoint in a private hosted zone The underlying EFS module does already create a Route53 DNS entry CNAME to the EFS DNS endpoint, but it isn’t in a private zone. The pod could not resolve the Route53 DNS. Deploying EFS into the _same_ VPC as the k8s nodes worked a treat and finally the pods were able to resolve and mount EFS volumes. [1] https://docs.aws.amazon.com/efs/latest/ug/manage-fs-access-vpc-peering.html

efs.tf

goruha · 2019-01-25T08:03:48Z

Remove efs
efs is not really service for codefresh, this is just extention for k8s.

goruha · 2019-01-25T08:04:07Z

Remove mq
mq is not compatible with codefresh - I tested yesterday.

goruha · 2019-01-25T08:04:24Z

Fix postgreql
postgreql does not open security group ports from k8s security group requests.

To be honest I'm not sure I set tfvars right.
Probably when terraform-aws-codefresh-backing-services in terraform-root-modules would be fine.

examples/complete/main.tf

CodeFresh Enterprise is not compatible with AmazonMQ on the protocol level and EFS will be moved into a general Kops backing service.

These should be in the caller module, not this generic module.

README.yaml

goruha · 2019-01-25T13:43:05Z

README.yaml

  - AWS Elasticache Redis
  - AWS AmazonMQ (ActiveMQ)
+  - AWS S3 bucket with associated user and permissions
+  - AWS EFS


Suggested change

- AWS EFS

examples/complete/main.tf

goruha · 2019-01-25T13:47:45Z

elasticache-redis.tf

+
+variable "redis_cluster_enabled" {
+  type        = "string"
+  default     = ""


Suggested change

default = ""

default = "true"

@goruha This is not needed and will fall back on enabled for the whole module if we don't override a specific service enabled flag like this. It was done like this so we could turn enable/disable individual services if we needed, rather than the entire module enable/disabled

goruha · 2019-01-25T13:48:06Z

aurora-postgres.tf

+
+variable "postgres_cluster_enabled" {
+  type        = "string"
+  default     = ""


Suggested change

default = ""

default = "true"

@goruha This is not needed and will fall back on enabled for the whole module if we don't override a specific service enabled flag like this. It was done like this so we could turn enable/disable individual services if we needed, rather than the entire module enable/disabled

For terraform-root-modules I 💯 agree. For this module, I am on the fence.

elasticache-redis.tf

osterman · 2019-01-25T16:10:32Z

aurora-postgres.tf

+
+variable "postgres_cluster_enabled" {
+  type        = "string"
+  default     = ""


For terraform-root-modules I 💯 agree. For this module, I am on the fence.

Resolved

joshmyers added 2 commits January 16, 2019 19:58

Add basic main.tf

a8ebeb5

This will hold our global config to be used in other resources

Add default global variables

6990ce1

osterman requested a review from aknysh January 17, 2019 05:05

joshmyers force-pushed the initial_implementation branch 2 times, most recently from 7a09b5d to 7fc4711 Compare January 17, 2019 16:39

joshmyers added 13 commits January 17, 2019 16:41

Add VPC backing-services

15dc1fc

This is part of terraform-root-modules backing service but I think we will likely want to create the VPC outside of this module and pass it in. I’ve left in for now but we can take out later.

Add Postgres read replica

e16471d

Add Elasticache Redis backing service

43d7c2e

Add Kops metadata module

6fc9306

This is used to pull information from our kops cluser by doing a load of data lookups based on some tags. We can then pass things like Kops node security groups around to be used by other modules.

Add docs

815a2af

Remove RDS Aurora Postgres replica

8217a04

Remove VPC and subnet modules

cc37e9d

We will be injecting these resources into an existing VPC

Remove need for kops metadata

703f468

Move AmazonMQ into own module

8ea0f6f

Add EFS

90fd5c4

Drop Redis Elasticache version to 3.2.6

e37c248

3.2.10 doesn’t support encryption, see below: ``` Error creating Elasticache Replication Group: InvalidParameterCombination: Encryption features are not supported for engine version 3.2.10. Please use engine version 3.2.6 ```

joshmyers force-pushed the initial_implementation branch from 7fc4711 to abaee64 Compare January 17, 2019 16:41

aknysh reviewed Jan 17, 2019

View reviewed changes

aurora-postgres.tf Outdated Show resolved Hide resolved

aknysh reviewed Jan 17, 2019

View reviewed changes

aurora-postgres.tf Outdated Show resolved Hide resolved

aknysh reviewed Jan 17, 2019

View reviewed changes