This module helps manage interoperability between the Kubernetes & AWS abstractions. The module does not create any Kubernetes Deployments, Pods, Services (these should be managed outside this module for now). The module does support adding the following AWS resources to integrate with your Kubernetes DeploymentSet.
This module supports Terraform 0.13.0 (with backwards compatibility for Terraform 0.12.31)
Datastores store data in which the owning service is the only means of access. This module creates the associated Execution Role with the required policy to access the s3 & dynamodb datastores (RDS access is managed by user/password credentials). The module can accept multiple different datastore options but can only create one of each datastore type.
This module is dependent on the Terraform-aws-datastorage-module.
RDS datastores support both Postgres and MySQL engines and provide many configuration options (see below). The module requires a security group id that defines the access policies within the VPC. Orchestration of the security groups between the k8s worker node and provided security group is not supported by the module.
This module creates a Kubernetes secret to help provide a more secure means of integration. The secret has the following design.
The secret name is generated by appending -db to the app_name Terraform variable, i.e., whereapp_name has a value of foo-service, foo-service-db will be the name of the secret created. See the example
The secret has 5 key-value pairs.
- username - user of the RDS instance
- password - users password
- dbname - the database name
- endpoint - the endpoint of the RDS instance (includes port number)
- url - the connection-url to the RDS database
- url_encoded - as per the url with the password being url-encoded
Creates an S3 bucket, Execution Role, and binds an access policy to the role. At this stage, the module does not support adding a custom resource policy, nor does it configure any explicit deny rules for the bucket. It is helpful to remember one should prevent multiple service access to a datastore s3 bucket. The bucket region and name should be passed as env variables in the manifest file or Helm chart.
Creates a dynamodb table, an Execution Role, and access policy and provides many configuration options (see below). The table region and name should be passed as env variables in the manifest file or Helm chart.
The module supports adding a custom policy that allows the service to integrate with additional AWS resources. The policy, if provided, will be attached to the execution role in addition to any other datastore resources. If the module determines no policy is required, it will not create the execution role.
By default, container workloads run under the IAM role context of the worker node. The eks_trusted_assume_role_arn variable defines the IAM role used by the worker nodes, which it uses to create the trust policy for the execution role. For a Kubernetes service to assume a role, an integration solution like Kube2Iam or OIDC integration must be installed and configured on the cluster. The role name or ARN (dependent on the integration) then needs to be configured as an annotation (Kube2am) or in the RBAC resource (OIDC) implementation. The k8s_deployment_execution_role_name_override variable allows the name to be configured as required.
The following examples have been provided
The examples do require access to an eks-cluster via the eks_cluster_name variable. The examples do not provision any EKS cluster for running the examples. the also require the eks_trusted_assume_role_arn variable to allow the worker node to assume the execution role.
Branch 0.11 is compatible with Terraform 0.11 but is no longer supported or maintained. The branch will be deleted in the near future.
| Name | Version |
|---|---|
| terraform | >= 0.12.31 |
| aws | >= 3.38.0 |
| kubernetes | >= 2.0 |
| Name | Version |
|---|---|
| aws | >= 3.38.0 |
| kubernetes | >= 2.0 |
| Name | Source | Version |
|---|---|---|
| service_datastore | git::git@github.com:hyprnz/terraform-aws-data-storage-module?ref=3.0.0 |
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| app_name | The name of the Kubernetes service | string |
n/a | yes |
| eks_cluster_name | Name of EKS cluster | string |
n/a | yes |
| backup_retention_period | The backup retention period in days | number |
7 |
no |
| create_dynamodb_table | Whether or not to enable DynamoDB resources | bool |
false |
no |
| create_rds_instance | Controls if an RDS instance should be provisioned. Will take precedence if this and use_rds_snapshot are both true. |
bool |
false |
no |
| create_s3_bucket | Controls if an S3 bucket should be provisioned | bool |
false |
no |
| datastore_tags | Additional tags to add to all datastore resources | map(string) |
{} |
no |
| dynamodb_attributes | Additional DynamoDB attributes in the form of a list of mapped values | list |
[] |
no |
| dynamodb_autoscale_max_read_capacity | DynamoDB autoscaling max read capacity | number |
20 |
no |
| dynamodb_autoscale_max_write_capacity | DynamoDB autoscaling max write capacity | number |
20 |
no |
| dynamodb_autoscale_min_read_capacity | DynamoDB autoscaling min read capacity | number |
5 |
no |
| dynamodb_autoscale_min_write_capacity | DynamoDB autoscaling min write capacity | number |
5 |
no |
| dynamodb_autoscale_read_target | The target value (in %) for DynamoDB read autoscaling | number |
50 |
no |
| dynamodb_autoscale_write_target | The target value (in %) for DynamoDB write autoscaling | number |
50 |
no |
| dynamodb_billing_mode | DynamoDB Billing mode. Can be PROVISIONED or PAY_PER_REQUEST | string |
"PROVISIONED" |
no |
| dynamodb_enable_autoscaler | Whether or not to enable DynamoDB autoscaling | bool |
false |
no |
| dynamodb_enable_encryption | Enable DynamoDB server-side encryption | bool |
true |
no |
| dynamodb_enable_point_in_time_recovery | Enable DynamoDB point in time recovery | bool |
true |
no |
| dynamodb_enable_streams | Enable DynamoDB streams | bool |
false |
no |
| dynamodb_global_secondary_index_map | Additional global secondary indexes in the form of a list of mapped values | any |
[] |
no |
| dynamodb_hash_key | DynamoDB table Hash Key | string |
"" |
no |
| dynamodb_hash_key_type | Hash Key type, which must be a scalar type: S, N, or B for (S)tring, (N)umber or (B)inary data |
string |
"S" |
no |
| dynamodb_local_secondary_index_map | Additional local secondary indexes in the form of a list of mapped values | list |
[] |
no |
| dynamodb_range_key | DynamoDB table Range Key | string |
"" |
no |
| dynamodb_range_key_type | Range Key type, which must be a scalar type: S, N or B for (S)tring, (N)umber or (B)inary data |
string |
"S" |
no |
| dynamodb_stream_view_type | When an item in a table is modified, what information is written to the stream | string |
"" |
no |
| dynamodb_table_name | DynamoDB table name. Must be supplied if creating a dynamodb table | string |
"" |
no |
| dynamodb_tags | Additional tags (e.g map(BusinessUnit,XYX) |
map |
{} |
no |
| dynamodb_ttl_attribute | DynamoDB table ttl attribute | string |
"Expires" |
no |
| dynamodb_ttl_enabled | Whether ttl is enabled or disabled | bool |
true |
no |
| eks_trusted_assume_role_arn | The arn of the Kubernetes worker IAM role that is configured to allow assuming the service execution role. | string |
"" |
no |
| enable_datastore_module | Enables the data store module that will provision data storage resources | bool |
true |
no |
| k8s_custom_execution_policy_description | Allows to override the custom k8s deployment policy's description | string |
"The custom policy for the k8s deployment execution role" |
no |
| k8s_custom_execution_policy_document_json | A valid policy json string that defines additional actions required by the execution role of the k8s deployment | string |
"" |
no |
| k8s_deployment_execution_role_name_override | Allows to override the default Execution Role name of k8s-{var.app_name}-ExecutionRole. |
string |
"" |
no |
| namespace | The namespace of the Kubernetes resources | string |
"default" |
no |
| rds_allocated_storage | Amount of storage allocated to RDS instance | number |
100 |
no |
| rds_apply_immediately | Specifies whether any database modifications are applied immediately, or during the next maintenance window. Defaults to false. |
bool |
false |
no |
| rds_auto_minor_version_upgrade | Indicates that minor engine upgrades will be applied automatically to the DB instance during the maintenance window. Defaults to true. |
bool |
true |
no |
| rds_backup_window | The daily time range (in UTC) during which automated backups are created if they are enabled. Example: '09:46-10:16'. Must not overlap with maintenance_window | string |
"16:19-16:49" |
no |
| rds_database_name | The name of the database. Can only contain alphanumeric characters | string |
"" |
no |
| rds_enable_deletion_protection | If the DB instance should have deletion protection enabled. The database can't be deleted when this value is set to true. The default is false. |
bool |
false |
no |
| rds_enable_performance_insights | Controls the enabling of RDS Performance insights. Default to true |
bool |
true |
no |
| rds_enable_storage_encryption | Specifies whether the DB instance is encrypted | bool |
false |
no |
| rds_engine | The Database engine for the rds instance | string |
"postgres" |
no |
| rds_engine_version | The version of the database engine. | string |
"11" |
no |
| rds_final_snapshot_identifier | The name of your final DB snapshot when this DB instance is deleted. Must be provided if rds_skip_final_snapshot is set to false. The value must begin with a letter, only contain alphanumeric characters and hyphens, and not end with a hyphen or contain two consecutive hyphens. |
string |
null |
no |
| rds_identifier | Identifier of datastore instance | string |
"" |
no |
| rds_instance_class | The instance type to use | string |
"db.t3.small" |
no |
| rds_iops | The amount of provisioned IOPS. Setting this implies a storage_type of 'io1' | number |
0 |
no |
| rds_max_allocated_storage | The upper limit to which Amazon RDS can automatically scale the storage of the DB instance. Configuring this will automatically ignore differences to allocated_storage. Must be greater than or equal to allocated_storage or 0 to disable Storage Autoscaling. |
number |
200 |
no |
| rds_monitoring_interval | The interval, in seconds, between points when Enhanced Monitoring metrics are collected for the DB instance. To disable collecting Enhanced Monitoring metrics, specify 0. The default is 0. Valid Values: 0, 1, 5, 10, 15, 30, 60. | number |
0 |
no |
| rds_monitoring_role_arn | The ARN for the IAM role that permits RDS to send enhanced monitoring metrics to CloudWatch Logs. Must be specified if monitoring_interval is non-zero. | string |
"" |
no |
| rds_multi_az | Specifies if the RDS instance is multi-AZ. | bool |
false |
no |
| rds_option_group_name | Name of the DB option group to associate | string |
null |
no |
| rds_password | RDS database password for the user | string |
"" |
no |
| rds_security_group_ids | A List of security groups to bind to the rds instance | list(string) |
[] |
no |
| rds_skip_final_snapshot | Determines whether a final DB snapshot is created before the DB instance is deleted. If true is specified, no DBSnapshot is created. If false is specified, a DB snapshot is created before the DB instance is deleted, using the value from final_snapshot_identifier | bool |
true |
no |
| rds_storage_encryption_kms_key_arn | The ARN for the KMS encryption key. If creating an encrypted replica, set this to the destination KMS ARN. If storage_encrypted is set to true and kms_key_id is not specified the default KMS key created in your account will be used | string |
"" |
no |
| rds_subnet_group | Subnet group for RDS instances | string |
"" |
no |
| rds_tags | Additional tags for rds datastore resources | map |
{} |
no |
| rds_username | RDS database user name | string |
"" |
no |
| s3_bucket_name | The name of the bucket. It is recommended to add a namespace/suffix to the name to avoid naming collisions | string |
"" |
no |
| s3_enable_versioning | If versioning should be configured on the bucket | bool |
true |
no |
| s3_tags | Additional tags to be added to the s3 resources | map |
{} |
no |
| tags | Additional tags for all resources in the module. | map |
{} |
no |
| use_rds_snapshot | Controls if an RDS snapshot should be used when creating the rds instance. Will use the latest snapshot of the rds_identifier variable. |
bool |
false |
no |
| Name | Description |
|---|---|
| cluster_config | Kube config file of the current cluster |
| datastore_dynamodb_global_secondary_index_names | DynamoDB secondary index names |
| datastore_dynamodb_local_secondary_index_names | DynamoDB local index names |
| datastore_dynamodb_table_arn | DynamoDB table ARN |
| datastore_dynamodb_table_id | DynamoDB table ID |
| datastore_dynamodb_table_name | DynamoDB table name |
| datastore_dynamodb_table_policy_arn | Policy arn to be attached to the execution role that provide access to the datastore dynamodb. |
| datastore_dynamodb_table_stream_arn | DynamoDB table stream ARN |
| datastore_dynamodb_table_stream_label | DynamoDB table stream label |
| datastore_rds_db_name | The RDS database name |
| datastore_rds_db_url | The RDS connection url in the format of engine://user:password@endpoint/db_name |
| datastore_rds_db_url_encoded | The RDS connection url in the format of engine://user:urlencode(password)@endpoint/db_name |
| datastore_rds_db_user | The RDS db username |
| datastore_rds_engine_version | The actual engine version used by the RDS instance. |
| datastore_rds_instance_address | The address of the RDS instance |
| datastore_rds_instance_arn | The ARN of the RDS instance |
| datastore_rds_instance_endpoint | The connection endpoint |
| datastore_rds_instance_id | The RDS instance ID |
| datastore_s3_bucket_name | The name of the s3 bucket |
| datastore_s3_bucket_policy_arn | Policy arn to be attached to the execution role that provide access to the datastore s3 bucket. |
| k8s_deployment_custom_policy_arn | The custom policy arn created for the service which is attached to the execution role. |
| k8s_deployment_execution_role_arn | The execution role arn created for the service |
| k8s_deployment_execution_role_name | The execution role name created for the service |
See LICENSE for full details.
Copyright 2020 Hypr NZ
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Copyright © 2020 Hypr NZ