Skip to content

Commit

Permalink
Merge pull request #25 from Ontotext-AD/feature/GDB-9377-setup-automa…
Browse files Browse the repository at this point in the history 
…tic-cluster-creation

Setup automatic cluster creation
  • Loading branch information
@viktor-ribchev
viktor-ribchev authored Mar 6, 2024
2 parents e96773a + 0a85d5c commit 005f372
Show file tree
Hide file tree
Showing 14 changed files with 753 additions and 401 deletions.
4 changes: 2 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ The following are the important variables you should configure when using this m

- `instance_type`: The instance type for the GDB cluster nodes. This should match your performance and cost requirements.

- `node_count`: The number of instances in the cluster. Recommended is 3, 5 or 7 in order to have consensus according to the [Raft algorithm](https://raft.github.io/).
- `node_count`: The number of instances in the cluster. Recommended is 3, 5 or 7 in order to have consensus according to the [Raft algorithm](https://raft.github.io/).



Expand Down Expand Up @@ -87,7 +87,7 @@ This Terraform module creates AWS Systems Manager (SSM) parameters for managing
This Terraform module creates a private hosted zone in Amazon Route 53 for DNS resolution in your Virtual Private Cloud (VPC).

#### [IAM](modules/iam/README.md)
This Terraform module creates an AWS Identity and Access Management (IAM) role and an instance profile that can be used for EC2 instances. It also supports the option to set a permissions boundary on the IAM role.
This Terraform module creates an AWS Identity and Access Management (IAM) role and an instance profile that can be used for EC2 instances. It also supports the option to set a permissions boundary on the IAM role.

#### [Load Balancer](modules/load_balancer/README.md)
This Terraform module sets up an AWS Elastic Load Balancer (Network Load Balancer) with optional TLS listeners. The module is designed to be flexible and customizable by accepting various input variables to tailor the NLB configuration to your specific requirements.
Expand Down
225 changes: 112 additions & 113 deletions modules/user_data/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,113 +1,112 @@
# Terraform Module:

This Terraform module configures an AWS EC2 instance for running GraphDB with various optional parameters and user-supplied userdata.

## Usage:

To use this module, include it in your Terraform configuration and provide the required and optional variables:
```hcl
module "graphdb_instance" {
source = "path/to/module"
# Provide required and optional variables
var.aws_region = "us-east-1"
var.resource_name_prefix = "my-graphdb-instance"
var.device_name = "/dev/sdh"
var.backup_schedule = "0 0 * * *"
var.backup_bucket_name = "my-backup-bucket"
var.ebs_volume_type = "gp2"
var.ebs_volume_size = 100
var.ebs_volume_throughput = 100
var.ebs_volume_iops = 100
var.ebs_kms_key_arn = "arn:aws:kms:us-east-1:123456789012:key/abcd1234"
var.zone_dns_name = "myprivatedns.local"
var.zone_id = "Z1234567890"
var.instance_type = "m5.large"
# Optional variables
var.user_supplied_userdata_path = "path/to/userdata_script.sh"
var.backup_retention_count = 7
}
```

## Variables:

### Required Parameters:

`var.aws_region` (string): AWS region where GraphDB is being deployed.

`var.resource_name_prefix` (string): Resource name prefix used for tagging and naming AWS resources.

`var.device_name` (string): The device to which EBS volumes for the GraphDB data directory will be mapped.

`var.backup_schedule` (string): Cron expression for the backup job.

`var.backup_bucket_name` (string): Name of the S3 bucket for storing GraphDB backups.

`var.ebs_volume_type` (string): Type of the EBS volumes used by the GraphDB nodes.

`var.ebs_volume_size` (number): The size of the EBS volumes used by the GraphDB nodes.

`var.ebs_volume_throughput` (number): Throughput for the EBS volumes used by the GraphDB nodes.

`var.ebs_volume_iops` (number): IOPS for the EBS volumes used by the GraphDB nodes.

`var.ebs_kms_key_arn` (string): KMS key used for EBS volume encryption.

`var.zone_dns_name` (string): DNS name for the private hosted zone in Route 53.

`var.zone_id` (string): Route 53 private hosted zone ID.

`var.instance_type` (string): EC2 instance type.

### Optional Parameters:

`var.user_supplied_userdata_path` (string, default: null): File path to custom userdata script supplied by the user.

`var.backup_retention_count` (number, default: 7): Number of backups to keep.

## What the Module Creates

This Terraform module creates an AWS EC2 instance configured for running GraphDB with the following components and settings:

An EC2 instance with specifications based on the specified var.instance_type.
EBS volumes for the GraphDB data directory with the specified type, size, throughput, IOPS, and encryption using the provided KMS key.
A user data script to initialize and start GraphDB, which can be customized using the var.user_supplied_userdata_path variable or a default template.
A backup job schedule using the specified var.backup_schedule.
Configuration for backing up GraphDB to the specified S3 bucket (var.backup_bucket_name).
Private hosted zone DNS settings for Route 53 using the specified var.zone_dns_name and var.zone_id.
The module combines these components and settings to create a fully configured AWS EC2 instance ready to run GraphDB, with the flexibility to customize various parameters to suit your requirements.

## Outputs

The module provides two output values for reference in your Terraform configuration:
`graphdb_userdata_base64_encoded` (string): Base64-encoded user data for the GraphDB instance.
`graphdb_max_memory` (number): Maximum memory for the JVM in GiB, computed based on the EC2 instance type and adjusted for GraphDB's memory requirements.

## Example

Here's a complete example that demonstrates how to use the module:

```hcl
module "graphdb_instance" {
source = "path/to/module" # Replace with the actual source
var.aws_region = "us-east-1"
var.resource_name_prefix = "my-graphdb-instance"
var.device_name = "/dev/sdh"
var.backup_schedule = "0 0 * * *"
var.backup_bucket_name = "my-backup-bucket"
var.ebs_volume_type = "gp2"
var.ebs_volume_size = 100
var.ebs_volume_throughput = 100
var.ebs_volume_iops = 100
var.ebs_kms_key_arn = "arn:aws:kms:us-east-1:123456789012:key/abcd1234"
var.zone_dns_name = "myprivatedns.local"
var.zone_id = "Z1234567890"
var.instance_type = "m5.large"
var.user_supplied_userdata_path = "path/to/userdata_script.sh"
var.backup_retention_count = 7
}
```
This example demonstrates how to use the module to configure an AWS EC2 instance for running GraphDB. Adjust the variables as needed for your specific use case.
# Terraform Module:

This Terraform module configures an AWS EC2 instance for running GraphDB with various optional parameters and user-supplied userdata.

## Usage:

To use this module, include it in your Terraform configuration and provide the required and optional variables:
```hcl
module "graphdb_instance" {
source = "path/to/module"
# Provide required and optional variables
var.aws_region = "us-east-1"
var.resource_name_prefix = "my-graphdb-instance"
var.device_name = "/dev/sdh"
var.backup_schedule = "0 0 * * *"
var.backup_bucket_name = "my-backup-bucket"
var.ebs_volume_type = "gp3"
var.ebs_volume_size = 100
var.ebs_volume_throughput = 150
var.ebs_volume_iops = 3000
var.ebs_kms_key_arn = "arn:aws:kms:us-east-1:123456789012:key/abcd1234"
var.zone_dns_name = "myprivatedns.local"
var.zone_id = "Z1234567890"
var.instance_type = "m5.large"
# Optional variables
var.user_supplied_userdata_path = "path/to/userdata_script.sh"
var.backup_retention_count = 7
}
```

## Variables:

### Required Parameters:

`var.aws_region` (string): AWS region where GraphDB is being deployed.

`var.resource_name_prefix` (string): Resource name prefix used for tagging and naming AWS resources.

`var.device_name` (string): The device to which EBS volumes for the GraphDB data directory will be mapped.

`var.backup_schedule` (string): Cron expression for the backup job.

`var.backup_bucket_name` (string): Name of the S3 bucket for storing GraphDB backups.

`var.ebs_volume_type` (string): Type of the EBS volumes used by the GraphDB nodes.

`var.ebs_volume_size` (number): The size of the EBS volumes used by the GraphDB nodes.

`var.ebs_volume_throughput` (number): Throughput for the EBS volumes used by the GraphDB nodes.

`var.ebs_volume_iops` (number): IOPS for the EBS volumes used by the GraphDB nodes.

`var.ebs_kms_key_arn` (string): KMS key used for EBS volume encryption.

`var.zone_dns_name` (string): DNS name for the private hosted zone in Route 53.

`var.zone_id` (string): Route 53 private hosted zone ID.

`var.instance_type` (string): EC2 instance type.

### Optional Parameters:

`var.user_supplied_userdata_path` (string, default: null): File path to custom userdata script supplied by the user.

`var.backup_retention_count` (number, default: 7): Number of backups to keep.

## What the Module Creates

This Terraform module creates an AWS EC2 instance configured for running GraphDB with the following components and settings:

An EC2 instance with specifications based on the specified var.instance_type.
EBS volumes for the GraphDB data directory with the specified type, size, throughput, IOPS, and encryption using the provided KMS key.
A user data script to initialize and start GraphDB, which can be customized using the var.user_supplied_userdata_path variable or a default template.
A backup job schedule using the specified var.backup_schedule.
Configuration for backing up GraphDB to the specified S3 bucket (var.backup_bucket_name).
Private hosted zone DNS settings for Route 53 using the specified var.zone_dns_name and var.zone_id.
The module combines these components and settings to create a fully configured AWS EC2 instance ready to run GraphDB, with the flexibility to customize various parameters to suit your requirements.

## Outputs

The module provides two output values for reference in your Terraform configuration:
`graphdb_userdata_base64_encoded` (string): Base64-encoded user data for the GraphDB instance.

## Example

Here's a complete example that demonstrates how to use the module:

```hcl
module "graphdb_instance" {
source = "path/to/module" # Replace with the actual source
var.aws_region = "us-east-1"
var.resource_name_prefix = "my-graphdb-instance"
var.device_name = "/dev/sdh"
var.backup_schedule = "0 0 * * *"
var.backup_bucket_name = "my-backup-bucket"
var.ebs_volume_type = "gp3"
var.ebs_volume_size = 100
var.ebs_volume_throughput = 150
var.ebs_volume_iops = 3000
var.ebs_kms_key_arn = "arn:aws:kms:us-east-1:123456789012:key/abcd1234"
var.zone_dns_name = "myprivatedns.local"
var.zone_id = "Z1234567890"
var.instance_type = "m5.large"
var.user_supplied_userdata_path = "path/to/userdata_script.sh"
var.backup_retention_count = 7
}
```
This example demonstrates how to use the module to configure an AWS EC2 instance for running GraphDB. Adjust the variables as needed for your specific use case.
112 changes: 81 additions & 31 deletions modules/user_data/main.tf
View file
Original file line number Diff line number Diff line change
@@ -1,33 +1,83 @@
data "aws_ec2_instance_type" "graphdb" {
instance_type = var.instance_type
}
data "cloudinit_config" "graphdb_user_data" {
base64_encode = true
gzip = true

part {
content_type = "text/x-shellscript"
content = <<-EOF
#!/bin/bash
set -euo pipefail
until ping -c 1 google.com &> /dev/null; do
echo "waiting for outbound connectivity"
sleep 5
done
# Stop GraphDB to override configurations
echo "Stopping GraphDB"
systemctl stop graphdb
EOF
}

part {
content_type = "text/x-shellscript"
content = templatefile("${path.module}/templates/01_disk_management.sh.tpl", {
name : var.resource_name_prefix
ebs_volume_type : var.ebs_volume_type
ebs_volume_size : var.ebs_volume_size
ebs_volume_iops : var.ebs_volume_iops
ebs_volume_throughput : var.ebs_volume_throughput
ebs_kms_key_arn : var.ebs_kms_key_arn
device_name : var.device_name
})
}

part {
content_type = "text/x-shellscript"
content = templatefile("${path.module}/templates/02_dns_provisioning.sh.tpl", {
zone_id : var.zone_id
zone_dns_name : var.zone_dns_name
})
}

part {
content_type = "text/x-shellscript"
content = templatefile("${path.module}/templates/03_gdb_conf_overrides.sh.tpl", {
name : var.resource_name_prefix
region : var.aws_region
})
}

part {
content_type = "text/x-shellscript"
content = templatefile("${path.module}/templates/04_gdb_backup_conf.sh.tpl", {
name : var.resource_name_prefix
region : var.aws_region
backup_schedule : var.backup_schedule
backup_retention_count : var.backup_retention_count
backup_bucket_name : var.backup_bucket_name
})
}

part {
content_type = "text/x-shellscript"
content = templatefile("${path.module}/templates/05_linux_overrides.sh.tpl", {})
}

part {
content_type = "text/x-shellscript"
content = templatefile("${path.module}/templates/06_cloudwatch_setup.sh.tpl", {
name : var.resource_name_prefix
region : var.aws_region
})
}

locals {
# MiB to GiB - 10
jvm_max_memory = ceil(data.aws_ec2_instance_type.graphdb.memory_size * 0.0009765625 - 10)

graphdb_user_data = templatefile(
var.user_supplied_userdata_path != null ? var.user_supplied_userdata_path : "${path.module}/templates/start_graphdb.sh.tpl",
{
region = var.aws_region
name = var.resource_name_prefix
device_name = var.device_name

backup_schedule = var.backup_schedule
backup_retention_count = var.backup_retention_count
backup_bucket_name = var.backup_bucket_name

ebs_volume_type = var.ebs_volume_type
ebs_volume_size = var.ebs_volume_size
ebs_volume_iops = var.ebs_volume_iops
ebs_volume_throughput = var.ebs_volume_throughput
ebs_kms_key_arn = var.ebs_kms_key_arn

zone_dns_name = var.zone_dns_name
zone_id = var.zone_id

jvm_max_memory = local.jvm_max_memory
resource_name_prefix = var.resource_name_prefix
}
)
part {
content_type = "text/x-shellscript"
content = templatefile("${path.module}/templates/07_cluster_setup.sh.tpl", {
name : var.resource_name_prefix
region : var.aws_region
zone_id : var.zone_id
})
}
}
6 changes: 2 additions & 4 deletions modules/user_data/outputs.tf
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
output "graphdb_userdata_base64_encoded" {
value = base64encode(local.graphdb_user_data)
description = "User data script for GraphDB VM scale set."
value = data.cloudinit_config.graphdb_user_data.rendered
}

output "graphdb_max_memory" {
value = local.jvm_max_memory
}
Loading

0 comments on commit 005f372

Please sign in to comment.