Skip to content

This module provides an easy way to set up Athena queries, The module can set up Athena database and table from an input S3 bucket and create-table SQL script.

License

Notifications You must be signed in to change notification settings

clouddrove/terraform-aws-athena

Folders and files

NameName
Last commit message
Last commit date
Oct 13, 2023
Oct 15, 2023
Aug 24, 2023
Aug 24, 2023
Aug 24, 2023
Aug 24, 2023
Aug 24, 2023
Aug 24, 2023
Aug 24, 2023
Aug 25, 2023
Aug 25, 2023
Oct 26, 2023
Aug 25, 2023
Oct 15, 2023
Aug 25, 2023

Repository files navigation

Terraform Module Athena

Terraform Athena module to create new modules using this as baseline

Latest Release tfsec Licence


We eat, drink, sleep and most importantly love DevOps. We are working towards strategies for standardizing architecture while ensuring security for the infrastructure. We are strong believer of the philosophy Bigger problems are always solved by breaking them into smaller manageable problems. Resonating with microservices architecture, it is considered best-practice to run database, cluster, storage in smaller connected yet manageable pieces within the infrastructure.

This module is basically combination of Terraform open source and includes automatation tests and examples. It also helps to create and improve your infrastructure with minimalistic code instead of maintaining the whole infrastructure code yourself.

We have fifty plus terraform modules. A few of them are comepleted and are available for open source usage while a few others are in progress.

Prerequisites

This module has a few dependencies:

Examples

IMPORTANT: Since the master branch used in source varies based on new modifications, we suggest that you use the release versions here.

Here are some examples of how you can use this module in your inventory structure:

Basic Example to create a basic AWS Athena without extra configs

module "athena" {
  source  = "clouddrove/athena/aws"
  version = "1.0.0"

  name        = "athena"
  environment = "test"
  label_order = ["name", "environment"]

  enabled                 = true
  workgroup_force_destroy = true

  # S3 Bucket Configuration
  bucket_force_destroy = true
  s3_output_path       = "accessLogs/queryresults/" # The S3 bucket path used to store query results

  # Database for Athena
  databases = {
    database1 = {
      force_destroy = true
      properties    = {
      custom_prop_1 = "example"
      }
    } 
  }
}

Complete Example to create a basic AWS Athena without extra configs

locals {
  name        = "athena"
  environment = "test"
  label_order = ["name", "environment"]
}

module "s3_bucket" {
  source        = "clouddrove/s3/aws"
  version       = "1.3.0"
  name          = format("%s-bucket-test", local.name)
  versioning    = true
  acl           = "private"
  force_destroy = true
}

module "athena" {
  source                  = "clouddrove/athena/aws"
  version                 = "1.0.0"
  name                    = local.name
  environment             = local.environment
  label_order             = local.label_order
  enabled                 = true
  workgroup_force_destroy = true

  # S3 Bucket Configuration
  create_s3_bucket     =  false
  athena_s3_bucket_id  = module.s3_bucket.id
  s3_output_path       = "outputs/" # The S3 bucket path used to store query results

  # Database for Athena
  databases = {
    database1 = {
      force_destroy = true
      properties = {
      custom_prop_1 = "example"
      }
    } 
  }

  # Data catalog to test terraform
  data_catalogs = {
    glue1 = {
      description = "This is an example to test Terraform"
      type        = "GLUE"
      parameters  = {
        catalog-id : "123456789012" # The catalog_id is the account ID of the AWS account to which the AWS Glue catalog belongs.
      }
    }
  }

  # Named Queries to test terarform
  named_queries = {
    query1 = {
      database    = "database1"
      description = "This is an example query to test Terraform"
      query       = "SELECT * FROM %s limit 10;"
    }
  }
}

Inputs

Name Description Type Default Required
athena_kms_key Use an existing KMS key for Athena if create_workgroup_kms_key is false. string null no
athena_s3_bucket_id Use an existing S3 bucket for Athena query results if create_s3_bucket is false. string null no
bucket_acl Canned ACL to apply to the S3 bucket. string null no
bucket_force_destroy A boolean that indicates all objects should be deleted from the bucket so that the bucket can be destroyed without error. These objects are not recoverable. bool false no
bucket_label_order Label order, e.g. name,application for S3 Bucket. list(any)
[
"name"
]
no
bucket_versioning Enable Versioning of S3. bool true no
bytes_scanned_cutoff_per_query Integer for the upper data usage limit (cutoff) for the amount of bytes a single query in a workgroup is allowed to scan. Must be at least 10485760. number null no
create_database_kms_key Enable the creation of a KMS key used by Athena database. bool true no
create_s3_bucket Conditionally create S3 bucket. bool true no
create_workgroup_kms_key Enable the creation of a KMS key used by Athena workgroup. bool true no
data_catalogs Map of Athena data catalogs and related configuration. map(any) {} no
databases Map of Athena databases and related configuration. map(any) n/a yes
deletion_window_in_days Duration in days after which the key is deleted after destruction of the resource. number 7 no
enabled Set to false to prevent the module from creating AWS Athena related resources. bool false no
enforce_workgroup_configuration Boolean whether the settings for the workgroup override client-side settings. bool true no
environment Environment (e.g. prod, dev, staging). string "" no
kms_key_enabled Specifies whether the kms is enabled or disabled. bool true no
label_order Label order, e.g. name,application. list(any) [] no
managedby ManagedBy, eg 'CloudDrove'. string "hello@clouddrove.com" no
multi_region Indicates whether the KMS key is a multi-Region (true) or regional (false) key. bool true no
name Name (e.g. app or cluster). string "" no
named_queries Map of Athena named queries and related configuration. map(map(string)) {} no
publish_cloudwatch_metrics_enabled Boolean whether Amazon CloudWatch metrics are enabled for the workgroup. bool true no
repository Terraform current module repo string "https://github.com/clouddrove/terraform-aws-athena" no
s3_output_path The S3 bucket path used to store query results. string "" no
tags Additional tags (e.g. map(BusinessUnit,XYZ). map(any) {} no
workgroup_encryption_option Indicates whether Amazon S3 server-side encryption with Amazon S3-managed keys (SSE_S3), server-side encryption with KMS-managed keys (SSE_KMS), or client-side encryption with KMS-managed keys (CSE_KMS) is used. string "SSE_KMS" no
workgroup_force_destroy The option to delete the workgroup and its contents even if the workgroup contains any named queries. bool false no

Outputs

Name Description
bucket_arn ID of S3 bucket used by Athena.
bucket_id ID of S3 bucket used by Athena.
data_catalogs List of newly created Athena data catalogs.
databases List of newly created Athena databases.
kms_key_arn ARN of KMS key used by Athena.
named_queries List of newly created Athena named queries.
workgroup_id ID of newly created Athena workgroup.

Testing

In this module testing is performed with terratest and it creates a small piece of infrastructure, matches the output like ARN, ID and Tags name etc and destroy infrastructure in your AWS account. This testing is written in GO, so you need a GO environment in your system.

You need to run the following command in the testing folder:

  go test -run Test

Feedback

If you come accross a bug or have any feedback, please log it in our issue tracker, or feel free to drop us an email at hello@clouddrove.com.

If you have found it worth your time, go ahead and give us a ★ on our GitHub!

About us

At CloudDrove, we offer expert guidance, implementation support and services to help organisations accelerate their journey to the cloud. Our services include docker and container orchestration, cloud migration and adoption, infrastructure automation, application modernisation and remediation, and performance engineering.

We are The Cloud Experts!


We ❤️ Open Source and you can check out our other modules to get help with your new Cloud ideas.

About

This module provides an easy way to set up Athena queries, The module can set up Athena database and table from an input S3 bucket and create-table SQL script.

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Packages

No packages published